Re: [PERFORM] Create and drop temp table in 8.3.4

2008-11-11 Thread Aidan Van Dyk
Seems like this didn't make it through to the list the first time... * Aidan Van Dyk [EMAIL PROTECTED] [081106 22:19]: * David Rees [EMAIL PROTECTED] [081106 21:22]: 2. A write barrier instructs the lower level hardware that commands issued before the barrier must be written to disk

Re: [PERFORM] Create and drop temp table in 8.3.4

2008-11-11 Thread Matthew Wakeling
On Thu, 6 Nov 2008, Scott Marlowe wrote: Without write barriers, the second we call an fsync it returns true. But that's assuming write barriers work as I understand them. Write barriers do not work as you understand them. Calling fsync always blocks until all the data has made it to safe

Re: [PERFORM] Create and drop temp table in 8.3.4

2008-11-06 Thread Kevin Grittner
Kevin Grittner [EMAIL PROTECTED] wrote: If I find a particular tweak to the background writer or some such is particularly beneficial, I'll post again. It turns out that it was not the PostgreSQL version which was primarily responsible for the performance difference. We updated the kernel

Re: [PERFORM] Create and drop temp table in 8.3.4

2008-11-06 Thread Joshua D. Drake
On Thu, 2008-11-06 at 13:02 -0600, Kevin Grittner wrote: Kevin Grittner [EMAIL PROTECTED] wrote: If I find a particular tweak to the background writer or some such is particularly beneficial, I'll post again. It turns out that it was not the PostgreSQL version which was primarily

Re: [PERFORM] Create and drop temp table in 8.3.4

2008-11-06 Thread Kevin Grittner
Joshua D. Drake [EMAIL PROTECTED] wrote: On Thu, 2008-11-06 at 13:02 -0600, Kevin Grittner wrote: the new kernel defaulted to using write barriers, while the old kernel didn't. Since we have a BBU RAID controller, we will add nobarrier to the fstab entries. This makes file creation and

Re: [PERFORM] Create and drop temp table in 8.3.4

2008-11-06 Thread Scott Carey
To others that may stumble upon this thread: Note that Write Barriers can be very important for data integrity when power loss or hardware failure are a concern. Only disable them if you know the consequences are mitigated by other factors (such as a BBU + db using the WAL log with sync writes),

Re: [PERFORM] Create and drop temp table in 8.3.4

2008-11-06 Thread Kevin Grittner
Scott Carey [EMAIL PROTECTED] wrote: Note that Write Barriers can be very important for data integrity when power loss or hardware failure are a concern. Only disable them if you know the consequences are mitigated by other factors (such as a BBU + db using the WAL log with sync writes), or

Re: [PERFORM] Create and drop temp table in 8.3.4

2008-11-06 Thread Scott Marlowe
On Thu, Nov 6, 2008 at 2:05 PM, Scott Carey [EMAIL PROTECTED] wrote: To others that may stumble upon this thread: Note that Write Barriers can be very important for data integrity when power loss or hardware failure are a concern. Only disable them if you know the consequences are mitigated

Re: [PERFORM] Create and drop temp table in 8.3.4

2008-11-06 Thread Kevin Grittner
Scott Marlowe [EMAIL PROTECTED] wrote: I am pretty sure that with no write barriers that even a BBU hardware caching raid controller cannot guarantee your data. That seems at odds with this: http://oss.sgi.com/projects/xfs/faq.html#wcache_persistent What evidence to you have that the SGI

Re: [PERFORM] Create and drop temp table in 8.3.4

2008-11-06 Thread Scott Marlowe
On Thu, Nov 6, 2008 at 3:33 PM, Kevin Grittner [EMAIL PROTECTED] wrote: Scott Marlowe [EMAIL PROTECTED] wrote: I am pretty sure that with no write barriers that even a BBU hardware caching raid controller cannot guarantee your data. That seems at odds with this:

Re: [PERFORM] Create and drop temp table in 8.3.4

2008-11-06 Thread Kevin Grittner
Scott Marlowe [EMAIL PROTECTED] wrote: On Thu, Nov 6, 2008 at 3:33 PM, Kevin Grittner [EMAIL PROTECTED] wrote: Scott Marlowe [EMAIL PROTECTED] wrote: I am pretty sure that with no write barriers that even a BBU hardware caching raid controller cannot guarantee your data. That seems at

Re: [PERFORM] Create and drop temp table in 8.3.4

2008-11-06 Thread Scott Marlowe
On Thu, Nov 6, 2008 at 4:04 PM, Kevin Grittner [EMAIL PROTECTED] wrote: Scott Marlowe [EMAIL PROTECTED] wrote: On Thu, Nov 6, 2008 at 3:33 PM, Kevin Grittner [EMAIL PROTECTED] wrote: Scott Marlowe [EMAIL PROTECTED] wrote: I am pretty sure that with no write barriers that even a BBU hardware

Re: [PERFORM] Create and drop temp table in 8.3.4

2008-11-06 Thread David Rees
On Thu, Nov 6, 2008 at 4:03 PM, Scott Marlowe [EMAIL PROTECTED] wrote: On Thu, Nov 6, 2008 at 4:04 PM, Kevin Grittner [EMAIL PROTECTED] wrote: Scott Marlowe [EMAIL PROTECTED] wrote: Without write barriers in my file system an fsync request will be immediately returned true, correct? Not as I

[PERFORM] Create and drop temp table in 8.3.4

2008-11-05 Thread Kevin Grittner
We recently upgraded the databases for our circuit court applications from PostgreSQL 8.2.5 to 8.3.4. The application software didn't change. Most software runs fine, and our benchmarks prior to the update tended to show a measurable, if not dramatic, performance improvement overall. We have

Re: [PERFORM] Create and drop temp table in 8.3.4

2008-11-05 Thread Kevin Grittner
Kevin Grittner [EMAIL PROTECTED] wrote: We have found one area where jobs are running much longer and having a greater impact on concurrent jobs -- those where the programmer creates and drops many temporary tables (thousands) within a database transaction. I forgot to include the standard

Re: [PERFORM] Create and drop temp table in 8.3.4

2008-11-05 Thread Kevin Grittner
Kevin Grittner [EMAIL PROTECTED] wrote: I'm going to get the latest snapshot to see if the issue has changed for 8.4devel In testing under today's snapshot, it seemed to take 150,000 writes to create and drop 1,000 temporary tables within a database transaction. The numbers for the various

Re: [PERFORM] Create and drop temp table in 8.3.4

2008-11-05 Thread Tom Lane
Kevin Grittner [EMAIL PROTECTED] writes: I'm trying to quantify the issue, and would appreciate any suggestions, either for mitigation or collecting useful data to find the cause of the performance regression. I create a script which brackets 1000 lines like the following within a single