Re: [HACKERS] no universally correct setting for fsync

2010-05-31 Thread Bruce Momjian
Josh Berkus wrote: All, Updated docs based on tracking this discussion. fsync through full page writes recorded below. I have applied this doc update with the attached patch. I added the change from every night to frequently, and reworded it slightly so it was clear it affects the entire

Re: [HACKERS] no universally correct setting for fsync

2010-05-11 Thread Yeb Havinga
Kevin Grittner wrote: Joshua D. Drake j...@commandprompt.com wrote: The answer to this is: PostgreSQL.org recommends that this setting be left on at all times. Turning it off, may lead to data corruption. Anything else is circumstantial and based on knowledge and facts we don't have

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Michael Tharp
On 05/08/2010 04:07 AM, Craig Ringer wrote: It's probably worth mentioning that people who want to turn off fsync to gain a performance boost should instead look at a RAID controller with a BBU so they can safely enable write-back caching, getting most of the benefits of fsync=off safely.

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Bruce Momjian
Michael Tharp wrote: On 05/08/2010 04:07 AM, Craig Ringer wrote: It's probably worth mentioning that people who want to turn off fsync to gain a performance boost should instead look at a RAID controller with a BBU so they can safely enable write-back caching, getting most of the benefits

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Robert Haas
On Mon, May 10, 2010 at 11:12 AM, Bruce Momjian br...@momjian.us wrote: Michael Tharp wrote: On 05/08/2010 04:07 AM, Craig Ringer wrote: It's probably worth mentioning that people who want to turn off fsync to gain a performance boost should instead look at a RAID controller with a BBU so

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Bruce Momjian
Robert Haas wrote: On Mon, May 10, 2010 at 11:12 AM, Bruce Momjian br...@momjian.us wrote: Michael Tharp wrote: On 05/08/2010 04:07 AM, Craig Ringer wrote: It's probably worth mentioning that people who want to turn off fsync to gain a performance boost should instead look at a RAID

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: It might be safe is a bit of a waffle. It would be nice if we could provide some more clear guidance as to whether it is or is not, or how someone could go about testing their hardware to find out. I think that the issue is that you could have

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Greg Stark
On Mon, May 10, 2010 at 4:55 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Robert Haas robertmh...@gmail.com wrote: It might be safe is a bit of a waffle.  It would be nice if we could provide some more clear guidance as to whether it is or is not, or how someone could go about

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Joshua D. Drake
On Mon, 2010-05-10 at 18:46 +0100, Greg Stark wrote: On Mon, May 10, 2010 at 4:55 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Robert Haas robertmh...@gmail.com wrote: It might be safe is a bit of a waffle. It would be nice if we could provide some more clear guidance as to

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Kevin Grittner
Joshua D. Drake j...@commandprompt.com wrote: The answer to this is: PostgreSQL.org recommends that this setting be left on at all times. Turning it off, may lead to data corruption. Anything else is circumstantial and based on knowledge and facts we don't have about environmental

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Tom Lane
Kevin Grittner kevin.gritt...@wicourts.gov writes: Perhaps Josh's language for fsync could be modified to work here (we're now talking about full_page_writes, for anyone who's lost track): | it is only advisable to turn off fsync if you can easily recreate | your entire database from

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Cédric Villemain
2010/5/8 Bernd Helmle maili...@oopsware.de: --On 7. Mai 2010 09:48:53 -0500 Kevin Grittner kevin.gritt...@wicourts.gov wrote: I think it goes beyond tweaking -- I think we should have a bald statement like don't turn this off unless you're OK with losing the entire contents of the database

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Josh Berkus
All, Updated docs based on tracking this discussion. fsync through full page writes recorded below. varlistentry id=guc-fsync xreflabel=fsync indexterm primaryvarnamefsync/ configuration parameter/primary /indexterm termvarnamefsync/varname

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Kevin Grittner
Cédric Villemaincedric.villemain.deb...@gmail.com wrote: On a recent pg_restore -j 32, with perc 6i with BBU, RAID10 8 hd, results were not so bas with fsync turn on. (XFS with nobarrier su and sw) -- deactivate fsync time pg_restore -U postgres -d foodb -j 32 foo.psql real170m0.527s

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Josh Berkus
Wow. In a situation where you save seven minutes (4%), it's hardly worth turning off. I've had it be much higher, especially for really large databases. -- -- Josh Berkus PostgreSQL Experts Inc.

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Ross J. Reedstrom
On Mon, May 10, 2010 at 01:35:32PM -0700, Josh Berkus wrote: deleted, or on a reporting read-only clone of your database which gets recreated very night and is not used for failover. High quality hardware alone s/very/every/ or s/very night/periodically/ Ross -- Ross

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Greg Smith
Josh Berkus wrote: Wow. In a situation where you save seven minutes (4%), it's hardly worth turning off. I've had it be much higher, especially for really large databases. Cedric's system had a non-volatile write cache in it. In that case, a few percentage points of improvement is

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Josh Berkus
The real question is how much of a speed-up fsync provides compared to the same workload with synchronous_commit disabled. The only case for fsync=off is one where that number is much faster. I can't say I've tested this. Most of my head-to-heads on fsync were before asych existed. --

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Josh Berkus
On 5/10/10 2:21 PM, Ross J. Reedstrom wrote: On Mon, May 10, 2010 at 01:35:32PM -0700, Josh Berkus wrote: deleted, or on a reporting read-only clone of your database which gets recreated very night and is not used for failover. High quality hardware alone s/very/every/ or

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Greg Smith
Josh Berkus wrote: The real question is how much of a speed-up fsync provides compared to the same workload with synchronous_commit disabled. The only case for fsync=off is one where that number is much faster. I can't say I've tested this. Most of my head-to-heads on fsync were before

Re: [HACKERS] no universally correct setting for fsync

2010-05-08 Thread Craig Ringer
On 8/05/2010 1:56 AM, Josh Berkus wrote: I never meant to suggest any statement in that section is factually wrong; it's just all too rosy, leading people to believe it's no big deal to turn it off. Yeah, that section is overdue for an update. I'll write some new text and post it to

[HACKERS] no universally correct setting for fsync

2010-05-07 Thread Kevin Grittner
Someone just posted to the -admin list with a database corrupted while running with fsync=off. I was all set to refer him to the documentation explaining why he should stop doing this, but to my surprise the documentation waffles on the issue way past what I think is reasonable.

Re: [HACKERS] no universally correct setting for fsync

2010-05-07 Thread Robert Haas
On Fri, May 7, 2010 at 9:47 AM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Someone just posted to the -admin list with a database corrupted while running with fsync=off.  I was all set to refer him to the documentation explaining why he should stop doing this, but to my surprise the

Re: [HACKERS] no universally correct setting for fsync

2010-05-07 Thread Magnus Hagander
On Fri, May 7, 2010 at 16:00, Robert Haas robertmh...@gmail.com wrote: On Fri, May 7, 2010 at 9:47 AM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Someone just posted to the -admin list with a database corrupted while running with fsync=off.  I was all set to refer him to the

Re: [HACKERS] no universally correct setting for fsync

2010-05-07 Thread Tom Lane
Kevin Grittner kevin.gritt...@wicourts.gov writes: | If you trust your operating system, your hardware, and your | utility company (or your battery backup), you can consider | disabling fsync. Isn't this a little too rosy a picture to paint? I think that statement is true as far as it goes,

Re: [HACKERS] no universally correct setting for fsync

2010-05-07 Thread Andrew Dunstan
Kevin Grittner wrote: There are dire-sounding statements interspersed with: | using fsync results in a performance penalty | Due to the risks involved, there is no universally correct setting | for fsync. | If you trust your operating system, your hardware, and your | utility company

Re: [HACKERS] no universally correct setting for fsync

2010-05-07 Thread Kevin Grittner
Andrew Dunstan and...@dunslane.net wrote: I think the critical question is really whether you are prepared to lose your database. Precisely; and the docs don't make that at all clear. They mention the possibility of database corruption, but downplay it: | When fsync is disabled, the

Re: [HACKERS] no universally correct setting for fsync

2010-05-07 Thread Josh Berkus
I never meant to suggest any statement in that section is factually wrong; it's just all too rosy, leading people to believe it's no big deal to turn it off. Yeah, that section is overdue for an update. I'll write some new text and post it to pgsql-docs. --

Re: [HACKERS] no universally correct setting for fsync

2010-05-07 Thread Bernd Helmle
--On 7. Mai 2010 09:48:53 -0500 Kevin Grittner kevin.gritt...@wicourts.gov wrote: I think it goes beyond tweaking -- I think we should have a bald statement like don't turn this off unless you're OK with losing the entire contents of the database cluster. A brief listing of some cases

Re: [HACKERS] no universally correct setting for fsync

2010-05-07 Thread Tom Lane
Bernd Helmle maili...@oopsware.de writes: I've recently even started to wonder if the performance gain with fsync=off is still that large on modern hardware. While testing large migration procedures to a new version some time ago (on an admitedly fast storage) i forgot here and then to turn

Re: [HACKERS] no universally correct setting for fsync

2010-05-07 Thread Bernd Helmle
--On 7. Mai 2010 19:49:15 -0400 Tom Lane t...@sss.pgh.pa.us wrote: Bernd Helmle maili...@oopsware.de writes: I've recently even started to wonder if the performance gain with fsync=off is still that large on modern hardware. While testing large migration procedures to a new version some