Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-03-05 Thread A.M.
On Mar 3, 2007, at 23:19 , Robert Treat wrote: A similar idea we've been kicking around would be having a set storage parameter = nologging option for alter table which would, as it's name implies, cause the system to ignore writing wal logs for the table, much like it does for temp tables

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-03-03 Thread Robert Treat
On Monday 26 February 2007 19:27, A.M. wrote: On Feb 26, 2007, at 18:58 , Simon Riggs wrote: On Mon, 2007-02-26 at 23:25 +, Richard Huxton wrote: Simon Riggs wrote: Proposal: Implement a new option for COMMIT, for enhancing performance, providing a MySQL-like trade-off between

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-03-01 Thread Zeugswetter Andreas ADI SD
Under normal operations, shutting down the database does a checkpoint, right? So unless you're in recovery mode, there's no additional cost. And I can't think of any reason you'd need to see any pages before the last checkpoint (unless you don't trust your disk and just want to check

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-03-01 Thread Gregory Stark
Zeugswetter Andreas ADI SD [EMAIL PROTECTED] writes: Maybe a suitable replacement for full-page would be to sync the first WAL record for a page change before writing the buffer We *always* sync WAL records for page changes before writing the buffer for the page. -- Gregory Stark

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-03-01 Thread Zeugswetter Andreas ADI SD
Maybe a suitable replacement for full-page would be to sync the first WAL record for a page change before writing the buffer We *always* sync WAL records for page changes before writing the buffer for the page. Um, is that so ? And how is that done ? (e.g. bgwriter would need to wait

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-03-01 Thread Heikki Linnakangas
Zeugswetter Andreas ADI SD wrote: Maybe a suitable replacement for full-page would be to sync the first WAL record for a page change before writing the buffer We *always* sync WAL records for page changes before writing the buffer for the page. Um, is that so ? And how is that done ? (e.g.

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-03-01 Thread Jim C. Nasby
On Wed, Feb 28, 2007 at 12:16:10PM -0500, Bruce Momjian wrote: background writer, and I think after a server crash, all pages would have to be read and checked. The good news is that both of these are Would they? If you're doing recovery you'd have to read all pages dirtied since the last

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Zeugswetter Andreas ADI SD
But we do don't we? fsync = off, full_page_writes = off? BTW, our testing seems to indicate that full_page_writes = off is safe on Solaris 10 on good hardware. At least, we haven't been able to break it yet. Is that an OS-dependent parameter? I always assumed it depended

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
Jonah H. Harris [EMAIL PROTECTED] writes: First, rather than using 16-bytes per page and having to deal with handling the non-contiguous space, why not just use a page-level checksum like everyone else? Most of the systems I've seen seem to employ a simple CRC16 or CRC32. I think a CRC

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Gregory Stark [EMAIL PROTECTED] wrote: But we've already seen that CRC checks can be expensive. Not everyone will want to take the cpu hit. Storing a byte counter in every block is cheap. CRC checking a page is most certainly the simplest. And, I disagree that it would be worse

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
Jonah H. Harris [EMAIL PROTECTED] writes: This proposed design is overcomplicated and a waste of space. I mean, we reduce storage overhead using phantom command id and variable varlena, but let's just fill it up again with unnecessary junk bytes. We reduced storage overhead using phantom

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Stefan Kaltenbrunner
Gregory Stark wrote: Jonah H. Harris [EMAIL PROTECTED] writes: This proposed design is overcomplicated and a waste of space. I mean, we reduce storage overhead using phantom command id and variable varlena, but let's just fill it up again with unnecessary junk bytes. We reduced storage

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Gregory Stark [EMAIL PROTECTED] wrote: We reduced storage overhead using phantom command id by 8 bytes *per tuple*. I hardly think 8 bytes per page is much of a concern. You're already losing an average of 1/2 a tuple per page to rounding and that's a minimum of 16 bytes for the

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Joshua D. Drake
Do you really doubt that memcpy is faster than CRC32 checking? Especially when you're already doing memcpy anyways and the only overhead is the few unaligned bytes at the end and the 8 one-byte copies? I'm saying the complexity and implementation of it is going to get you a bit more than

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
I think we need to think about when these CRCs would be read and written. It would be written when it hits the disk, hopefully by the background writer, and I think after a server crash, all pages would have to be read and checked. The good news is that both of these are non-critical paths.

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
Gregory Stark [EMAIL PROTECTED] writes: We've already seen wal CRC checking show up at the top of profiles. Do you really doubt that memcpy is faster than CRC32 checking? Especially when you're already doing memcpy anyways and the only overhead is the few unaligned bytes at the end and the 8

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
Joshua D. Drake [EMAIL PROTECTED] writes: LIVE FROM THE WWE, CAGE MATCH! Jonah (the Theorist) Harris versus Greg (the Brain) Stark. What is going to happen between these two brothers in arms when they must both prove their theory! Darn, I wish I had seen this post before I posted the

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Joshua D. Drake [EMAIL PROTECTED] wrote: LIVE FROM THE WWE, CAGE MATCH! Jonah (the Theorist) Harris versus Greg (the Brain) Stark. What is going to happen between these two brothers in arms when they must both prove their theory! Heh, I like it :) -- Jonah H. Harris, Software

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
Bruce Momjian [EMAIL PROTECTED] writes: I think we need to think about when these CRCs would be read and written. It would be written when it hits the disk, hopefully by the background writer, and I think after a server crash, all pages would have to be read and checked. The good news is

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Joshua D. Drake
Gregory Stark wrote: Joshua D. Drake [EMAIL PROTECTED] writes: LIVE FROM THE WWE, CAGE MATCH! Jonah (the Theorist) Harris versus Greg (the Brain) Stark. What is going to happen between these two brothers in arms when they must both prove their theory! Darn, I wish I had seen this post

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Gregory Stark wrote: Bruce Momjian [EMAIL PROTECTED] writes: I think we need to think about when these CRCs would be read and written. It would be written when it hits the disk, hopefully by the background writer, and I think after a server crash, all pages would have to be read and

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: But if the system was shut down uncleanly as the result of a Postgres crash or fast shutdown of Postgres then that isn't an issue. And many users may prefer to bring the system up as soon as possible as long as they know any corrupt pages

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Jonah H. Harris wrote: On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: But if the system was shut down uncleanly as the result of a Postgres crash or fast shutdown of Postgres then that isn't an issue. And many users may prefer to bring the system up as soon as possible as

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Jonah H. Harris wrote: On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: Am am not comfortable starting and having something fail later. Then do you have some other idea for protecting pages from being torn without storing an entire backup copy or performing a block-level consistency

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: We have several methods suggested to check the blocks, like CRC. My point was that, whatever check method we use, we should be prepared to check on startup, or at least make it the default for a crash restart. Sounds like it should be a guc.

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Jonah H. Harris wrote: On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: We have several methods suggested to check the blocks, like CRC. My point was that, whatever check method we use, we should be prepared to check on startup, or at least make it the default for a crash restart.

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jeff Davis
On Wed, 2007-02-28 at 14:10 -0500, Jonah H. Harris wrote: On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: But if the system was shut down uncleanly as the result of a Postgres crash or fast shutdown of Postgres then that isn't an issue. And many users may prefer to bring the

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: Keep in mind if you don't check it on startup, you will be checking it for every read, which for rare crashes, might not be wise. Well understood. That's how most everyone configures their database systems; they certainly don't optimize for

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Jeff Davis [EMAIL PROTECTED] wrote: That's 5 minutes of data, in the default configuration. Right, but I don't know anyone that keeps checkpoints at 5 minutes. At least not on OLTP configurations. -- Jonah H. Harris, Software Architect | phone: 732.331.1324 EnterpriseDB

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Heikki Linnakangas
Jonah H. Harris wrote: On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: But if the system was shut down uncleanly as the result of a Postgres crash or fast shutdown of Postgres then that isn't an issue. And many users may prefer to bring the system up as soon as possible as long as they

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Joshua D. Drake
Jonah H. Harris wrote: On 2/28/07, Jeff Davis [EMAIL PROTECTED] wrote: That's 5 minutes of data, in the default configuration. Right, but I don't know anyone that keeps checkpoints at 5 minutes. At least not on OLTP configurations. Uhmm... most do because most don't ever touch the

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jeff Davis
On Wed, 2007-02-28 at 14:54 -0500, Jonah H. Harris wrote: On 2/28/07, Jeff Davis [EMAIL PROTECTED] wrote: That's 5 minutes of data, in the default configuration. Right, but I don't know anyone that keeps checkpoints at 5 minutes. At least not on OLTP configurations. It's got a hard

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Jeff Davis [EMAIL PROTECTED] wrote: I'm not saying there's no cost, but the extra recovery cost seems lower to me than the CRC cost on every data page read during operation. I agree, I just think it should be configurable. Also, if we find an error, do we even have the ability to

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Joshua D. Drake [EMAIL PROTECTED] wrote: Right, but I don't know anyone that keeps checkpoints at 5 minutes. At least not on OLTP configurations. Uhmm... most do because most don't ever touch the postgresql.conf and those that do, don't touch checkpoints because they don't

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Joshua D. Drake
Jonah H. Harris wrote: On 2/28/07, Joshua D. Drake [EMAIL PROTECTED] wrote: Right, but I don't know anyone that keeps checkpoints at 5 minutes. At least not on OLTP configurations. Uhmm... most do because most don't ever touch the postgresql.conf and those that do, don't touch checkpoints

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Joshua D. Drake [EMAIL PROTECTED] wrote: I am not trying to pick on the issue but I do think it is important to recognize that literally only those in the know, are going to ever touch the postgresql.conf. I agree. -- Jonah H. Harris, Software Architect | phone: 732.331.1324

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
Jonah H. Harris [EMAIL PROTECTED] writes: Which is, of course, how everyone else does it. I happen to agree with your conclusion but this line of argument is exceptionally unconvincing. In fact in this crowd you'll tend to turn people off and lose people if you say things like that rather

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Gregory Stark [EMAIL PROTECTED] wrote: Jonah H. Harris [EMAIL PROTECTED] writes: Which is, of course, how everyone else does it. I happen to agree with your conclusion but this line of argument is exceptionally unconvincing. In fact in this crowd you'll tend to turn people off and

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: Am am not comfortable starting and having something fail later. Then do you have some other idea for protecting pages from being torn without storing an entire backup copy or performing a block-level consistency check? How other databases do

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jeff Davis
On Wed, 2007-02-28 at 21:13 +, Gregory Stark wrote: Hm that's an interesting thought. We only really have to check pages that would have received a full page write since the last checkpoint. So if we made Do we ever do a partial page write, or is what you're saying equivalent to we only

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Simon Riggs
On Tue, 2007-02-27 at 09:32 -0800, Josh Berkus wrote: Simon, I think if you address me in a mail, it would be best not to explicitly *remove* my name from the address list. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Jonah H. Harris wrote: On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: Am am not comfortable starting and having something fail later. Then do you have some other idea for protecting pages from being torn without storing an entire backup copy or performing a block-level consistency

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
Jonah H. Harris [EMAIL PROTECTED] writes: On 2/28/07, Gregory Stark [EMAIL PROTECTED] wrote: Jonah H. Harris [EMAIL PROTECTED] writes: Which is, of course, how everyone else does it. I happen to agree with your conclusion but this line of argument is exceptionally unconvincing. In fact in

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Josh Berkus
Simon, I think if you address me in a mail, it would be best not to explicitly *remove* my name from the address list. I was trying to remove everyone but the list address. -- --Josh Josh Berkus PostgreSQL @ Sun San Francisco ---(end of

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
Jeff Davis [EMAIL PROTECTED] writes: On Wed, 2007-02-28 at 21:13 +, Gregory Stark wrote: Hm that's an interesting thought. We only really have to check pages that would have received a full page write since the last checkpoint. So if we made Do we ever do a partial page write, or is

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Gregory Stark [EMAIL PROTECTED] wrote: Except that's not what you're doing. There's nothing wrong with saying foo does this clever thing I think we should copy because insert argument. Nor even foo does this thing, would that help us? But what you seem to be saying is *Because* foo

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Gregory Stark [EMAIL PROTECTED] wrote: Well color me surprised, writev is not nearly so much faster than CRC as I had expected: All fun aside, are you going to be submitting a patch for this? -- Jonah H. Harris, Software Architect | phone: 732.331.1324 EnterpriseDB Corporation

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread J. Andrew Rogers
On Feb 28, 2007, at 4:40 PM, Jonah H. Harris wrote: Oracle, Microsoft, IBM, Sybase, Teradata, MySQL, and Firebird have a clever feature called page checksumming which I think we should copy because it's simple and effective at detecting page-level corruption due to torn pages and/or faulty

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, J. Andrew Rogers [EMAIL PROTECTED] wrote: A popular alternative to CRC32 for this purpose is the significantly cheaper and almost as effective is the Adler32 algorithm. I know Google used this algorithm when they added checksumming to their database to tame inexplicable transient

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Jonah H. Harris [EMAIL PROTECTED] wrote: Last time I checked, Teradata used a modified Fletcher checksum as well; I wasn't aware of Adler32. I added adler32 to Gurjeet Singh's crc testing utility, compiled it with gcc 4.0.0 on a single-core Opteron running FC4, and configured it

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Jonah H. Harris [EMAIL PROTECTED] wrote: I added adler32 to Gurjeet Singh's crc testing utility, compiled it with gcc 4.0.0 on a single-core Opteron running FC4, and configured it for 8K block sizes. Tested Fletcher and seeing similar results ratio-wise. Using O3, it's:

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Simon Riggs
On Mon, 2007-02-26 at 23:04 -0500, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: COMMIT NOWAIT can co-exist with the normal form of COMMIT and does not threaten the consistency or robustness of other COMMIT modes. Read that again and think about it, before we go further, please.

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Jim C. Nasby
On Tue, Feb 27, 2007 at 10:49:32AM +, Simon Riggs wrote: I dislike introducing new nonstandard syntax (Oracle compatible is not standard). If we did this I'd vote for control via a GUC setting only; I think that is more useful anyway, as an application can be made to run with such a

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Josh Berkus
Simon, One of the things I love about doing informal online user support in the PostgreSQL community, and formal user support for Sun's customers, is the almost-ironclad guarentee that if a user has a corrupt database or data loss, one of three things is true: a) they didn't apply some

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Tom Lane
Jim C. Nasby [EMAIL PROTECTED] writes: So would you set commit_fsync_delay on a per-transaction basis? That doesn't make much sense to me... I guess I'm not seeing how you would explicitly mark transactions that you didn't want to fsync immediately. My assumption was that most of the time

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Simon Riggs
On Tue, 2007-02-27 at 11:32 -0600, Jim C. Nasby wrote: On Tue, Feb 27, 2007 at 10:49:32AM +, Simon Riggs wrote: I dislike introducing new nonstandard syntax (Oracle compatible is not standard). If we did this I'd vote for control via a GUC setting only; I think that is more useful

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Zeugswetter Andreas ADI SD
There are 2 GUCs that would control the behaviour here: transaction_guarantee = on | off has been enabled. Use this parameter with care; if you find yourself wanting to use this parameter all of the time you should consult a psychiatrist or change open source databases.

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Jonah H. Harris
On 2/27/07, Josh Berkus josh@agliodbs.com wrote: It seriously narrows down the problem space to know that PostgreSQL does *not* allow data loss if it's physically possible to prevent it. Seems like we're trying to protect users from themselves again. This is not a PostgreSQL database issue;

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Josh Berkus
Jonah, Under Oracle, NOWAIT is an asynchronous commit... anyone that uses it should understand that it's still not on-disk and that they can lose it in the event of a failure. That's what Oracle's docs even say. It's just a risk vs. reward trade off. You're missing my point, which is that

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Joshua D. Drake
Josh Berkus wrote: Simon, One of the things I love about doing informal online user support in the PostgreSQL community, and formal user support for Sun's customers, is the almost-ironclad guarentee that if a user has a corrupt database or data loss, one of three things is true: a) they

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Dave Page
Joshua D. Drake wrote: 2. We have to accept that not everyone wants IRON clad data integrity. We have many, many options for dealing with that now, including PITR and REPLICATION. 100% agreed - our own stats collector is extremely similar (in that it may drop data under high load) to a system

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Jordan Henderson
I am not sure about some of this. The Oracle option does not change the engine fsync behavior I believe. All that is changed is whether the client side waits for the complete of the fsync or not. If this is true, the data store, logs, etc, are all protected. The user may still experience a

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Tom Lane
Joshua D. Drake [EMAIL PROTECTED] writes: Josh Berkus wrote: It seriously narrows down the problem space to know that PostgreSQL does *not* allow data loss if it's physically possible to prevent it. But we do don't we? fsync = off, full_page_writes = off? One of the things that's really

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Jonah H. Harris
On 2/27/07, Josh Berkus josh@agliodbs.com wrote: You're missing my point, which is that nobody has demonstrated that there is any real performance gain from this. I see no reason to implement it if there is no performance gain. While I'll back your request for results, it seems nearly

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Bruce Momjian
Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: COMMIT NOWAIT can co-exist with the normal form of COMMIT and does not threaten the consistency or robustness of other COMMIT modes. Read that again and think about it, before we go further, please. I read that, and thought about it,

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Bruce Momjian
Jonah H. Harris wrote: On 2/27/07, Josh Berkus josh@agliodbs.com wrote: You're missing my point, which is that nobody has demonstrated that there is any real performance gain from this. I see no reason to implement it if there is no performance gain. While I'll back your request for

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Bruce Momjian
Jim C. Nasby wrote: On Mon, Feb 26, 2007 at 10:56:58PM +, Simon Riggs wrote: 2. remove fsync parameter Why? Wouldn't fsync=off still speed up checkpoints? ISTM you'd still want this for things like database restores. I think we will remove fsync in favor of the new delay, and allow -1

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Joshua D. Drake
Bruce Momjian wrote: Jonah H. Harris wrote: On 2/27/07, Josh Berkus josh@agliodbs.com wrote: You're missing my point, which is that nobody has demonstrated that there is any real performance gain from this. I see no reason to implement it if there is no performance gain. While I'll back

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Jim C. Nasby
On Tue, Feb 27, 2007 at 05:18:28PM -0500, Bruce Momjian wrote: Jim C. Nasby wrote: On Mon, Feb 26, 2007 at 10:56:58PM +, Simon Riggs wrote: 2. remove fsync parameter Why? Wouldn't fsync=off still speed up checkpoints? ISTM you'd still want this for things like database restores.

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Bruce Momjian
Jim C. Nasby wrote: On Tue, Feb 27, 2007 at 05:18:28PM -0500, Bruce Momjian wrote: Jim C. Nasby wrote: On Mon, Feb 26, 2007 at 10:56:58PM +, Simon Riggs wrote: 2. remove fsync parameter Why? Wouldn't fsync=off still speed up checkpoints? ISTM you'd still want this for

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Jim C. Nasby
On Tue, Feb 27, 2007 at 07:17:37PM -0500, Bruce Momjian wrote: Actually, I don't know that combining both settings is a wise move. The delay should still provide crash protection, whereas with fsync=off you've got absolutely no protection from anything. That's a huge difference, and one

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Gregory Stark
Joshua D. Drake [EMAIL PROTECTED] writes: On 2/27/07, Josh Berkus josh@agliodbs.com wrote: I see no reason to implement it if there is no performance gain. However, I strongly concur that we need at least some evidence. It could easily be that a misstep in the code, causes a loop over the

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Josh Berkus
Tom, One of the things that's really attractive about the proposed mode is that it does *not* create a risk of data corruption Oh, ok. That wasn't how I understood Simon's case. I agree that we ought to look at some performance numbers before accepting the patch, but I think Josh's

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Joshua D. Drake
Gregory Stark wrote: Joshua D. Drake [EMAIL PROTECTED] writes: On 2/27/07, Josh Berkus josh@agliodbs.com wrote: I see no reason to implement it if there is no performance gain. However, I strongly concur that we need at least some evidence. It could easily be that a misstep in the code,

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Bruce Momjian
Joshua D. Drake wrote: Gregory Stark wrote: Joshua D. Drake [EMAIL PROTECTED] writes: On 2/27/07, Josh Berkus josh@agliodbs.com wrote: I see no reason to implement it if there is no performance gain. However, I strongly concur that we need at least some evidence. It could easily

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Josh Berkus
All, But we do don't we? fsync = off, full_page_writes = off? BTW, our testing seems to indicate that full_page_writes = off is safe on Solaris 10 on good hardware. At least, we haven't been able to break it yet. -- Josh Berkus PostgreSQL @ Sun San Francisco

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Jeff Davis
On Tue, 2007-02-27 at 17:40 -0800, Josh Berkus wrote: All, But we do don't we? fsync = off, full_page_writes = off? BTW, our testing seems to indicate that full_page_writes = off is safe on Solaris 10 on good hardware. At least, we haven't been able to break it yet. Is that an

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Josh Berkus
Jeff, Is that an OS-dependent parameter? I always assumed it depended entirely on hardware. I have no way to test it for myself though, so I just leave full_page_writes=on to be safe. It's a question of whether your HW+OS can guarentee no torn page writes for the xlog. Running on Sun

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Gregory Stark
Josh Berkus josh@agliodbs.com writes: OK. I've seen no performance numbers yet though. It just seems to me that any performance patch proposal should start a discussion of what amount of performance we expect to gain. There exist proposals that can be prototyped and measured to see what

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Joshua D. Drake
Bruce Momjian wrote: Joshua D. Drake wrote: Gregory Stark wrote: Joshua D. Drake [EMAIL PROTECTED] writes: On 2/27/07, Josh Berkus josh@agliodbs.com wrote: I see no reason to implement it if there is no performance gain. However, I strongly concur that we need at least some evidence. It

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Gregory Stark
Josh Berkus josh@agliodbs.com writes: It's a question of whether your HW+OS can guarentee no torn page writes for the xlog. no, the data files. torn pages in the xlog is also a problem but we protect ourselves with a CRC and stop replay if it the CRC doesn't match. So the cost there is a

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Jonah H. Harris
On 2/27/07, Gregory Stark [EMAIL PROTECTED] wrote: I suggested a while back implementing torn page detection by writing a sequential number ever 512 bytes in the blocks. (I was talking about WAL at the time but the same principle applies.) Do it at the smgr layer using readv/writev and the upper

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Jeroen T. Vermeulen
On Wed, February 28, 2007 06:59, Jim C. Nasby wrote: On Tue, Feb 27, 2007 at 05:18:28PM -0500, Bruce Momjian wrote: I think we will remove fsync in favor of the new delay, and allow -1 to be the same behavior as fsync off. Well, presumably we'd still allow fsync for some number of

[HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Simon Riggs
Proposal: Implement a new option for COMMIT, for enhancing performance, providing a MySQL-like trade-off between performance and robustness for *only* those that want it. COMMIT NOWAIT This form of COMMIT will *not* perform XLogFlush(), but will rely on a special background process to

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Joshua D. Drake
Why do we want this?? Because some apps have *lots* of data and many really don't care whether they lose a few records. Honestly, I've met people that want this, even after 2 hours of discussion and understanding. Plus probably lots of MySQLers also. Most users will take speed over data loss

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Richard Huxton
Simon Riggs wrote: Proposal: Implement a new option for COMMIT, for enhancing performance, providing a MySQL-like trade-off between performance and robustness for *only* those that want it. COMMIT NOWAIT This form of COMMIT will *not* perform XLogFlush(), but will rely on a special

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Simon Riggs
On Mon, 2007-02-26 at 23:25 +, Richard Huxton wrote: Simon Riggs wrote: Proposal: Implement a new option for COMMIT, for enhancing performance, providing a MySQL-like trade-off between performance and robustness for *only* those that want it. COMMIT NOWAIT This form of

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Alvaro Herrera
Simon Riggs wrote: The interesting point is you can have a huge data grinding app, yet with other tables alongside that hold more important data. In that scenario, 90% of the data would be COMMIT NOWAIT, whilst the small important data is safe. Does this means that the regular COMMIT is

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread A.M.
On Feb 26, 2007, at 18:58 , Simon Riggs wrote: On Mon, 2007-02-26 at 23:25 +, Richard Huxton wrote: Simon Riggs wrote: Proposal: Implement a new option for COMMIT, for enhancing performance, providing a MySQL-like trade-off between performance and robustness for *only* those that

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Jeff Davis
On Mon, 2007-02-26 at 22:56 +, Simon Riggs wrote: Proposal: Implement a new option for COMMIT, for enhancing performance, providing a MySQL-like trade-off between performance and robustness for *only* those that want it. COMMIT NOWAIT This form of COMMIT will *not* perform

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Alvaro Herrera
Jeff Davis wrote: On Mon, 2007-02-26 at 22:56 +, Simon Riggs wrote: Proposal: Implement a new option for COMMIT, for enhancing performance, providing a MySQL-like trade-off between performance and robustness for *only* those that want it. COMMIT NOWAIT This form of

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: COMMIT NOWAIT can co-exist with the normal form of COMMIT and does not threaten the consistency or robustness of other COMMIT modes. Read that again and think about it, before we go further, please. I read that, and thought about it, and don't think I

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Jeroen T. Vermeulen
On Tue, February 27, 2007 06:06, Joshua D. Drake wrote: Why do we want this?? Because some apps have *lots* of data and many really don't care whether they lose a few records. Honestly, I've met people that want this, even after 2 hours of discussion and understanding. Plus probably lots of

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Jim C. Nasby
On Mon, Feb 26, 2007 at 10:56:58PM +, Simon Riggs wrote: 2. remove fsync parameter Why? Wouldn't fsync=off still speed up checkpoints? ISTM you'd still want this for things like database restores. -- Jim Nasby[EMAIL PROTECTED] EnterpriseDB

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Jim C. Nasby
On Tue, Feb 27, 2007 at 11:05:45AM +0700, Jeroen T. Vermeulen wrote: On Tue, February 27, 2007 06:06, Joshua D. Drake wrote: Why do we want this?? Because some apps have *lots* of data and many really don't care whether they lose a few records. Honestly, I've met people that want this,

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Simon Riggs
On Mon, 2007-02-26 at 22:50 -0600, Jim C. Nasby wrote: On Mon, Feb 26, 2007 at 10:56:58PM +, Simon Riggs wrote: 2. remove fsync parameter Why? Wouldn't fsync=off still speed up checkpoints? ISTM you'd still want this for things like database restores. Well, it seemed to be part of the

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Simon Riggs
On Mon, 2007-02-26 at 21:20 -0300, Alvaro Herrera wrote: Simon Riggs wrote: The interesting point is you can have a huge data grinding app, yet with other tables alongside that hold more important data. In that scenario, 90% of the data would be COMMIT NOWAIT, whilst the small important