Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-03-05 Thread A.M.
On Mar 3, 2007, at 23:19 , Robert Treat wrote: A similar idea we've been kicking around would be having a set storage parameter = nologging option for alter table which would, as it's name implies, cause the system to ignore writing wal logs for the table, much like it does for temp tables n

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-03-03 Thread Robert Treat
On Monday 26 February 2007 19:27, A.M. wrote: > On Feb 26, 2007, at 18:58 , Simon Riggs wrote: > > On Mon, 2007-02-26 at 23:25 +, Richard Huxton wrote: > >> Simon Riggs wrote: > >>> Proposal: Implement a new option for COMMIT, for enhancing > >>> performance, > >>> providing a MySQL-like trade-

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-03-01 Thread Jim C. Nasby
On Wed, Feb 28, 2007 at 12:16:10PM -0500, Bruce Momjian wrote: > background writer, and I think after a server crash, all pages would > have to be read and checked. The good news is that both of these are Would they? If you're doing recovery you'd have to read all pages dirtied since the last che

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-03-01 Thread Heikki Linnakangas
Zeugswetter Andreas ADI SD wrote: Maybe a suitable replacement for full-page would be to sync the first WAL record for a page change before writing the buffer We *always* sync WAL records for page changes before writing the buffer for the page. Um, is that so ? And how is that done ? (e.g. b

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-03-01 Thread Zeugswetter Andreas ADI SD
> > Maybe a suitable replacement for full-page would be to sync the first > > WAL record for a page change before writing the buffer > > We *always* sync WAL records for page changes before writing > the buffer for the page. Um, is that so ? And how is that done ? (e.g. bgwriter would need to

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-03-01 Thread Gregory Stark
"Zeugswetter Andreas ADI SD" <[EMAIL PROTECTED]> writes: > Maybe a suitable replacement for full-page would be to sync the first WAL > record for a page change before writing the buffer We *always* sync WAL records for page changes before writing the buffer for the page. -- Gregory Stark En

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-03-01 Thread Zeugswetter Andreas ADI SD
> Under normal operations, shutting down the database does a > checkpoint, right? So unless you're in recovery mode, there's > no additional cost. > And I can't think of any reason you'd need to see any pages > before the last checkpoint (unless you don't trust your disk > and just want to che

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Jonah H. Harris <[EMAIL PROTECTED]> wrote: I added adler32 to Gurjeet Singh's crc testing utility, compiled it with gcc 4.0.0 on a single-core Opteron running FC4, and configured it for 8K block sizes. Tested Fletcher and seeing similar results ratio-wise. Using O3, it's: Generati

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Jonah H. Harris <[EMAIL PROTECTED]> wrote: Last time I checked, Teradata used a modified Fletcher checksum as well; I wasn't aware of Adler32. I added adler32 to Gurjeet Singh's crc testing utility, compiled it with gcc 4.0.0 on a single-core Opteron running FC4, and configured it f

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, J. Andrew Rogers <[EMAIL PROTECTED]> wrote: A popular alternative to CRC32 for this purpose is the significantly cheaper and almost as effective is the Adler32 algorithm. I know Google used this algorithm when they added checksumming to their database to tame inexplicable transient c

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread J. Andrew Rogers
On Feb 28, 2007, at 4:40 PM, Jonah H. Harris wrote: Oracle, Microsoft, IBM, Sybase, Teradata, MySQL, and Firebird have a clever feature called page checksumming which I think we should copy because it's simple and effective at detecting page-level corruption due to torn pages and/or faulty stora

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Gregory Stark <[EMAIL PROTECTED]> wrote: Well color me surprised, writev is not nearly so much faster than CRC as I had expected: All fun aside, are you going to be submitting a patch for this? -- Jonah H. Harris, Software Architect | phone: 732.331.1324 EnterpriseDB Corporation

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Gregory Stark <[EMAIL PROTECTED]> wrote: Except that's not what you're doing. There's nothing wrong with saying "foo does this clever thing I think we should copy because ". Nor even "foo does this thing, would that help us?" But what you seem to be saying is "*Because* foo does this

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
"Jeff Davis" <[EMAIL PROTECTED]> writes: > On Wed, 2007-02-28 at 21:13 +, Gregory Stark wrote: >> Hm that's an interesting thought. We only really have to check pages that >> would have received a full page write since the last checkpoint. So if we >> made > > Do we ever do a partial page wri

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Josh Berkus
Simon, > I think if you address me in a mail, it would be best not to explicitly > *remove* my name from the address list. I was trying to remove everyone but the list address. -- --Josh Josh Berkus PostgreSQL @ Sun San Francisco ---(end of broadcast)--

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
"Jonah H. Harris" <[EMAIL PROTECTED]> writes: > On 2/28/07, Gregory Stark <[EMAIL PROTECTED]> wrote: >> "Jonah H. Harris" <[EMAIL PROTECTED]> writes: >> >> > Which is, of course, how everyone else does it. >> >> I happen to agree with your conclusion but this line of argument is >> exceptionally u

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Jonah H. Harris wrote: > On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: > > Am am not comfortable starting and having something fail later. > > Then do you have some other idea for protecting pages from being torn > without storing an entire backup copy or performing a block-level > consiste

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Simon Riggs
On Tue, 2007-02-27 at 09:32 -0800, Josh Berkus wrote: > Simon, I think if you address me in a mail, it would be best not to explicitly *remove* my name from the address list. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcas

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jeff Davis
On Wed, 2007-02-28 at 21:13 +, Gregory Stark wrote: > Hm that's an interesting thought. We only really have to check pages that > would have received a full page write since the last checkpoint. So if we made Do we ever do a partial page write, or is what you're saying equivalent to "we only h

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: Am am not comfortable starting and having something fail later. Then do you have some other idea for protecting pages from being torn without storing an entire backup copy or performing a block-level consistency check? How other databases d

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Gregory Stark <[EMAIL PROTECTED]> wrote: "Jonah H. Harris" <[EMAIL PROTECTED]> writes: > Which is, of course, how everyone else does it. I happen to agree with your conclusion but this line of argument is exceptionally unconvincing. In fact in this crowd you'll tend to turn people o

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
"Jonah H. Harris" <[EMAIL PROTECTED]> writes: > Which is, of course, how everyone else does it. I happen to agree with your conclusion but this line of argument is exceptionally unconvincing. In fact in this crowd you'll tend to turn people off and lose people if you say things like that rather

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Joshua D. Drake <[EMAIL PROTECTED]> wrote: I am not trying to pick on the issue but I do think it is important to recognize that literally only those in the know, are going to ever touch the postgresql.conf. I agree. -- Jonah H. Harris, Software Architect | phone: 732.331.1324 Ente

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Joshua D. Drake
Jonah H. Harris wrote: > On 2/28/07, Joshua D. Drake <[EMAIL PROTECTED]> wrote: >> > Right, but I don't know anyone that keeps checkpoints at 5 minutes. >> > At least not on OLTP configurations. >> >> Uhmm... most do because most don't ever touch the postgresql.conf and >> those that do, don't touc

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Joshua D. Drake <[EMAIL PROTECTED]> wrote: > Right, but I don't know anyone that keeps checkpoints at 5 minutes. > At least not on OLTP configurations. Uhmm... most do because most don't ever touch the postgresql.conf and those that do, don't touch checkpoints because they don't unde

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Jeff Davis <[EMAIL PROTECTED]> wrote: I'm not saying there's no cost, but the extra recovery cost seems lower to me than the CRC cost on every data page read during operation. I agree, I just think it should be configurable. Also, if we find an error, do we even have the ability t

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jeff Davis
On Wed, 2007-02-28 at 14:54 -0500, Jonah H. Harris wrote: > On 2/28/07, Jeff Davis <[EMAIL PROTECTED]> wrote: > > That's 5 minutes of data, in the default configuration. > > Right, but I don't know anyone that keeps checkpoints at 5 minutes. > At least not on OLTP configurations. > It's got a ha

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Joshua D. Drake
Jonah H. Harris wrote: > On 2/28/07, Jeff Davis <[EMAIL PROTECTED]> wrote: >> That's 5 minutes of data, in the default configuration. > > Right, but I don't know anyone that keeps checkpoints at 5 minutes. > At least not on OLTP configurations. Uhmm... most do because most don't ever touch the po

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Heikki Linnakangas
Jonah H. Harris wrote: On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: > But if the system was shut down uncleanly as the result of a Postgres crash or > fast shutdown of Postgres then that isn't an issue. And many users may prefer > to bring the system up as soon as possible as long as th

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Jeff Davis <[EMAIL PROTECTED]> wrote: That's 5 minutes of data, in the default configuration. Right, but I don't know anyone that keeps checkpoints at 5 minutes. At least not on OLTP configurations. -- Jonah H. Harris, Software Architect | phone: 732.331.1324 EnterpriseDB Corporati

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: Keep in mind if you don't check it on startup, you will be checking it for every read, which for rare crashes, might not be wise. Well understood. That's how most everyone configures their database systems; they certainly don't optimize for

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jeff Davis
On Wed, 2007-02-28 at 14:10 -0500, Jonah H. Harris wrote: > On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: > > > But if the system was shut down uncleanly as the result of a Postgres > > > crash or > > > fast shutdown of Postgres then that isn't an issue. And many users may > > > prefer > >

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Jonah H. Harris wrote: > On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: > > We have several methods suggested to check the blocks, like CRC. My > > point was that, whatever check method we use, we should be prepared to > > check on startup, or at least make it the default for a crash restart

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: We have several methods suggested to check the blocks, like CRC. My point was that, whatever check method we use, we should be prepared to check on startup, or at least make it the default for a crash restart. Sounds like it should be a guc.

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Jonah H. Harris wrote: > On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: > > Am am not comfortable starting and having something fail later. > > Then do you have some other idea for protecting pages from being torn > without storing an entire backup copy or performing a block-level > consiste

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Jonah H. Harris wrote: > On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: > > > But if the system was shut down uncleanly as the result of a Postgres > > > crash or > > > fast shutdown of Postgres then that isn't an issue. And many users may > > > prefer > > > to bring the system up as soon a

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: > But if the system was shut down uncleanly as the result of a Postgres crash or > fast shutdown of Postgres then that isn't an issue. And many users may prefer > to bring the system up as soon as possible as long as they know any corrupt > pag

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Gregory Stark wrote: > "Bruce Momjian" <[EMAIL PROTECTED]> writes: > > > I think we need to think about when these CRCs would be read and > > written. It would be written when it hits the disk, hopefully by the > > background writer, and I think after a server crash, all pages would > > have to b

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Joshua D. Drake
Gregory Stark wrote: > "Joshua D. Drake" <[EMAIL PROTECTED]> writes: > >> LIVE FROM THE WWE, CAGE MATCH! >> >> Jonah (the Theorist) Harris versus Greg (the Brain) Stark. >> >> What is going to happen between these two brothers in arms when they >> must both prove their theory! > > Darn, I wish I

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
"Bruce Momjian" <[EMAIL PROTECTED]> writes: > I think we need to think about when these CRCs would be read and > written. It would be written when it hits the disk, hopefully by the > background writer, and I think after a server crash, all pages would > have to be read and checked. The good new

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Joshua D. Drake <[EMAIL PROTECTED]> wrote: LIVE FROM THE WWE, CAGE MATCH! Jonah (the Theorist) Harris versus Greg (the Brain) Stark. What is going to happen between these two brothers in arms when they must both prove their theory! Heh, I like it :) -- Jonah H. Harris, Software A

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > LIVE FROM THE WWE, CAGE MATCH! > > Jonah (the Theorist) Harris versus Greg (the Brain) Stark. > > What is going to happen between these two brothers in arms when they > must both prove their theory! Darn, I wish I had seen this post before I posted

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
"Gregory Stark" <[EMAIL PROTECTED]> writes: > We've already seen wal CRC checking show up at the top of profiles. > > Do you really doubt that memcpy is faster than CRC32 checking? Especially when > you're already doing memcpy anyways and the only overhead is the few unaligned > bytes at the end a

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
I think we need to think about when these CRCs would be read and written. It would be written when it hits the disk, hopefully by the background writer, and I think after a server crash, all pages would have to be read and checked. The good news is that both of these are non-critical paths. ---

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Joshua D. Drake
>> Do you really doubt that memcpy is faster than CRC32 checking? >> Especially when >> you're already doing memcpy anyways and the only overhead is the few >> unaligned >> bytes at the end and the 8 one-byte copies? > > I'm saying the complexity and implementation of it is going to get you > a b

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Gregory Stark <[EMAIL PROTECTED]> wrote: We reduced storage overhead using phantom command id by 8 bytes *per tuple*. I hardly think 8 bytes per page is much of a concern. You're already losing an average of 1/2 a tuple per page to rounding and that's a minimum of 16 bytes for the nar

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Stefan Kaltenbrunner
Gregory Stark wrote: "Jonah H. Harris" <[EMAIL PROTECTED]> writes: This proposed design is overcomplicated and a waste of space. I mean, we reduce storage overhead using phantom command id and variable varlena, but let's just fill it up again with unnecessary junk bytes. We reduced storage o

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
"Jonah H. Harris" <[EMAIL PROTECTED]> writes: > This proposed design is overcomplicated and a waste of space. I mean, > we reduce storage overhead using phantom command id and variable > varlena, but let's just fill it up again with unnecessary junk bytes. We reduced storage overhead using phan

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Gregory Stark <[EMAIL PROTECTED]> wrote: But we've already seen that CRC checks can be expensive. Not everyone will want to take the cpu hit. Storing a byte counter in every block is cheap. CRC checking a page is most certainly the simplest. And, I disagree that it would be worse t

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
"Jonah H. Harris" <[EMAIL PROTECTED]> writes: > First, rather than using 16-bytes per page and having to deal with > handling the non-contiguous space, why not just use a page-level > checksum like everyone else? Most of the systems I've seen seem to > employ a simple CRC16 or CRC32. I think a C

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Zeugswetter Andreas ADI SD
> > > > But we do don't we? fsync = off, full_page_writes = off? > > > > BTW, our testing seems to indicate that full_page_writes = off is safe > > on Solaris 10 on good hardware. At least, we haven't been able to break it yet. > > > > Is that an OS-dependent parameter? I always assumed it de

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Jeroen T. Vermeulen
On Wed, February 28, 2007 06:59, Jim C. Nasby wrote: > On Tue, Feb 27, 2007 at 05:18:28PM -0500, Bruce Momjian wrote: >> I think we will remove fsync in favor of the new delay, and allow -1 to >> be the same behavior as fsync off. > > Well, presumably we'd still allow fsync for some number of vers

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Jonah H. Harris
On 2/27/07, Gregory Stark <[EMAIL PROTECTED]> wrote: I suggested a while back implementing torn page detection by writing a sequential number ever 512 bytes in the blocks. (I was talking about WAL at the time but the same principle applies.) Do it at the smgr layer using readv/writev and the uppe

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Gregory Stark
"Josh Berkus" writes: > It's a question of whether your HW+OS can guarentee no torn page writes for > the xlog. no, the data files. torn pages in the xlog is also a problem but we protect ourselves with a CRC and stop replay if it the CRC doesn't match. So the cost there is a bit of cpu, not

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Joshua D. Drake
Bruce Momjian wrote: > Joshua D. Drake wrote: >> Gregory Stark wrote: >>> "Joshua D. Drake" <[EMAIL PROTECTED]> writes: >>> >> On 2/27/07, Josh Berkus wrote: >>> I see no reason to implement it if there is no performance gain. However, I strongly concur that we need at least some evid

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Gregory Stark
"Josh Berkus" writes: > OK. I've seen no performance numbers yet though. It just seems to me that > any performance patch proposal should start a discussion of what amount of > performance we expect to gain. There exist proposals that can be prototyped and measured to see what potential they

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Josh Berkus
Jeff, > Is that an OS-dependent parameter? I always assumed it depended entirely > on hardware. I have no way to test it for myself though, so I just leave > full_page_writes=on to be safe. It's a question of whether your HW+OS can guarentee no torn page writes for the xlog. Running on Sun hard

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Jeff Davis
On Tue, 2007-02-27 at 17:40 -0800, Josh Berkus wrote: > All, > > > > But we do don't we? fsync = off, full_page_writes = off? > > BTW, our testing seems to indicate that full_page_writes = off is safe on > Solaris 10 on good hardware. At least, we haven't been able to break it yet. > Is that

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Josh Berkus
All, > > But we do don't we? fsync = off, full_page_writes = off? BTW, our testing seems to indicate that full_page_writes = off is safe on Solaris 10 on good hardware. At least, we haven't been able to break it yet. -- Josh Berkus PostgreSQL @ Sun San Francisco ---(

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Bruce Momjian
Joshua D. Drake wrote: > Gregory Stark wrote: > > "Joshua D. Drake" <[EMAIL PROTECTED]> writes: > > > On 2/27/07, Josh Berkus wrote: > > I see no reason to implement it if there is no performance gain. > > > >> However, I strongly concur that we need at least some evidence. It could > >

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Joshua D. Drake
Gregory Stark wrote: > "Joshua D. Drake" <[EMAIL PROTECTED]> writes: > On 2/27/07, Josh Berkus wrote: > I see no reason to implement it if there is no performance gain. > >> However, I strongly concur that we need at least some evidence. It could >> easily be that a misstep in the code,

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Josh Berkus
Tom, > One of the things that's really attractive about the proposed mode is > that it does *not* create a risk of data corruption Oh, ok. That wasn't how I understood Simon's case. > I agree that we ought to look at some performance numbers before > accepting the patch, but I think Josh's a

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Gregory Stark
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: >>> On 2/27/07, Josh Berkus wrote: I see no reason to implement it if there is no performance gain. > However, I strongly concur that we need at least some evidence. It could > easily be that a misstep in the code, causes a loop over the w

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Jim C. Nasby
On Tue, Feb 27, 2007 at 07:17:37PM -0500, Bruce Momjian wrote: > > Actually, I don't know that combining both settings is a wise move. The > > delay should still provide crash protection, whereas with fsync=off > > you've got absolutely no protection from anything. That's a huge > > difference, and

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Bruce Momjian
Jim C. Nasby wrote: > On Tue, Feb 27, 2007 at 05:18:28PM -0500, Bruce Momjian wrote: > > Jim C. Nasby wrote: > > > On Mon, Feb 26, 2007 at 10:56:58PM +, Simon Riggs wrote: > > > > 2. remove fsync parameter > > > > > > Why? Wouldn't fsync=off still speed up checkpoints? ISTM you'd still > > > w

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Jim C. Nasby
On Tue, Feb 27, 2007 at 05:18:28PM -0500, Bruce Momjian wrote: > Jim C. Nasby wrote: > > On Mon, Feb 26, 2007 at 10:56:58PM +, Simon Riggs wrote: > > > 2. remove fsync parameter > > > > Why? Wouldn't fsync=off still speed up checkpoints? ISTM you'd still > > want this for things like database

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Joshua D. Drake
Bruce Momjian wrote: > Jonah H. Harris wrote: >> On 2/27/07, Josh Berkus wrote: >>> You're missing my point, which is that nobody has demonstrated that there >>> is any real performance gain from this. I see no reason to implement it >>> if there is no performance gain. >> While I'll back your re

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Bruce Momjian
Jim C. Nasby wrote: > On Mon, Feb 26, 2007 at 10:56:58PM +, Simon Riggs wrote: > > 2. remove fsync parameter > > Why? Wouldn't fsync=off still speed up checkpoints? ISTM you'd still > want this for things like database restores. I think we will remove fsync in favor of the new delay, and allo

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Bruce Momjian
Jonah H. Harris wrote: > On 2/27/07, Josh Berkus wrote: > > You're missing my point, which is that nobody has demonstrated that there > > is any real performance gain from this. I see no reason to implement it > > if there is no performance gain. > > While I'll back your request for results, it

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Bruce Momjian
Tom Lane wrote: > "Simon Riggs" <[EMAIL PROTECTED]> writes: > > COMMIT NOWAIT can co-exist with the normal form of COMMIT and does not > > threaten the consistency or robustness of other COMMIT modes. Read that > > again and think about it, before we go further, please. > > I read that, and though

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Jonah H. Harris
On 2/27/07, Josh Berkus wrote: You're missing my point, which is that nobody has demonstrated that there is any real performance gain from this. I see no reason to implement it if there is no performance gain. While I'll back your request for results, it seems nearly impossible to expect no p

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Tom Lane
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > Josh Berkus wrote: >> It seriously narrows down the problem space to know that PostgreSQL does >> *not* >> allow data loss if it's physically possible to prevent it. > But we do don't we? fsync = off, full_page_writes = off? One of the things that

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Jordan Henderson
I am not sure about some of this. The Oracle option does not change the engine fsync behavior I believe. All that is changed is whether the client side waits for the complete of the fsync or not. If this is true, the data store, logs, etc, are all protected. The user may still experience a d

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Dave Page
Joshua D. Drake wrote: > 2. We have to accept that not everyone wants IRON clad data integrity. > We have many, many options for dealing with that now, including PITR and > REPLICATION. 100% agreed - our own stats collector is extremely similar (in that it may drop data under high load) to a syst

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Joshua D. Drake
Josh Berkus wrote: > Simon, > > One of the things I love about doing informal online user support in the > PostgreSQL community, and formal user support for Sun's customers, is the > almost-ironclad guarentee that if a user has a corrupt database or data loss, > one of three things is true: > a

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Josh Berkus
Jonah, > Under Oracle, NOWAIT is an asynchronous commit... anyone that uses it > should understand that it's still not on-disk and that they can lose > it in the event of a failure. That's what Oracle's docs even say. > It's just a risk vs. reward trade off. You're missing my point, which is tha

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Jonah H. Harris
On 2/27/07, Josh Berkus wrote: It seriously narrows down the problem space to know that PostgreSQL does *not* allow data loss if it's physically possible to prevent it. Seems like we're trying to protect users from themselves again. This is not a PostgreSQL database issue; it's a feature desi

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Zeugswetter Andreas ADI SD
> There are 2 GUCs that would control the behaviour here: > > transaction_guarantee = on | off > has been enabled. Use this parameter with care; if you find > yourself wanting to use this parameter all of the time you > should consult a psychiatrist or change open source databas

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Simon Riggs
On Tue, 2007-02-27 at 11:32 -0600, Jim C. Nasby wrote: > On Tue, Feb 27, 2007 at 10:49:32AM +, Simon Riggs wrote: > > > I dislike introducing new nonstandard syntax ("Oracle compatible" is not > > > standard). If we did this I'd vote for control via a GUC setting only; > > > I think that is mo

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Tom Lane
"Jim C. Nasby" <[EMAIL PROTECTED]> writes: > So would you set commit_fsync_delay on a per-transaction basis? That > doesn't make much sense to me... I guess I'm not seeing how you would > explicitly mark transactions that you didn't want to fsync immediately. My assumption was that most of the tim

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Josh Berkus
Simon, One of the things I love about doing informal online user support in the PostgreSQL community, and formal user support for Sun's customers, is the almost-ironclad guarentee that if a user has a corrupt database or data loss, one of three things is true: a) they didn't apply some recommen

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Jim C. Nasby
On Tue, Feb 27, 2007 at 10:49:32AM +, Simon Riggs wrote: > > I dislike introducing new nonstandard syntax ("Oracle compatible" is not > > standard). If we did this I'd vote for control via a GUC setting only; > > I think that is more useful anyway, as an application can be made to run > > with

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-27 Thread Simon Riggs
On Mon, 2007-02-26 at 23:04 -0500, Tom Lane wrote: > "Simon Riggs" <[EMAIL PROTECTED]> writes: > > COMMIT NOWAIT can co-exist with the normal form of COMMIT and does not > > threaten the consistency or robustness of other COMMIT modes. Read that > > again and think about it, before we go further, p

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Simon Riggs
On Mon, 2007-02-26 at 21:20 -0300, Alvaro Herrera wrote: > Simon Riggs wrote: > > > The interesting point is you can have a huge data grinding app, yet with > > other tables alongside that hold more important data. In that scenario, > > 90% of the data would be COMMIT NOWAIT, whilst the small impo

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Simon Riggs
On Mon, 2007-02-26 at 22:50 -0600, Jim C. Nasby wrote: > On Mon, Feb 26, 2007 at 10:56:58PM +, Simon Riggs wrote: > > 2. remove fsync parameter > > Why? Wouldn't fsync=off still speed up checkpoints? ISTM you'd still > want this for things like database restores. Well, it seemed to be part of

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Jim C. Nasby
On Tue, Feb 27, 2007 at 11:05:45AM +0700, Jeroen T. Vermeulen wrote: > On Tue, February 27, 2007 06:06, Joshua D. Drake wrote: > > > >> Why do we want this?? Because some apps have *lots* of data and many > >> really don't care whether they lose a few records. Honestly, I've met > >> people that wa

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Jim C. Nasby
On Mon, Feb 26, 2007 at 10:56:58PM +, Simon Riggs wrote: > 2. remove fsync parameter Why? Wouldn't fsync=off still speed up checkpoints? ISTM you'd still want this for things like database restores. -- Jim Nasby[EMAIL PROTECTED] EnterpriseDB ht

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Jeroen T. Vermeulen
On Tue, February 27, 2007 06:06, Joshua D. Drake wrote: > >> Why do we want this?? Because some apps have *lots* of data and many >> really don't care whether they lose a few records. Honestly, I've met >> people that want this, even after 2 hours of discussion and >> understanding. Plus probably l

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Tom Lane
"Simon Riggs" <[EMAIL PROTECTED]> writes: > COMMIT NOWAIT can co-exist with the normal form of COMMIT and does not > threaten the consistency or robustness of other COMMIT modes. Read that > again and think about it, before we go further, please. I read that, and thought about it, and don't think

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Alvaro Herrera
Jeff Davis wrote: > On Mon, 2007-02-26 at 22:56 +, Simon Riggs wrote: > > Proposal: Implement a new option for COMMIT, for enhancing performance, > > providing a MySQL-like trade-off between performance and robustness for > > *only* those that want it. > > > > COMMIT NOWAIT > > > > Th

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Jeff Davis
On Mon, 2007-02-26 at 22:56 +, Simon Riggs wrote: > Proposal: Implement a new option for COMMIT, for enhancing performance, > providing a MySQL-like trade-off between performance and robustness for > *only* those that want it. > > COMMIT NOWAIT > > This form of COMMIT will *not* perfo

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread A.M.
On Feb 26, 2007, at 18:58 , Simon Riggs wrote: On Mon, 2007-02-26 at 23:25 +, Richard Huxton wrote: Simon Riggs wrote: Proposal: Implement a new option for COMMIT, for enhancing performance, providing a MySQL-like trade-off between performance and robustness for *only* those that want

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Alvaro Herrera
Simon Riggs wrote: > The interesting point is you can have a huge data grinding app, yet with > other tables alongside that hold more important data. In that scenario, > 90% of the data would be COMMIT NOWAIT, whilst the small important data > is safe. Does this means that the regular COMMIT is s

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Simon Riggs
On Mon, 2007-02-26 at 23:25 +, Richard Huxton wrote: > Simon Riggs wrote: > > Proposal: Implement a new option for COMMIT, for enhancing performance, > > providing a MySQL-like trade-off between performance and robustness for > > *only* those that want it. > > > > COMMIT NOWAIT > > >

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Richard Huxton
Simon Riggs wrote: Proposal: Implement a new option for COMMIT, for enhancing performance, providing a MySQL-like trade-off between performance and robustness for *only* those that want it. COMMIT NOWAIT This form of COMMIT will *not* perform XLogFlush(), but will rely on a special back

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Joshua D. Drake
> Why do we want this?? Because some apps have *lots* of data and many > really don't care whether they lose a few records. Honestly, I've met > people that want this, even after 2 hours of discussion and > understanding. Plus probably lots of MySQLers also. Most users will take speed over data l

[HACKERS] COMMIT NOWAIT Performance Option

2007-02-26 Thread Simon Riggs
Proposal: Implement a new option for COMMIT, for enhancing performance, providing a MySQL-like trade-off between performance and robustness for *only* those that want it. COMMIT NOWAIT This form of COMMIT will *not* perform XLogFlush(), but will rely on a special background process to per