Re: [HACKERS] Group commit, revised

2012-02-07 Thread Heikki Linnakangas
On 04.02.2012 07:24, Jeff Janes wrote: Is it safe to assume that, under #ifdef LWLOCK_STATS, a call to LWLockAcquire will always precede any calls to LWLockWaitUntilFree when a new process is started, to calloc the stats assays? I guess it is right now, because the only user is WALWrite,

Re: [HACKERS] Group commit, revised

2012-02-03 Thread Jeff Janes
On Mon, Jan 30, 2012 at 6:57 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 27.01.2012 15:38, Robert Haas wrote: On Fri, Jan 27, 2012 at 8:35 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com  wrote: Yeah, we have to be careful with any overhead in there, it

Re: [HACKERS] Group commit, revised

2012-01-31 Thread Simon Riggs
On Tue, Jan 31, 2012 at 7:43 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: That seems like a pretty marginal gain. If you're bound by the speed of fsyncs, this will reduce the latency by the time it takes to mark the clog, which is tiny in comparison to all the other stuff

Re: [HACKERS] Group commit, revised

2012-01-31 Thread Robert Haas
On Tue, Jan 31, 2012 at 3:02 AM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, Jan 31, 2012 at 7:43 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: That seems like a pretty marginal gain. If you're bound by the speed of fsyncs, this will reduce the latency by the time it

Re: [HACKERS] Group commit, revised

2012-01-31 Thread Peter Geoghegan
On 31 January 2012 14:24, Robert Haas robertmh...@gmail.com wrote: I think you're trying to muddy the waters.  Heikki's implementation was different than yours, and there are some things about it I'm not 100% thrilled with, but it's fundamentally the same concept.  The new idea you're

Re: [HACKERS] Group commit, revised

2012-01-30 Thread Heikki Linnakangas
On 27.01.2012 15:38, Robert Haas wrote: On Fri, Jan 27, 2012 at 8:35 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Yeah, we have to be careful with any overhead in there, it can be a hot spot. I wouldn't expect any measurable difference from the above, though. Could I ask

Re: [HACKERS] Group commit, revised

2012-01-30 Thread Simon Riggs
On Mon, Jan 30, 2012 at 2:57 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I committed this. I ran pgbench test on an 8-core box and didn't see any slowdown. It would still be good if you get a chance to rerun the bigger test, but I feel confident that there's no measurable

Re: [HACKERS] Group commit, revised

2012-01-30 Thread Heikki Linnakangas
On 30.01.2012 17:18, Simon Riggs wrote: On Mon, Jan 30, 2012 at 2:57 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I committed this. I ran pgbench test on an 8-core box and didn't see any slowdown. It would still be good if you get a chance to rerun the bigger test, but I

Re: [HACKERS] Group commit, revised

2012-01-30 Thread Jeff Janes
On Sun, Jan 29, 2012 at 1:20 PM, Greg Smith g...@2ndquadrant.com wrote: On 01/28/2012 07:48 PM, Jeff Janes wrote: I haven't inspected that deep fall off at 30 clients for the patch. By way of reference, if I turn off synchronous commit, I get tps=1245.8 which is 100% CPU limited.  This sets

Re: [HACKERS] Group commit, revised

2012-01-30 Thread Simon Riggs
On Mon, Jan 30, 2012 at 3:32 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 30.01.2012 17:18, Simon Riggs wrote: I asked clearly and specifically for you to hold back committing anything. Not sure why you would ignore that and commit without actually asking myself or

Re: [HACKERS] Group commit, revised

2012-01-30 Thread Heikki Linnakangas
On 30.01.2012 21:55, Simon Riggs wrote: On Mon, Jan 30, 2012 at 3:32 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 30.01.2012 17:18, Simon Riggs wrote: Peter and I have been working on a new version that seems likely to improve performance over your suggestions. We

Re: [HACKERS] Group commit, revised

2012-01-30 Thread Simon Riggs
On Mon, Jan 30, 2012 at 8:04 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: So, what's the approach you're working on? I've had a few days leave at end of last week, so no time to fully discuss the next steps with the patch. That's why you were requested not to commit

Re: [HACKERS] Group commit, revised

2012-01-30 Thread Heikki Linnakangas
On 31.01.2012 01:35, Simon Riggs wrote: The plan here is to allow WAL flush and clog updates to occur concurrently. Which allows the clog contention and update time to be completely hidden behind the wait for the WAL flush. That is only possible if we have the WALwriter involved since we need

Re: [HACKERS] Group commit, revised

2012-01-29 Thread Jesper Krogh
On 2012-01-29 01:48, Jeff Janes wrote: I ran three modes, head, head with commit_delay, and the group_commit patch shared_buffers = 600MB wal_sync_method=fsync optionally with: commit_delay=5 commit_siblings=1 pgbench -i -s40 for clients in 1 5 10 15 20 25 30 pgbench -T 30 -M prepared -c

Re: [HACKERS] Group commit, revised

2012-01-29 Thread Greg Smith
On 01/28/2012 07:48 PM, Jeff Janes wrote: Others are going to test this out on high-end systems. I wanted to try it out on the other end of the scale. I've used a Pentium 4, 3.2GHz, with 2GB of RAM and with a single IDE drive running ext4. ext4 is amazingly bad on IDE, giving about 25 fsync's

Re: [HACKERS] Group commit, revised

2012-01-28 Thread Jeff Janes
On Fri, Jan 27, 2012 at 5:35 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 26.01.2012 04:10, Robert Haas wrote: I think you should break this off into a new function, LWLockWaitUntilFree(), rather than treating it as a new LWLockMode. Also, instead of adding

Re: [HACKERS] Group commit, revised

2012-01-27 Thread Heikki Linnakangas
On 26.01.2012 04:10, Robert Haas wrote: On Wed, Jan 25, 2012 at 3:11 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Attached is a patch to do that. It adds a new mode to LWLockConditionalAcquire(), LW_EXCLUSIVE_BUT_WAIT. If the lock is free, it is acquired and the function

Re: [HACKERS] Group commit, revised

2012-01-27 Thread Robert Haas
On Fri, Jan 27, 2012 at 8:35 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Yeah, we have to be careful with any overhead in there, it can be a hot spot. I wouldn't expect any measurable difference from the above, though. Could I ask you to rerun the pgbench tests you did

Re: [HACKERS] Group commit, revised

2012-01-25 Thread Heikki Linnakangas
I've been thinking, what exactly is the important part of this group commit patch that gives the benefit? Keeping the queue sorted isn't all that important - XLogFlush() requests for commits will come in almost the correct order anyway. I also don't much like the division of labour between

Re: [HACKERS] Group commit, revised

2012-01-25 Thread Robert Haas
On Wed, Jan 25, 2012 at 3:11 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I've been thinking, what exactly is the important part of this group commit patch that gives the benefit? Keeping the queue sorted isn't all that important - XLogFlush() requests for commits will come

Re: [HACKERS] Group commit, revised

2012-01-24 Thread Simon Riggs
On Fri, Jan 20, 2012 at 10:30 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I spent some time cleaning this up. Details below, but here are the highlights: * Reverted the removal of wal_writer_delay * Removed heuristic on big flushes No contested viewpoints on anything

Re: [HACKERS] Group commit, revised

2012-01-20 Thread Robert Haas
On Thu, Jan 19, 2012 at 10:46 PM, Peter Geoghegan pe...@2ndquadrant.com wrote: On 19 January 2012 17:40, Robert Haas robertmh...@gmail.com wrote: I don't know what you mean by this.  I think removing wal_writer_delay is premature, because I think it still may have some utility, and the patch

Re: [HACKERS] Group commit, revised

2012-01-20 Thread Heikki Linnakangas
I spent some time cleaning this up. Details below, but here are the highlights: * Reverted the removal of wal_writer_delay * Doesn't rely on WAL writer. Any process can act as the leader now. * Removed heuristic on big flushes * Uses PGSemaphoreLock/Unlock instead of latches On 20.01.2012

Re: [HACKERS] Group commit, revised

2012-01-20 Thread Peter Geoghegan
On 20 January 2012 22:30, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Maybe we should have a heuristic to split a large flush into smaller chunks. The WAL segment boundary would be a quite natural split point, for example, because when crossing the file boundary you have to

Re: [HACKERS] Group commit, revised

2012-01-20 Thread Peter Geoghegan
On 21 January 2012 03:13, Peter Geoghegan pe...@2ndquadrant.com wrote: I have taken the time to re-run the benchmark and update the wiki with that new information - I'd call it a draw. On second though, the occasional latency spikes that we see with my patch (which uses the poll() based latch

Re: [HACKERS] Group commit, revised

2012-01-19 Thread Robert Haas
On Wed, Jan 18, 2012 at 5:38 PM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, Jan 18, 2012 at 1:23 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, Jan 17, 2012 at 12:37 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I found it very helpful to reduce wal_writer_delay

Re: [HACKERS] Group commit, revised

2012-01-19 Thread Peter Geoghegan
On 19 January 2012 17:40, Robert Haas robertmh...@gmail.com wrote: I don't know what you mean by this.  I think removing wal_writer_delay is premature, because I think it still may have some utility, and the patch removes it.  That's a separate change that should be factored out of this patch

Re: [HACKERS] Group commit, revised

2012-01-18 Thread Simon Riggs
On Wed, Jan 18, 2012 at 1:23 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, Jan 17, 2012 at 12:37 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I found it very helpful to reduce wal_writer_delay in pgbench tests, when running with synchronous_commit=off. The reason is

Re: [HACKERS] Group commit, revised

2012-01-17 Thread Peter Geoghegan
On 16 January 2012 08:11, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Impressive results. How about uploading the PDF to the community wiki? Sure. http://wiki.postgresql.org/wiki/Group_commit . I think it might be simpler if it wasn't the background writer that's responsible

Re: [HACKERS] Group commit, revised

2012-01-17 Thread Heikki Linnakangas
On 17.01.2012 16:35, Peter Geoghegan wrote: On 16 January 2012 08:11, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I think it might be simpler if it wasn't the background writer that's responsible for driving the group commit queue, but the backends themselves. When a flush

Re: [HACKERS] Group commit, revised

2012-01-17 Thread Peter Geoghegan
On 17 January 2012 17:37, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I found it very helpful to reduce wal_writer_delay in pgbench tests, when running with synchronous_commit=off. The reason is that hint bits don't get set until the commit record is flushed to disk, so making

Re: [HACKERS] Group commit, revised

2012-01-17 Thread Jim Nasby
On Jan 15, 2012, at 4:42 PM, Peter Geoghegan wrote: Attached is a patch that myself and Simon Riggs collaborated on. I took the group commit patch that Simon posted to the list back in November, and partially rewrote it. Forgive me if this is a dumb question, but I noticed a few places doing

Re: [HACKERS] Group commit, revised

2012-01-17 Thread Alvaro Herrera
Excerpts from Jim Nasby's message of mar ene 17 21:21:57 -0300 2012: On Jan 15, 2012, at 4:42 PM, Peter Geoghegan wrote: Attached is a patch that myself and Simon Riggs collaborated on. I took the group commit patch that Simon posted to the list back in November, and partially rewrote it.

Re: [HACKERS] Group commit, revised

2012-01-17 Thread Robert Haas
On Tue, Jan 17, 2012 at 12:37 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I found it very helpful to reduce wal_writer_delay in pgbench tests, when running with synchronous_commit=off. The reason is that hint bits don't get set until the commit record is flushed to disk,

Re: [HACKERS] Group commit, revised

2012-01-16 Thread Heikki Linnakangas
On 16.01.2012 00:42, Peter Geoghegan wrote: I've also attached the results of a pgbench-tools driven benchmark, which are quite striking (Just the most relevant image - e-mail me privately if you'd like a copy of the full report, as I don't want to send a large PDF file to the list as a courtesy

[HACKERS] Group commit, revised

2012-01-15 Thread Peter Geoghegan
Attached is a patch that myself and Simon Riggs collaborated on. I took the group commit patch that Simon posted to the list back in November, and partially rewrote it. Here is that original thread: http://archives.postgresql.org/pgsql-hackers/2011-11/msg00802.php I've also attached the results