On 04.02.2012 07:24, Jeff Janes wrote:
Is it safe to assume that, under #ifdef LWLOCK_STATS, a call to
LWLockAcquire will always precede any calls to LWLockWaitUntilFree
when a new process is started, to calloc the stats assays?
I guess it is right now, because the only user is WALWrite,
On Mon, Jan 30, 2012 at 6:57 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 27.01.2012 15:38, Robert Haas wrote:
On Fri, Jan 27, 2012 at 8:35 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Yeah, we have to be careful with any overhead in there, it
On Tue, Jan 31, 2012 at 7:43 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
That seems like a pretty marginal gain. If you're bound by the speed of
fsyncs, this will reduce the latency by the time it takes to mark the clog,
which is tiny in comparison to all the other stuff
On Tue, Jan 31, 2012 at 3:02 AM, Simon Riggs si...@2ndquadrant.com wrote:
On Tue, Jan 31, 2012 at 7:43 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
That seems like a pretty marginal gain. If you're bound by the speed of
fsyncs, this will reduce the latency by the time it
On 31 January 2012 14:24, Robert Haas robertmh...@gmail.com wrote:
I think you're trying to muddy the waters. Heikki's implementation
was different than yours, and there are some things about it I'm not
100% thrilled with, but it's fundamentally the same concept. The new
idea you're
On 27.01.2012 15:38, Robert Haas wrote:
On Fri, Jan 27, 2012 at 8:35 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Yeah, we have to be careful with any overhead in there, it can be a hot
spot. I wouldn't expect any measurable difference from the above, though.
Could I ask
On Mon, Jan 30, 2012 at 2:57 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
I committed this. I ran pgbench test on an 8-core box and didn't see any
slowdown. It would still be good if you get a chance to rerun the bigger
test, but I feel confident that there's no measurable
On 30.01.2012 17:18, Simon Riggs wrote:
On Mon, Jan 30, 2012 at 2:57 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
I committed this. I ran pgbench test on an 8-core box and didn't see any
slowdown. It would still be good if you get a chance to rerun the bigger
test, but I
On Sun, Jan 29, 2012 at 1:20 PM, Greg Smith g...@2ndquadrant.com wrote:
On 01/28/2012 07:48 PM, Jeff Janes wrote:
I haven't inspected that deep fall off at 30 clients for the patch.
By way of reference, if I turn off synchronous commit, I get
tps=1245.8 which is 100% CPU limited. This sets
On Mon, Jan 30, 2012 at 3:32 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 30.01.2012 17:18, Simon Riggs wrote:
I asked clearly and specifically for you to hold back committing
anything. Not sure why you would ignore that and commit without
actually asking myself or
On 30.01.2012 21:55, Simon Riggs wrote:
On Mon, Jan 30, 2012 at 3:32 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 30.01.2012 17:18, Simon Riggs wrote:
Peter and I have been working on a new version that seems likely to
improve performance over your suggestions. We
On Mon, Jan 30, 2012 at 8:04 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
So, what's the approach you're working on?
I've had a few days leave at end of last week, so no time to fully
discuss the next steps with the patch. That's why you were requested
not to commit
On 31.01.2012 01:35, Simon Riggs wrote:
The plan here is to allow WAL flush and clog updates to occur
concurrently. Which allows the clog contention and update time to be
completely hidden behind the wait for the WAL flush. That is only
possible if we have the WALwriter involved since we need
On 2012-01-29 01:48, Jeff Janes wrote:
I ran three modes, head, head with commit_delay, and the group_commit patch
shared_buffers = 600MB
wal_sync_method=fsync
optionally with:
commit_delay=5
commit_siblings=1
pgbench -i -s40
for clients in 1 5 10 15 20 25 30
pgbench -T 30 -M prepared -c
On 01/28/2012 07:48 PM, Jeff Janes wrote:
Others are going to test this out on high-end systems. I wanted to
try it out on the other end of the scale. I've used a Pentium 4,
3.2GHz,
with 2GB of RAM and with a single IDE drive running ext4. ext4 is
amazingly bad on IDE, giving about 25 fsync's
On Fri, Jan 27, 2012 at 5:35 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
On 26.01.2012 04:10, Robert Haas wrote:
I think you should break this off into a new function,
LWLockWaitUntilFree(), rather than treating it as a new LWLockMode.
Also, instead of adding
On 26.01.2012 04:10, Robert Haas wrote:
On Wed, Jan 25, 2012 at 3:11 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Attached is a patch to do that. It adds a new mode to
LWLockConditionalAcquire(), LW_EXCLUSIVE_BUT_WAIT. If the lock is free, it
is acquired and the function
On Fri, Jan 27, 2012 at 8:35 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Yeah, we have to be careful with any overhead in there, it can be a hot
spot. I wouldn't expect any measurable difference from the above, though.
Could I ask you to rerun the pgbench tests you did
I've been thinking, what exactly is the important part of this group
commit patch that gives the benefit? Keeping the queue sorted isn't all
that important - XLogFlush() requests for commits will come in almost
the correct order anyway.
I also don't much like the division of labour between
On Wed, Jan 25, 2012 at 3:11 AM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
I've been thinking, what exactly is the important part of this group commit
patch that gives the benefit? Keeping the queue sorted isn't all that
important - XLogFlush() requests for commits will come
On Fri, Jan 20, 2012 at 10:30 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
I spent some time cleaning this up. Details below, but here are the
highlights:
* Reverted the removal of wal_writer_delay
* Removed heuristic on big flushes
No contested viewpoints on anything
On Thu, Jan 19, 2012 at 10:46 PM, Peter Geoghegan pe...@2ndquadrant.com wrote:
On 19 January 2012 17:40, Robert Haas robertmh...@gmail.com wrote:
I don't know what you mean by this. I think removing wal_writer_delay
is premature, because I think it still may have some utility, and the
patch
I spent some time cleaning this up. Details below, but here are the
highlights:
* Reverted the removal of wal_writer_delay
* Doesn't rely on WAL writer. Any process can act as the leader now.
* Removed heuristic on big flushes
* Uses PGSemaphoreLock/Unlock instead of latches
On 20.01.2012
On 20 January 2012 22:30, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Maybe we should have a heuristic to split a large flush into smaller chunks.
The WAL segment boundary would be a quite natural split point, for example,
because when crossing the file boundary you have to
On 21 January 2012 03:13, Peter Geoghegan pe...@2ndquadrant.com wrote:
I have taken the time to re-run the benchmark and update the wiki with
that new information - I'd call it a draw.
On second though, the occasional latency spikes that we see with my
patch (which uses the poll() based latch
On Wed, Jan 18, 2012 at 5:38 PM, Simon Riggs si...@2ndquadrant.com wrote:
On Wed, Jan 18, 2012 at 1:23 AM, Robert Haas robertmh...@gmail.com wrote:
On Tue, Jan 17, 2012 at 12:37 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
I found it very helpful to reduce wal_writer_delay
On 19 January 2012 17:40, Robert Haas robertmh...@gmail.com wrote:
I don't know what you mean by this. I think removing wal_writer_delay
is premature, because I think it still may have some utility, and the
patch removes it. That's a separate change that should be factored
out of this patch
On Wed, Jan 18, 2012 at 1:23 AM, Robert Haas robertmh...@gmail.com wrote:
On Tue, Jan 17, 2012 at 12:37 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
I found it very helpful to reduce wal_writer_delay in pgbench tests, when
running with synchronous_commit=off. The reason is
On 16 January 2012 08:11, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
Impressive results. How about uploading the PDF to the community wiki?
Sure. http://wiki.postgresql.org/wiki/Group_commit .
I think it might be simpler if it wasn't the background writer that's
responsible
On 17.01.2012 16:35, Peter Geoghegan wrote:
On 16 January 2012 08:11, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
I think it might be simpler if it wasn't the background writer that's
responsible for driving the group commit queue, but the backends
themselves. When a flush
On 17 January 2012 17:37, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
I found it very helpful to reduce wal_writer_delay in pgbench tests, when
running with synchronous_commit=off. The reason is that hint bits don't get
set until the commit record is flushed to disk, so making
On Jan 15, 2012, at 4:42 PM, Peter Geoghegan wrote:
Attached is a patch that myself and Simon Riggs collaborated on. I
took the group commit patch that Simon posted to the list back in
November, and partially rewrote it.
Forgive me if this is a dumb question, but I noticed a few places doing
Excerpts from Jim Nasby's message of mar ene 17 21:21:57 -0300 2012:
On Jan 15, 2012, at 4:42 PM, Peter Geoghegan wrote:
Attached is a patch that myself and Simon Riggs collaborated on. I
took the group commit patch that Simon posted to the list back in
November, and partially rewrote it.
On Tue, Jan 17, 2012 at 12:37 PM, Heikki Linnakangas
heikki.linnakan...@enterprisedb.com wrote:
I found it very helpful to reduce wal_writer_delay in pgbench tests, when
running with synchronous_commit=off. The reason is that hint bits don't get
set until the commit record is flushed to disk,
On 16.01.2012 00:42, Peter Geoghegan wrote:
I've also attached the results of a pgbench-tools driven benchmark,
which are quite striking (Just the most relevant image - e-mail me
privately if you'd like a copy of the full report, as I don't want to
send a large PDF file to the list as a courtesy
Attached is a patch that myself and Simon Riggs collaborated on. I
took the group commit patch that Simon posted to the list back in
November, and partially rewrote it. Here is that original thread:
http://archives.postgresql.org/pgsql-hackers/2011-11/msg00802.php
I've also attached the results
On Wed, Nov 16, 2011 at 1:17 AM, Robert Haas robertmh...@gmail.com wrote:
On Tue, Nov 15, 2011 at 7:18 PM, Jeff Janes jeff.ja...@gmail.com wrote:
When I apply this to HEAD and run make check, it freezes at:
test tablespace ...
[...]
Does anyone else see this?
It hangs for me,
On Mon, Nov 14, 2011 at 2:08 AM, Simon Riggs si...@2ndquadrant.com wrote:
Enclosed patch implements Group Commit and also powersave mode for WALWriter.
XLogFlush() waits for WALWriter to run XLogBackgroundFlush(), which
flushes WAL and then wakes waiters. Uses same concepts and similar
code
On Tue, Nov 15, 2011 at 7:18 PM, Jeff Janes jeff.ja...@gmail.com wrote:
When I apply this to HEAD and run make check, it freezes at:
test tablespace ...
[...]
Does anyone else see this?
It hangs for me, too, just slightly later:
== running regression test queries
Enclosed patch implements Group Commit and also powersave mode for WALWriter.
XLogFlush() waits for WALWriter to run XLogBackgroundFlush(), which
flushes WAL and then wakes waiters. Uses same concepts and similar
code to sync rep.
Purpose is to provide consistent WAL writes, even when
Purpose is to provide consistent WAL writes, even when WALInsertLock
contended. Currently no off option, thinking is that the overhead of
doing this is relatively low and so it can be always on - exactly as
it is for sync rep.
Hmmm, have you had a chance to do any performance tests?
--
On 11/14/2011 03:43 PM, Josh Berkus wrote:
Purpose is to provide consistent WAL writes, even when WALInsertLock
contended. Currently no off option, thinking is that the overhead of
doing this is relatively low and so it can be always on - exactly as
it is for sync rep.
Hmmm, have you
Should we remove these now that we have async commit?
#commit_delay = 0 # range 0-10, in
microseconds
#commit_siblings = 5# range 1-1000
They seem unfixable.
On Thu, 6 Mar 2008, Bruce Momjian wrote:
Should we remove these now that we have async commit?
#commit_delay = 0 # range 0-10, in
microseconds
#commit_siblings = 5# range 1-1000
They seem unfixable.
commit_delay offers a small but
Greg Smith [EMAIL PROTECTED] writes:
I actually have a paper design for something that builds a little model
for how likely it is another commit will be coming soon that essentially
turns this into something that can be tuned automatically, better than any
person can do it. No idea if I'll
On Thu, 6 Mar 2008, Tom Lane wrote:
What about the other idea of just having committers wait for the next
walwriter-cycle flush before reporting commit?
I haven't considered that too much yet; it may very well be superior to
anything I was thinking of. The only point I was trying to make
This is not ready for 8.3.
This has been saved for the 8.4 release:
http://momjian.postgresql.org/cgi-bin/pgpatches_hold
---
Heikki Linnakangas wrote:
It's been known for years that commit_delay isn't very good
On Tue, 2007-04-10 at 11:40 +0100, Heikki Linnakangas wrote:
Tom Lane wrote:
Heikki Linnakangas [EMAIL PROTECTED] writes:
I've been working on the patch to enhance our group commit behavior. The
patch is a dirty hack at the moment, but I'm settled on the algorithm
I'm going to use and I
I've been working on the patch to enhance our group commit
behavior.
The patch is a dirty hack at the moment, but I'm settled on the
algorithm I'm going to use and I know the issues involved.
One question that just came to mind is whether Simon's
no-commit-wait
patch doesn't
Tom Lane wrote:
Heikki Linnakangas [EMAIL PROTECTED] writes:
I've been working on the patch to enhance our group commit behavior. The
patch is a dirty hack at the moment, but I'm settled on the algorithm
I'm going to use and I know the issues involved.
One question that just came to mind is
Heikki Linnakangas [EMAIL PROTECTED] writes:
I've refrained from spending time on group commit until the
commit-no-wait patch lands, because it's going to conflict anyway. I'm
starting to feel we should not try to rush group commit into 8.3, unless
it somehow falls out of the commit-no-wait
Heikki Linnakangas [EMAIL PROTECTED] writes:
I've been working on the patch to enhance our group commit behavior. The
patch is a dirty hack at the moment, but I'm settled on the algorithm
I'm going to use and I know the issues involved.
One question that just came to mind is whether Simon's
Tom Lane wrote:
Heikki Linnakangas [EMAIL PROTECTED] writes:
I've been working on the patch to enhance our group commit behavior. The
patch is a dirty hack at the moment, but I'm settled on the algorithm
I'm going to use and I know the issues involved.
One question that just came to
On Mon, 9 Apr 2007, Bruce Momjian wrote:
The big question is who is going to care about the milliseconds delay
and is using a configuration that is going to benefit from commit_delay.
I care. WAL writes are a major bottleneck when many clients are
committing near the same time. Both times
Greg Smith [EMAIL PROTECTED] writes:
An alternate mechanism that tells the client the commit is done when it
hasn't hit disk is of no use for the applications I work with, so I
haven't even been paying attention to no-commit-wait.
Agreed, if you need committed to mean committed then no-wait
On Mon, 9 Apr 2007, Bruce Momjian wrote:
The big question is who is going to care about the milliseconds delay
and is using a configuration that is going to benefit from commit_delay.
I care. WAL writes are a major bottleneck when many clients are
committing near the same time. Both
I've been working on the patch to enhance our group commit behavior. The
patch is a dirty hack at the moment, but I'm settled on the algorithm
I'm going to use and I know the issues involved.
Here's the patch as it is if you want to try it out:
I wrote:
What are the chances of getting this in 8.3, assuming that I rewrite and
submit a patch within the next week or two?
I also intend to do performance testing with different workloads to
ensure the patch doesn't introduce a performance regression under some
conditions.
--
Heikki
Heikki Linnakangas [EMAIL PROTECTED] writes:
I've been working on the patch to enhance our group commit behavior. The
patch is a dirty hack at the moment, but I'm settled on the algorithm
I'm going to use and I know the issues involved.
...
The timeout is currently hard-coded at 1 ms.
This
Tom Lane wrote:
Heikki Linnakangas [EMAIL PROTECTED] writes:
I've been working on the patch to enhance our group commit behavior. The
patch is a dirty hack at the moment, but I'm settled on the algorithm
I'm going to use and I know the issues involved.
...
The timeout is currently hard-coded
Heikki Linnakangas [EMAIL PROTECTED] writes:
Tom Lane wrote:
This is where my bogometer triggered. There's way too many platforms
where 1 msec timeout is a sheer fantasy. If you cannot make it perform
well with a 10-msec timeout then I don't think it's going to be at all
portable.
The 1
Heikki Linnakangas wrote:
Here's what's happening:
1. Client 1 issues fsync (A)
2. Clients 2-5 write their commit record, and try to fsync, but they
have to wait for fsync (A) to finish.
It contains a graph that shows that the patch works very well for this
test case. It's not very good for
It's been known for years that commit_delay isn't very good at giving us
group commit behavior. I did some experiments with this simple test
case: BEGIN; INSERT INTO test VALUES (1); COMMIT;, with different
numbers of concurrent clients and with and without commit_delay.
Summary for the
63 matches
Mail list logo
