On 04.02.2012 07:24, Jeff Janes wrote:
Is it safe to assume that, under "#ifdef LWLOCK_STATS", a call to
LWLockAcquire will always precede any calls to LWLockWaitUntilFree
when a new process is started, to calloc the stats assays?
>
I guess it is right now, because the only user is WALWrite, wh
On Mon, Jan 30, 2012 at 6:57 AM, Heikki Linnakangas
wrote:
> On 27.01.2012 15:38, Robert Haas wrote:
>>
>> On Fri, Jan 27, 2012 at 8:35 AM, Heikki Linnakangas
>> wrote:
>>>
>>> Yeah, we have to be careful with any overhead in there, it can be a hot
>>> spot. I wouldn't expect any measurable diff
On 31 January 2012 14:24, Robert Haas wrote:
> I think you're trying to muddy the waters. Heikki's implementation
> was different than yours, and there are some things about it I'm not
> 100% thrilled with, but it's fundamentally the same concept. The new
> idea you're describing here is somethi
On Tue, Jan 31, 2012 at 3:02 AM, Simon Riggs wrote:
> On Tue, Jan 31, 2012 at 7:43 AM, Heikki Linnakangas
> wrote:
>
>> That seems like a pretty marginal gain. If you're bound by the speed of
>> fsyncs, this will reduce the latency by the time it takes to mark the clog,
>> which is tiny in compar
On Tue, Jan 31, 2012 at 7:43 AM, Heikki Linnakangas
wrote:
> That seems like a pretty marginal gain. If you're bound by the speed of
> fsyncs, this will reduce the latency by the time it takes to mark the clog,
> which is tiny in comparison to all the other stuff that needs to happen,
> like, flu
On 31.01.2012 01:35, Simon Riggs wrote:
The plan here is to allow WAL flush and clog updates to occur
concurrently. Which allows the clog contention and update time to be
completely hidden behind the wait for the WAL flush. That is only
possible if we have the WALwriter involved since we need two
On Mon, Jan 30, 2012 at 8:04 PM, Heikki Linnakangas
wrote:
> So, what's the approach you're working on?
I've had a few days leave at end of last week, so no time to fully
discuss the next steps with the patch. That's why you were requested
not to commit anything.
You've suggested there was no r
On 30.01.2012 21:55, Simon Riggs wrote:
On Mon, Jan 30, 2012 at 3:32 PM, Heikki Linnakangas
wrote:
On 30.01.2012 17:18, Simon Riggs wrote:
Peter and I have been working on a new version that seems likely to
improve performance over your suggestions. We should be showing
something soon.
Pl
On Mon, Jan 30, 2012 at 3:32 PM, Heikki Linnakangas
wrote:
> On 30.01.2012 17:18, Simon Riggs wrote:
>> I asked clearly and specifically for you to hold back committing
>> anything. Not sure why you would ignore that and commit without
>> actually asking myself or Peter. On a point of principle a
On Sun, Jan 29, 2012 at 1:20 PM, Greg Smith wrote:
> On 01/28/2012 07:48 PM, Jeff Janes wrote:
>>
>
>> I haven't inspected that deep fall off at 30 clients for the patch.
>> By way of reference, if I turn off synchronous commit, I get
>> tps=1245.8 which is 100% CPU limited. This sets an theoreti
On 30.01.2012 17:18, Simon Riggs wrote:
On Mon, Jan 30, 2012 at 2:57 PM, Heikki Linnakangas
wrote:
I committed this. I ran pgbench test on an 8-core box and didn't see any
slowdown. It would still be good if you get a chance to rerun the bigger
test, but I feel confident that there's no measu
On Mon, Jan 30, 2012 at 2:57 PM, Heikki Linnakangas
wrote:
> I committed this. I ran pgbench test on an 8-core box and didn't see any
> slowdown. It would still be good if you get a chance to rerun the bigger
> test, but I feel confident that there's no measurable slowdown.
I asked clearly and s
On 27.01.2012 15:38, Robert Haas wrote:
On Fri, Jan 27, 2012 at 8:35 AM, Heikki Linnakangas
wrote:
Yeah, we have to be careful with any overhead in there, it can be a hot
spot. I wouldn't expect any measurable difference from the above, though.
Could I ask you to rerun the pgbench tests you di
On 01/28/2012 07:48 PM, Jeff Janes wrote:
Others are going to test this out on high-end systems. I wanted to
try it out on the other end of the scale. I've used a Pentium 4,
3.2GHz,
with 2GB of RAM and with a single IDE drive running ext4. ext4 is
amazingly bad on IDE, giving about 25 fsync's p
On 2012-01-29 01:48, Jeff Janes wrote:
I ran three modes, head, head with commit_delay, and the group_commit patch
shared_buffers = 600MB
wal_sync_method=fsync
optionally with:
commit_delay=5
commit_siblings=1
pgbench -i -s40
for clients in 1 5 10 15 20 25 30
pgbench -T 30 -M prepared -c $cli
On Fri, Jan 27, 2012 at 5:35 AM, Heikki Linnakangas
wrote:
> On 26.01.2012 04:10, Robert Haas wrote:
>
>>
>> I think you should break this off into a new function,
>> LWLockWaitUntilFree(), rather than treating it as a new LWLockMode.
>> Also, instead of adding lwWaitOnly, I would suggest that we
On Fri, Jan 27, 2012 at 8:35 AM, Heikki Linnakangas
wrote:
> Yeah, we have to be careful with any overhead in there, it can be a hot
> spot. I wouldn't expect any measurable difference from the above, though.
> Could I ask you to rerun the pgbench tests you did recently with this patch?
> Or can y
On 26.01.2012 04:10, Robert Haas wrote:
On Wed, Jan 25, 2012 at 3:11 AM, Heikki Linnakangas
wrote:
Attached is a patch to do that. It adds a new mode to
LWLockConditionalAcquire(), LW_EXCLUSIVE_BUT_WAIT. If the lock is free, it
is acquired and the function returns immediately. However, unlike
On Wed, Jan 25, 2012 at 3:11 AM, Heikki Linnakangas
wrote:
> I've been thinking, what exactly is the important part of this group commit
> patch that gives the benefit? Keeping the queue sorted isn't all that
> important - XLogFlush() requests for commits will come in almost the correct
> order an
I've been thinking, what exactly is the important part of this group
commit patch that gives the benefit? Keeping the queue sorted isn't all
that important - XLogFlush() requests for commits will come in almost
the correct order anyway.
I also don't much like the division of labour between gro
On Fri, Jan 20, 2012 at 10:30 PM, Heikki Linnakangas
wrote:
> I spent some time cleaning this up. Details below, but here are the
> highlights:
>
> * Reverted the removal of wal_writer_delay
> * Removed heuristic on big flushes
No contested viewpoints on anything there.
> * Doesn't rely on WAL
On 21 January 2012 03:13, Peter Geoghegan wrote:
> I have taken the time to re-run the benchmark and update the wiki with
> that new information - I'd call it a draw.
On second though, the occasional latency spikes that we see with my
patch (which uses the poll() based latch in the run that is
be
On 20 January 2012 22:30, Heikki Linnakangas
wrote:
> Maybe we should have a heuristic to split a large flush into smaller chunks.
> The WAL segment boundary would be a quite natural split point, for example,
> because when crossing the file boundary you have to issue separate fsync()s
> for the f
I spent some time cleaning this up. Details below, but here are the
highlights:
* Reverted the removal of wal_writer_delay
* Doesn't rely on WAL writer. Any process can act as the "leader" now.
* Removed heuristic on big flushes
* Uses PGSemaphoreLock/Unlock instead of latches
On 20.01.2012 17:
On Thu, Jan 19, 2012 at 10:46 PM, Peter Geoghegan wrote:
> On 19 January 2012 17:40, Robert Haas wrote:
>> I don't know what you mean by this. I think removing wal_writer_delay
>> is premature, because I think it still may have some utility, and the
>> patch removes it. That's a separate change
On 19 January 2012 17:40, Robert Haas wrote:
> I don't know what you mean by this. I think removing wal_writer_delay
> is premature, because I think it still may have some utility, and the
> patch removes it. That's a separate change that should be factored
> out of this patch and discussed sepa
On Wed, Jan 18, 2012 at 5:38 PM, Simon Riggs wrote:
> On Wed, Jan 18, 2012 at 1:23 AM, Robert Haas wrote:
>> On Tue, Jan 17, 2012 at 12:37 PM, Heikki Linnakangas
>> wrote:
>>> I found it very helpful to reduce wal_writer_delay in pgbench tests, when
>>> running with synchronous_commit=off. The r
On Wed, Jan 18, 2012 at 1:23 AM, Robert Haas wrote:
> On Tue, Jan 17, 2012 at 12:37 PM, Heikki Linnakangas
> wrote:
>> I found it very helpful to reduce wal_writer_delay in pgbench tests, when
>> running with synchronous_commit=off. The reason is that hint bits don't get
>> set until the commit r
On Tue, Jan 17, 2012 at 12:37 PM, Heikki Linnakangas
wrote:
> I found it very helpful to reduce wal_writer_delay in pgbench tests, when
> running with synchronous_commit=off. The reason is that hint bits don't get
> set until the commit record is flushed to disk, so making the flushes more
> frequ
Excerpts from Jim Nasby's message of mar ene 17 21:21:57 -0300 2012:
> On Jan 15, 2012, at 4:42 PM, Peter Geoghegan wrote:
> > Attached is a patch that myself and Simon Riggs collaborated on. I
> > took the group commit patch that Simon posted to the list back in
> > November, and partially rewrot
On Jan 15, 2012, at 4:42 PM, Peter Geoghegan wrote:
> Attached is a patch that myself and Simon Riggs collaborated on. I
> took the group commit patch that Simon posted to the list back in
> November, and partially rewrote it.
Forgive me if this is a dumb question, but I noticed a few places doing
On 17 January 2012 17:37, Heikki Linnakangas
wrote:
> I found it very helpful to reduce wal_writer_delay in pgbench tests, when
> running with synchronous_commit=off. The reason is that hint bits don't get
> set until the commit record is flushed to disk, so making the flushes more
> frequent redu
On 17.01.2012 16:35, Peter Geoghegan wrote:
On 16 January 2012 08:11, Heikki Linnakangas
wrote:
I think it might be simpler if it wasn't the background writer that's
responsible for "driving" the group commit queue, but the backends
themselves. When a flush request comes in, you join the queue
On 16 January 2012 08:11, Heikki Linnakangas
wrote:
> Impressive results. How about uploading the PDF to the community wiki?
Sure. http://wiki.postgresql.org/wiki/Group_commit .
> I think it might be simpler if it wasn't the background writer that's
> responsible for "driving" the group commit q
On 16.01.2012 00:42, Peter Geoghegan wrote:
I've also attached the results of a pgbench-tools driven benchmark,
which are quite striking (Just the most relevant image - e-mail me
privately if you'd like a copy of the full report, as I don't want to
send a large PDF file to the list as a courtesy
Attached is a patch that myself and Simon Riggs collaborated on. I
took the group commit patch that Simon posted to the list back in
November, and partially rewrote it. Here is that original thread:
http://archives.postgresql.org/pgsql-hackers/2011-11/msg00802.php
I've also attached the results o
On Wed, Nov 16, 2011 at 1:17 AM, Robert Haas wrote:
> On Tue, Nov 15, 2011 at 7:18 PM, Jeff Janes wrote:
>> When I apply this to HEAD and run make check, it freezes at:
>> test tablespace ...
> [...]
>> Does anyone else see this?
>
> It hangs for me, too, just slightly later:
OK,
On Tue, Nov 15, 2011 at 7:18 PM, Jeff Janes wrote:
> When I apply this to HEAD and run make check, it freezes at:
> test tablespace ...
[...]
> Does anyone else see this?
It hangs for me, too, just slightly later:
== running regression test queries=
On Mon, Nov 14, 2011 at 2:08 AM, Simon Riggs wrote:
> Enclosed patch implements Group Commit and also powersave mode for WALWriter.
>
> XLogFlush() waits for WALWriter to run XLogBackgroundFlush(), which
> flushes WAL and then wakes waiters. Uses same concepts and similar
> code to sync rep.
>
> P
On 11/14/2011 03:43 PM, Josh Berkus wrote:
Purpose is to provide consistent WAL writes, even when WALInsertLock
contended. Currently no "off" option, thinking is that the overhead of
doing this is relatively low and so it can be "always on" - exactly as
it is for sync rep.
Hmmm, have y
> Purpose is to provide consistent WAL writes, even when WALInsertLock
> contended. Currently no "off" option, thinking is that the overhead of
> doing this is relatively low and so it can be "always on" - exactly as
> it is for sync rep.
Hmmm, have you had a chance to do any performance tests?
Enclosed patch implements Group Commit and also powersave mode for WALWriter.
XLogFlush() waits for WALWriter to run XLogBackgroundFlush(), which
flushes WAL and then wakes waiters. Uses same concepts and similar
code to sync rep.
Purpose is to provide consistent WAL writes, even when WALInsertLo
On Thu, 6 Mar 2008, Tom Lane wrote:
What about the other idea of just having committers wait for the next
walwriter-cycle flush before reporting commit?
I haven't considered that too much yet; it may very well be superior to
anything I was thinking of. The only point I was trying to make tod
Greg Smith <[EMAIL PROTECTED]> writes:
> I actually have a paper design for something that builds a little model
> for how likely it is another commit will be coming soon that essentially
> turns this into something that can be tuned automatically, better than any
> person can do it. No idea if
On Thu, 6 Mar 2008, Bruce Momjian wrote:
Should we remove these now that we have async commit?
#commit_delay = 0 # range 0-10, in
microseconds
#commit_siblings = 5# range 1-1000
They seem unfixable.
commit_delay offers a small but
Should we remove these now that we have async commit?
#commit_delay = 0 # range 0-10, in
microseconds
#commit_siblings = 5# range 1-1000
They seem unfixable.
--
This is not ready for 8.3.
This has been saved for the 8.4 release:
http://momjian.postgresql.org/cgi-bin/pgpatches_hold
---
Heikki Linnakangas wrote:
> It's been known for years that commit_delay isn't very good a
On Tue, 2007-04-10 at 11:40 +0100, Heikki Linnakangas wrote:
> Tom Lane wrote:
> > Heikki Linnakangas <[EMAIL PROTECTED]> writes:
> >> I've been working on the patch to enhance our group commit behavior. The
> >> patch is a dirty hack at the moment, but I'm settled on the algorithm
> >> I'm going
Heikki Linnakangas <[EMAIL PROTECTED]> writes:
> I've refrained from spending time on group commit until the
> commit-no-wait patch lands, because it's going to conflict anyway. I'm
> starting to feel we should not try to rush group commit into 8.3, unless
> it somehow falls out of the commit-no
Tom Lane wrote:
Heikki Linnakangas <[EMAIL PROTECTED]> writes:
I've been working on the patch to enhance our group commit behavior. The
patch is a dirty hack at the moment, but I'm settled on the algorithm
I'm going to use and I know the issues involved.
One question that just came to mind is
> > > I've been working on the patch to enhance our group commit
behavior.
> > > The patch is a dirty hack at the moment, but I'm settled on the
> > > algorithm I'm going to use and I know the issues involved.
> >
> > One question that just came to mind is whether Simon's
no-commit-wait
> > pa
> On Mon, 9 Apr 2007, Bruce Momjian wrote:
>
> > The big question is who is going to care about the milliseconds delay
> > and is using a configuration that is going to benefit from commit_delay.
>
> I care. WAL writes are a major bottleneck when many clients are
> committing near the same time
Greg Smith <[EMAIL PROTECTED]> writes:
> An alternate mechanism that tells the client the commit is done when it
> hasn't hit disk is of no use for the applications I work with, so I
> haven't even been paying attention to no-commit-wait.
Agreed, if you need "committed" to mean "committed" then
On Mon, 9 Apr 2007, Bruce Momjian wrote:
The big question is who is going to care about the milliseconds delay
and is using a configuration that is going to benefit from commit_delay.
I care. WAL writes are a major bottleneck when many clients are
committing near the same time. Both times I
Tom Lane wrote:
> Heikki Linnakangas <[EMAIL PROTECTED]> writes:
> > I've been working on the patch to enhance our group commit behavior. The
> > patch is a dirty hack at the moment, but I'm settled on the algorithm
> > I'm going to use and I know the issues involved.
>
> One question that just
Heikki Linnakangas <[EMAIL PROTECTED]> writes:
> I've been working on the patch to enhance our group commit behavior. The
> patch is a dirty hack at the moment, but I'm settled on the algorithm
> I'm going to use and I know the issues involved.
One question that just came to mind is whether Simo
Heikki Linnakangas <[EMAIL PROTECTED]> writes:
> Tom Lane wrote:
>> This is where my bogometer triggered. There's way too many platforms
>> where 1 msec timeout is a sheer fantasy. If you cannot make it perform
>> well with a 10-msec timeout then I don't think it's going to be at all
>> portable.
Tom Lane wrote:
Heikki Linnakangas <[EMAIL PROTECTED]> writes:
I've been working on the patch to enhance our group commit behavior. The
patch is a dirty hack at the moment, but I'm settled on the algorithm
I'm going to use and I know the issues involved.
...
The timeout is currently hard-coded
Heikki Linnakangas <[EMAIL PROTECTED]> writes:
> I've been working on the patch to enhance our group commit behavior. The
> patch is a dirty hack at the moment, but I'm settled on the algorithm
> I'm going to use and I know the issues involved.
> ...
> The timeout is currently hard-coded at 1 ms.
I wrote:
What are the chances of getting this in 8.3, assuming that I rewrite and
submit a patch within the next week or two?
I also intend to do performance testing with different workloads to
ensure the patch doesn't introduce a performance regression under some
conditions.
--
Heikki Li
I've been working on the patch to enhance our group commit behavior. The
patch is a dirty hack at the moment, but I'm settled on the algorithm
I'm going to use and I know the issues involved.
Here's the patch as it is if you want to try it out:
http://community.enterprisedb.com/groupcommit-pghe
Heikki Linnakangas wrote:
Here's what's happening:
1. Client 1 issues fsync (A)
2. Clients 2-5 write their commit record, and try to fsync, but they
have to wait for fsync (A) to finish.
It contains a graph that shows that the patch works very well for this
test case. It's not very good for
It's been known for years that commit_delay isn't very good at giving us
group commit behavior. I did some experiments with this simple test
case: "BEGIN; INSERT INTO test VALUES (1); COMMIT;", with different
numbers of concurrent clients and with and without commit_delay.
Summary for the impa
63 matches
Mail list logo
