Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-05-07 Thread Greg Smith
Erik Rijkers wrote: Everything together: the raid is what Areca call 'raid10(1E)'. (to be honest I don't remember what that 1E exactly means - extra flexibility in the number of disks, I think). Standard RAID10 only supports an even number of disks. The 1E variants also allow putting an

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-05-04 Thread Erik Rijkers
Hi Simon, In another thread you mentioned you were lacking information from me: On Tue, May 4, 2010 17:10, Simon Riggs wrote: There is no evidence that Erik's strange performance has anything to do with HS; it hasn't been seen elsewhere and he didn't respond to questions about the test setup

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-05-04 Thread Simon Riggs
On Tue, 2010-05-04 at 18:10 +0200, Erik Rijkers wrote: It would be interesting if anyone repeated these simple tests and produced evidence that these non-HS. (Unfortunately, I have at the moment not much time for more testing) Would you be able to make those systems available for further

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-05-04 Thread Erik Rijkers
On Tue, May 4, 2010 18:19, Simon Riggs wrote: On Tue, 2010-05-04 at 18:10 +0200, Erik Rijkers wrote: It would be interesting if anyone repeated these simple tests and produced evidence that these non-HS. (Unfortunately, I have at the moment not much time for more testing) Would you be able

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-05-04 Thread Greg Smith
Erik Rijkers wrote: OS: Centos 5.4 2 quadcores: Intel(R) Xeon(R) CPU X5482 @ 3.20GHz Areca 1280ML primary and standby db both on a 12 disk array (sata 7200rpm, Seagat Barracuda ES.2) To fill in from data you already mentioned upthread: 32 GB RAM CentOS release 5.4 (Final), x86_64

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-05-04 Thread Stefan Kaltenbrunner
Erik Rijkers wrote: Hi Simon, In another thread you mentioned you were lacking information from me: On Tue, May 4, 2010 17:10, Simon Riggs wrote: There is no evidence that Erik's strange performance has anything to do with HS; it hasn't been seen elsewhere and he didn't respond to questions

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-05-04 Thread Erik Rijkers
On Tue, May 4, 2010 20:26, Greg Smith wrote: Erik Rijkers wrote: OS: Centos 5.4 2 quadcores: Intel(R) Xeon(R) CPU X5482 @ 3.20GHz Areca 1280ML primary and standby db both on a 12 disk array (sata 7200rpm, Seagat Barracuda ES.2) To fill in from data you already mentioned upthread:

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-05-04 Thread Simon Riggs
On Tue, 2010-05-04 at 21:34 +0200, Stefan Kaltenbrunner wrote: FWIW - I'm seeing a behaviour here under pgbench -S workloads that looks kinda related. using -j 16 -c 16 -T 120 I get either 10tps and around 66 contextswitches per second or on some runs I end up with 15tps and

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-28 Thread Simon Riggs
On Tue, 2010-04-27 at 20:13 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Tue, 2010-04-27 at 18:08 -0400, Tom Lane wrote: Huh? How is a filter as coarse as an oldest-running-XID filter going to prevent that? And aren't we initializing from trustworthy data in

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-27 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: v3 attached This patch changes KnownAssignedXidsRemove() so that failure to find the target XID is elog(ERROR) (ie, a PANIC, since this is in the startup process). However, this comment is still there: /* * We can fail to find an xid

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-27 Thread Simon Riggs
On Tue, 2010-04-27 at 13:52 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: v3 attached This patch changes KnownAssignedXidsRemove() so that failure to find the target XID is elog(ERROR) (ie, a PANIC, since this is in the startup process). Not in all cases. The code is

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-27 Thread Tom Lane
Hmm ... there's another point here, which is that the array size creates a hard maximum on the number of entries, whereas the hash table was a bit more forgiving. What is the proof that the array won't overflow? The fact that the equivalent data structure on the master can't hold more than this

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-27 Thread Simon Riggs
On Tue, 2010-04-27 at 14:53 -0400, Tom Lane wrote: Hmm ... there's another point here, which is that the array size creates a hard maximum on the number of entries, whereas the hash table was a bit more forgiving. What is the proof that the array won't overflow? The fact that the equivalent

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-27 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Tue, 2010-04-27 at 13:52 -0400, Tom Lane wrote: WTF? Either the comment is wrong or this should not be an elog condition. That section of code has been rewritten many times. I think it is now inaccurate and should be removed. I left it there

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-27 Thread Simon Riggs
On Tue, 2010-04-27 at 16:18 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Tue, 2010-04-27 at 13:52 -0400, Tom Lane wrote: WTF? Either the comment is wrong or this should not be an elog condition. That section of code has been rewritten many times. I think it is

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-27 Thread Tom Lane
Isn't the snapshotOldestActiveXid filter in RecordKnownAssignedTransactionIds completely wrong/useless/bogus? AFAICS, snapshotOldestActiveXid is only set once at the start of recovery. This means it will soon be too old to provide any useful filtering. But what's far worse is that the XID space

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-27 Thread Simon Riggs
On Tue, 2010-04-27 at 17:24 -0400, Tom Lane wrote: Isn't the snapshotOldestActiveXid filter in RecordKnownAssignedTransactionIds completely wrong/useless/bogus? AFAICS, snapshotOldestActiveXid is only set once at the start of recovery. This means it will soon be too old to provide any

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-27 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Tue, 2010-04-27 at 17:24 -0400, Tom Lane wrote: I think we should just lose that test, as well as the variable. Yes, though it looks like it is still necessary in creating a valid initial state because otherwise we may have xids in KnownAssigned

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-27 Thread Simon Riggs
On Tue, 2010-04-27 at 18:08 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Tue, 2010-04-27 at 17:24 -0400, Tom Lane wrote: I think we should just lose that test, as well as the variable. Yes, though it looks like it is still necessary in creating a valid initial

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-27 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Tue, 2010-04-27 at 18:08 -0400, Tom Lane wrote: Huh? How is a filter as coarse as an oldest-running-XID filter going to prevent that? And aren't we initializing from trustworthy data in ProcArrayApplyRecoveryInfo, anyway? I still say it's

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-26 Thread Fujii Masao
On Mon, Apr 26, 2010 at 3:25 AM, Erik Rijkers e...@xs4all.nl wrote: FWIW, here are some more results from pgbench comparing primary and standby (both with Simon's patch). Was there a difference in CPU utilization between the primary and standby? Regards, -- Fujii Masao NIPPON TELEGRAPH AND

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-26 Thread Simon Riggs
On Sun, 2010-04-25 at 19:18 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: [ v2 patch ] I've been studying this some more while making notes for improved comments, and I've about come to the conclusion that having readers move the tail pointer (at the end of

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-26 Thread Simon Riggs
On Sun, 2010-04-25 at 23:52 +0200, Erik Rijkers wrote: I'll try to repeat this pattern on other hardware; although if my tests were run with faulty hardware I wouldn't know how/why that would give the above effect (such a 'regular aberration'). testing is more difficult than I thought...

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-26 Thread Erik Rijkers
On Mon, April 26, 2010 08:52, Fujii Masao wrote: On Mon, Apr 26, 2010 at 3:25 AM, Erik Rijkers e...@xs4all.nl wrote: FWIW, here are some more results from pgbench comparing primary and standby (both with Simon's patch). Was there a difference in CPU utilization between the primary and

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-26 Thread Erik Rijkers
On Mon, April 26, 2010 09:43, Simon Riggs wrote: On Sun, 2010-04-25 at 23:52 +0200, Erik Rijkers wrote: I'll try to repeat this pattern on other hardware; although if my tests were run with faulty hardware I wouldn't know how/why that would give the above effect (such a 'regular aberration').

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-26 Thread Simon Riggs
On Sun, 2010-04-25 at 13:51 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Sun, 2010-04-25 at 13:33 -0400, Tom Lane wrote: If you like I'll have a go at rewriting the comments for this patch, because I am currently thinking that the problem is not so much with the

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-26 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: v3 attached Thanks, will work on this tomorrow. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-25 Thread Robert Haas
On Sat, Apr 24, 2010 at 5:17 AM, Simon Riggs si...@2ndquadrant.com wrote: Both Heikki and I objected to that patch. Please explain your objection, based upon the patch and my explanations. Well, we objected to the locking. Having reread the patch a few times though, I think I'm starting to

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-25 Thread Simon Riggs
On Sun, 2010-04-25 at 06:53 -0400, Robert Haas wrote: On Sat, Apr 24, 2010 at 5:17 AM, Simon Riggs si...@2ndquadrant.com wrote: Both Heikki and I objected to that patch. Please explain your objection, based upon the patch and my explanations. Well, we objected to the locking. Having

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-25 Thread Robert Haas
On Sun, Apr 25, 2010 at 8:50 AM, Simon Riggs si...@2ndquadrant.com wrote: On Sun, 2010-04-25 at 06:53 -0400, Robert Haas wrote: On Sat, Apr 24, 2010 at 5:17 AM, Simon Riggs si...@2ndquadrant.com wrote: Both Heikki and I objected to that patch. Please explain your objection, based upon the

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-25 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Sat, Apr 24, 2010 at 5:17 AM, Simon Riggs si...@2ndquadrant.com wrote: Both Heikki and I objected to that patch. Please explain your objection, based upon the patch and my explanations. Well, we objected to the locking. Not only is the locking

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-25 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: [ v2 patch ] BTW, while I'm looking at this, why bother with the separate KnownAssignedXidsValid[] array? Wouldn't it be cheaper (certainly so in storage, probably so in access/update times) to have just the KnownAssignedXids[] array and store

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-25 Thread Simon Riggs
On Sun, 2010-04-25 at 11:50 -0400, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Sat, Apr 24, 2010 at 5:17 AM, Simon Riggs si...@2ndquadrant.com wrote: Both Heikki and I objected to that patch. Please explain your objection, based upon the patch and my explanations.

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-25 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Sun, 2010-04-25 at 11:50 -0400, Tom Lane wrote: This needs a redesign before it can be considered committable. I don't really care whether it makes things faster; it's too complicated and too poorly documented to be maintainable. There are more

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-25 Thread Simon Riggs
On Sun, 2010-04-25 at 12:43 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: [ v2 patch ] BTW, while I'm looking at this, why bother with the separate KnownAssignedXidsValid[] array? Wouldn't it be cheaper (certainly so in storage, probably so in access/update times) to

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-25 Thread Simon Riggs
On Sun, 2010-04-25 at 12:51 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Sun, 2010-04-25 at 11:50 -0400, Tom Lane wrote: This needs a redesign before it can be considered committable. I don't really care whether it makes things faster; it's too complicated and too

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-25 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Sun, 2010-04-25 at 12:51 -0400, Tom Lane wrote: If the comments were correct, I wouldn't be complaining. They're misleading or outright wrong on many points. In particular, I don't think you actually understand the weak-memory-ordering issue,

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-25 Thread Simon Riggs
On Sun, 2010-04-25 at 13:33 -0400, Tom Lane wrote: If you like I'll have a go at rewriting the comments for this patch, because I am currently thinking that the problem is not so much with the code as with the poor explanation of what it's doing. Sometimes the author is too close to the code

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-25 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Sun, 2010-04-25 at 13:33 -0400, Tom Lane wrote: If you like I'll have a go at rewriting the comments for this patch, because I am currently thinking that the problem is not so much with the code as with the poor explanation of what it's doing.

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-25 Thread Erik Rijkers
On Sat, April 24, 2010 01:17, Erik Rijkers wrote: On Sat, April 24, 2010 00:39, Simon Riggs wrote: On Fri, 2010-04-23 at 11:32 -0400, Robert Haas wrote: 99% of transactions happen in similar times between primary and standby, everything dragged down by rare but severe spikes. We're

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-25 Thread Tom Lane
Erik Rijkers e...@xs4all.nl writes: FWIW, here are some more results from pgbench comparing primary and standby (both with Simon's patch). That seems weird. Why do most of the runs show primary and standby as having comparable speed, but a few show the standby as much slower? The parameters

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-25 Thread Simon Riggs
On Sun, 2010-04-25 at 20:25 +0200, Erik Rijkers wrote: Sorry if it's too much data, but to me at least it was illuminating; I now understand the effects of the different parameters better. That's great, many thanks. A few observations * Standby performance is actually slightly above normal

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-25 Thread Erik Rijkers
On Sun, April 25, 2010 20:55, Tom Lane wrote: That seems weird. Why do most of the runs show primary and standby as having comparable speed, but a few show the standby as much slower? The parameters for those runs don't seem obviously different from cases where it's fast. I think there

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-25 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: [ v2 patch ] I've been studying this some more while making notes for improved comments, and I've about come to the conclusion that having readers move the tail pointer (at the end of KnownAssignedXidsGetAndSetXmin) is overly tricky and probably not a

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-24 Thread Simon Riggs
On Fri, 2010-04-23 at 19:07 -0400, Robert Haas wrote: On Fri, Apr 23, 2010 at 6:39 PM, Simon Riggs si...@2ndquadrant.com wrote: On Fri, 2010-04-23 at 11:32 -0400, Robert Haas wrote: 99% of transactions happen in similar times between primary and standby, everything dragged down by rare

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-23 Thread Simon Riggs
On Thu, 2010-04-22 at 23:45 +0100, Simon Riggs wrote: On Thu, 2010-04-22 at 20:39 +0200, Erik Rijkers wrote: On Sun, April 18, 2010 13:01, Simon Riggs wrote: any comment is welcome... Please can you re-run with -l and post me the file of times Erik has sent me details of a test run. My

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-23 Thread Robert Haas
On Fri, Apr 23, 2010 at 11:14 AM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, 2010-04-22 at 23:45 +0100, Simon Riggs wrote: On Thu, 2010-04-22 at 20:39 +0200, Erik Rijkers wrote: On Sun, April 18, 2010 13:01, Simon Riggs wrote: any comment is welcome... Please can you re-run with -l

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-23 Thread Marko Kreen
On 4/18/10, Simon Riggs si...@2ndquadrant.com wrote: On Sat, 2010-04-17 at 16:48 -0400, Tom Lane wrote: There are some places where we suppose that a *single* write into shared memory can safely be done without a lock, if we're not too concerned about how soon other transactions will see

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-23 Thread Tom Lane
Marko Kreen mark...@gmail.com writes: Um, you have been burned by exactly this on x86 also: http://archives.postgresql.org/pgsql-hackers/2009-03/msg01265.php Yeah, we never did figure out exactly how come you were observing that failure on Intel-ish hardware. I was under the impression that

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-23 Thread Marko Kreen
On 4/23/10, Tom Lane t...@sss.pgh.pa.us wrote: Marko Kreen mark...@gmail.com writes: Um, you have been burned by exactly this on x86 also: http://archives.postgresql.org/pgsql-hackers/2009-03/msg01265.php Yeah, we never did figure out exactly how come you were observing that failure

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-23 Thread Simon Riggs
On Fri, 2010-04-23 at 11:32 -0400, Robert Haas wrote: 99% of transactions happen in similar times between primary and standby, everything dragged down by rare but severe spikes. We're looking for something that would delay something that normally takes 0.1ms into something that takes

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-23 Thread Robert Haas
On Fri, Apr 23, 2010 at 6:39 PM, Simon Riggs si...@2ndquadrant.com wrote: On Fri, 2010-04-23 at 11:32 -0400, Robert Haas wrote: 99% of transactions happen in similar times between primary and standby, everything dragged down by rare but severe spikes. We're looking for something that

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-23 Thread Erik Rijkers
On Sat, April 24, 2010 00:39, Simon Riggs wrote: On Fri, 2010-04-23 at 11:32 -0400, Robert Haas wrote: 99% of transactions happen in similar times between primary and standby, everything dragged down by rare but severe spikes. We're looking for something that would delay something that

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-22 Thread Simon Riggs
On Thu, 2010-04-22 at 08:57 +0300, Heikki Linnakangas wrote: I think the assert is a good idea. If there's no real problem here, the assert won't trip. It's just a safety precaution. Right. And assertions also act as documentation, they are a precise and compact way to document

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-22 Thread Erik Rijkers
On Sun, April 18, 2010 13:01, Simon Riggs wrote: OK, I'll put a spinlock around access to the head of the array. v2 patch attached knownassigned_sortedarray.v2.diff applied to cvs HEAD (2010.04.21 22:36) I have done a few smaller tests (scale 500, clients 1, 20): init: pgbench -h /tmp -p

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-22 Thread Greg Smith
Erik Rijkers wrote: This is the same behaviour (i.e. extreme slow standby) that I saw earlier (and which caused the original post, btw). In that earlier instance, the extreme slowness disappeared later, after many hours maybe even days (without bouncing either primary or standby). Any

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-22 Thread Mark Kirkwood
Greg Smith wrote: Erik Rijkers wrote: This is the same behaviour (i.e. extreme slow standby) that I saw earlier (and which caused the original post, btw). In that earlier instance, the extreme slowness disappeared later, after many hours maybe even days (without bouncing either primary or

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-22 Thread Erik Rijkers
On Thu, April 22, 2010 23:54, Mark Kirkwood wrote: Greg Smith wrote: Erik Rijkers wrote: This is the same behaviour (i.e. extreme slow standby) that I saw earlier (and which caused the original post, btw). In that earlier instance, the extreme slowness disappeared later, after many hours

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-22 Thread Simon Riggs
On Thu, 2010-04-22 at 20:39 +0200, Erik Rijkers wrote: On Sun, April 18, 2010 13:01, Simon Riggs wrote: any comment is welcome... Please can you re-run with -l and post me the file of times Please also rebuild using --enable-profile so we can see what's happening. Can you also try the

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-22 Thread Mark Kirkwood
Erik Rijkers wrote: This is the same behaviour (i.e. extreme slow standby) that I saw earlier (and which caused the original post, btw). In that earlier instance, the extreme slowness disappeared later, after many hours maybe even days (without bouncing either primary or standby). I have no

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-21 Thread Simon Riggs
On Wed, 2010-04-21 at 15:09 +1200, Mark Kirkwood wrote: I did some testing of this patch (v2). Unfortunately I don't have access to hardware capable of doing tests at the same scale as Erik used. However I was still able to show a consistent difference (I think) between standby performance

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-21 Thread Heikki Linnakangas
Simon Riggs wrote: On Sun, 2010-04-18 at 08:24 +0100, Simon Riggs wrote: On Sat, 2010-04-17 at 18:52 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: What I'm not clear on is why you've used a spinlock everywhere when only weak-memory thang CPUs are a problem. Why not have a

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-21 Thread Heikki Linnakangas
Simon Riggs wrote: On Sun, 2010-04-18 at 08:24 +0100, Simon Riggs wrote: On Sat, 2010-04-17 at 18:52 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: What I'm not clear on is why you've used a spinlock everywhere when only weak-memory thang CPUs are a problem. Why not have a

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-21 Thread Simon Riggs
On Wed, 2010-04-21 at 15:20 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: On Sun, 2010-04-18 at 08:24 +0100, Simon Riggs wrote: On Sat, 2010-04-17 at 18:52 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: What I'm not clear on is why you've used a spinlock everywhere

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-21 Thread Simon Riggs
On Wed, 2010-04-21 at 15:27 +0300, Heikki Linnakangas wrote: Given the discussion about the cyclic nature of XIDs, it would be good to add an assertion that when a new XID is added to the array, it is a) larger than the biggest value already in the array (TransactionIdFollows(new, head)),

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-21 Thread Robert Haas
On Wed, Apr 21, 2010 at 9:37 AM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, 2010-04-21 at 15:27 +0300, Heikki Linnakangas wrote: Given the discussion about the cyclic nature of XIDs, it would be good to add an assertion that when a new XID is added to the array, it is a) larger than

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-21 Thread Robert Haas
On Wed, Apr 21, 2010 at 8:20 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: The locking seems overly complex to me. I tend to agree. ! /* !* Callers must hold either ProcArrayLock in Exclusive mode or !* ProcArrayLock in Shared mode *and*

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-21 Thread Simon Riggs
On Wed, 2010-04-21 at 09:51 -0400, Robert Haas wrote: Adding an assertion isn't going to do much because it's unlikely anybody is going to be running for 2^31 transactions with asserts enabled. I think the assert is a good idea. If there's no real problem here, the assert won't trip.

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-21 Thread marcin mank
On Wed, Apr 21, 2010 at 4:12 PM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, 2010-04-21 at 09:51 -0400, Robert Haas wrote: Adding an assertion isn't going to do much because it's unlikely anybody is going to be running for 2^31 transactions with asserts enabled. I think the assert

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-21 Thread Simon Riggs
On Wed, 2010-04-21 at 16:22 +0200, marcin mank wrote: Is that not a good idea that (at least for dev-builds, like with enable-cassert) the xid counter start at like 2^31 - 1000 ? It could help catch some bugs. It is a good idea, I'm sure that would help catch bugs. It wouldn't help here

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-21 Thread Robert Haas
On Wed, Apr 21, 2010 at 10:12 AM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, 2010-04-21 at 09:51 -0400, Robert Haas wrote: Adding an assertion isn't going to do much because it's unlikely anybody is going to be running for 2^31 transactions with asserts enabled. I think the assert

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-21 Thread Florian Pflug
On Apr 21, 2010, at 16:49 , Simon Riggs wrote: On Wed, 2010-04-21 at 16:22 +0200, marcin mank wrote: Is that not a good idea that (at least for dev-builds, like with enable-cassert) the xid counter start at like 2^31 - 1000 ? It could help catch some bugs. It is a good idea, I'm sure that

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-21 Thread Heikki Linnakangas
Robert Haas wrote: On Wed, Apr 21, 2010 at 9:37 AM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, 2010-04-21 at 15:27 +0300, Heikki Linnakangas wrote: Given the discussion about the cyclic nature of XIDs, it would be good to add an assertion that when a new XID is added to the array, it is

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-20 Thread Robert Haas
On Sun, Apr 18, 2010 at 4:22 PM, Simon Riggs si...@2ndquadrant.com wrote: On Sun, 2010-04-18 at 13:16 -0700, David Fetter wrote: On Sun, Apr 18, 2010 at 12:01:05PM +0100, Simon Riggs wrote: On Sun, 2010-04-18 at 08:24 +0100, Simon Riggs wrote: On Sat, 2010-04-17 at 18:52 -0400, Tom Lane

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-20 Thread Mark Kirkwood
Robert Haas wrote: So, does anyone have a few cycles to test this out? We are down to handful of remaining open items, so getting this tested and committed sooner = beta sooner. I did some testing of this patch (v2). Unfortunately I don't have access to hardware capable of doing tests

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-20 Thread Robert Haas
On Tue, Apr 20, 2010 at 11:09 PM, Mark Kirkwood mark.kirkw...@catalyst.net.nz wrote: Robert Haas wrote: So, does anyone have a few cycles to test this out?  We are down to handful of remaining open items, so getting this tested and committed sooner = beta sooner. I did some testing of

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-18 Thread Simon Riggs
On Sat, 2010-04-17 at 18:52 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: What I'm not clear on is why you've used a spinlock everywhere when only weak-memory thang CPUs are a problem. Why not have a weak-memory-protect macro that does does nada when the hardware already

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-18 Thread Simon Riggs
On Sun, 2010-04-18 at 08:24 +0100, Simon Riggs wrote: On Sat, 2010-04-17 at 18:52 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: What I'm not clear on is why you've used a spinlock everywhere when only weak-memory thang CPUs are a problem. Why not have a

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-18 Thread David Fetter
On Sun, Apr 18, 2010 at 12:01:05PM +0100, Simon Riggs wrote: On Sun, 2010-04-18 at 08:24 +0100, Simon Riggs wrote: On Sat, 2010-04-17 at 18:52 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: What I'm not clear on is why you've used a spinlock everywhere when only

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-18 Thread Simon Riggs
On Sun, 2010-04-18 at 13:16 -0700, David Fetter wrote: On Sun, Apr 18, 2010 at 12:01:05PM +0100, Simon Riggs wrote: On Sun, 2010-04-18 at 08:24 +0100, Simon Riggs wrote: On Sat, 2010-04-17 at 18:52 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: What I'm not clear

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-18 Thread David Fetter
On Sun, Apr 18, 2010 at 09:22:21PM +0100, Simon Riggs wrote: On Sun, 2010-04-18 at 13:16 -0700, David Fetter wrote: On Sun, Apr 18, 2010 at 12:01:05PM +0100, Simon Riggs wrote: On Sun, 2010-04-18 at 08:24 +0100, Simon Riggs wrote: On Sat, 2010-04-17 at 18:52 -0400, Tom Lane wrote:

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-17 Thread Simon Riggs
On Fri, 2010-04-16 at 11:10 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Fri, 2010-04-16 at 10:39 -0400, Tom Lane wrote: I think you're outsmarting yourself there. A binary search will in fact *not work* with circular xid comparison (this is exactly why there's no

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-17 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: AFAICS the example you give isn't correct. We would lay out the values like this W-3 W-2 W-1 W 3 4 5 where W is the wrap value Stop right there, you're already failing to think clearly. There is no unique wrap value, all values act the same in

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-17 Thread Robert Haas
On Sat, Apr 17, 2010 at 11:13 AM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: AFAICS the example you give isn't correct. We would lay out the values like this W-3 W-2 W-1 W 3 4 5 where W is the wrap value Stop right there, you're already failing to think

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-17 Thread Simon Riggs
On Sat, 2010-04-17 at 11:13 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: AFAICS the example you give isn't correct. We would lay out the values like this W-3 W-2 W-1 W 3 4 5 where W is the wrap value Stop right there, you're already failing to think clearly.

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-17 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Sat, 2010-04-17 at 11:13 -0400, Tom Lane wrote: It'd be cheaper anyway to sort and search the array using plain , so why are you so eager to use TransactionIdFollows? The array grows to the right and is laid out one xid per element, resulting in

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-17 Thread Simon Riggs
On Sat, 2010-04-17 at 15:45 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Sat, 2010-04-17 at 11:13 -0400, Tom Lane wrote: It'd be cheaper anyway to sort and search the array using plain , so why are you so eager to use TransactionIdFollows? The array grows to the

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-17 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Sat, 2010-04-17 at 15:45 -0400, Tom Lane wrote: How do you know that just adding items at the right will produce a sorted array? Xids don't arrive in sequence, but known assigned xids are added in sequence because we infer the existence of the

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-17 Thread Simon Riggs
On Sat, 2010-04-17 at 16:48 -0400, Tom Lane wrote: We search the array between tail and head. If the head moves by integer overwrite just as already happens for xid assignment, then we would use the new head for the search. The code is careful to fetch only once. ... but this will not.

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-17 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: What I'm not clear on is why you've used a spinlock everywhere when only weak-memory thang CPUs are a problem. Why not have a weak-memory-protect macro that does does nada when the hardware already protects us? (i.e. a spinlock only for the hardware

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-16 Thread Heikki Linnakangas
Simon Riggs wrote: On Tue, 2010-04-13 at 21:09 +0300, Heikki Linnakangas wrote: A quick fix would be to check if there's any entries in the hash table before scanning it. That would eliminate the overhead when there's no in-progress transactions in the master. But as soon as there's even one,

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-16 Thread Simon Riggs
On Fri, 2010-04-16 at 11:29 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: On Tue, 2010-04-13 at 21:09 +0300, Heikki Linnakangas wrote: A quick fix would be to check if there's any entries in the hash table before scanning it. That would eliminate the overhead when there's no

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-16 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2010-04-16 at 11:29 +0300, Heikki Linnakangas wrote: How does the attached patch look like? It's probably similar to what you had in mind. It looks like a second version of what I'm working on and about to publish. I'll take that as a compliment! My patch is

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-16 Thread Simon Riggs
On Fri, 2010-04-16 at 14:47 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: On Fri, 2010-04-16 at 11:29 +0300, Heikki Linnakangas wrote: How does the attached patch look like? It's probably similar to what you had in mind. It looks like a second version of what I'm working on and

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-16 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: I didn't handle xid wraparound correctly in the binary search, need to use TransactionIdFollows instead of plan . I think you're outsmarting yourself there. A binary search will in fact *not work* with circular xid comparison (this

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-16 Thread Simon Riggs
On Fri, 2010-04-16 at 10:39 -0400, Tom Lane wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: I didn't handle xid wraparound correctly in the binary search, need to use TransactionIdFollows instead of plan . I think you're outsmarting yourself there. A binary search

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-16 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Fri, 2010-04-16 at 10:39 -0400, Tom Lane wrote: I think you're outsmarting yourself there. A binary search will in fact *not work* with circular xid comparison (this is exactly why there's no btree opclass for XID). I don't understand the exact,

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-16 Thread Simon Riggs
On Fri, 2010-04-16 at 13:00 +0100, Simon Riggs wrote: Almost done Here's the finished patch. Internal changes only, no functional or user visible changes, so no docs, just detailed explanatory comments. Expectation is * performance no longer depends upon max_connections * recovery will be

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-14 Thread Dimitri Fontaine
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Changing the KnownAssignedXids data structure from hash table into something that's quicker to scan. Preferably something with O(N), where N is the number of entries in the data structure, not the maximum number of entries it can

Re: [HACKERS] testing HS/SR - 1 vs 2 performance

2010-04-14 Thread Simon Riggs
On Tue, 2010-04-13 at 21:09 +0300, Heikki Linnakangas wrote: Heikki Linnakangas wrote: I could reproduce this on my laptop, standby is about 20% slower. I ran oprofile, and what stands out as the difference between the master and standby is that on standby about 20% of the CPU time is spent

  1   2   >