Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-13 Thread Gavin Hamill
Tom Lane wrote: Gavin Hamill [EMAIL PROTECTED] writes: would a simple #define LWLOCK_PADDED_SIZE 128 be sufficient? Yeah, that's fine. OK I tried that but noticed no real improvement... in the interim I've installed Debian on the pSeries (using

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-10 Thread Simon Riggs
On Fri, 2006-04-07 at 19:05 -0400, Tom Lane wrote: It's plausible though that we are seeing contention across members of the LWLock array, with the semop storm just being a higher-level symptom of the real hardware-level problem. You might try increasing LWLOCK_PADDED_SIZE to 64 or even 128,

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-10 Thread Gavin Hamill
Simon Riggs wrote: pSeries cache lines are 128 bytes wide, so I'd go straight to 128. Hello :) OK, that line of code is: #define LWLOCK_PADDED_SIZE (sizeof(LWLock) = 16 ? 16 : 32) What should I change this to? I don't understand the syntax of the = 16 ? : stuff... would a simple

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-10 Thread Tom Lane
Gavin Hamill [EMAIL PROTECTED] writes: would a simple #define LWLOCK_PADDED_SIZE 128 be sufficient? Yeah, that's fine. regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-10 Thread Brad Nicholson
Tom Lane wrote: This is unfortunately not going to help you as far as getting that machine into production now (unless you're brave enough to run CVS tip as production, which I certainly am not). I'm afraid you're most likely going to have to ship that pSeries back at the end of the month,

[PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Gavin Hamill
Bing-bong, passenger announcement.. the panic train is now pulling into platform 8.1.3. Bing-bong. =) OK, having moved from our quad-xeon to an 8-CPU IBM pSeries 650 (8x1.45GHz POWER4 instead of 4 x 3GHz Xeon), our query times have shot up and our website is next to unusable. The IBM is not

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Tom Lane
Gavin Hamill [EMAIL PROTECTED] writes: OK, having moved from our quad-xeon to an 8-CPU IBM pSeries 650 (8x1.45GHz POWER4 instead of 4 x 3GHz Xeon), our query times have shot up and our website is next to unusable. The IBM is not swapping (not with 16GB of RAM!), disk i/o is low, but there must

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Scott Marlowe
On Fri, 2006-04-07 at 12:58, Gavin Hamill wrote: Bing-bong, passenger announcement.. the panic train is now pulling into platform 8.1.3. Bing-bong. =) OK, having moved from our quad-xeon to an 8-CPU IBM pSeries 650 (8x1.45GHz POWER4 instead of 4 x 3GHz Xeon), our query times have shot up

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Gavin Hamill
On Fri, 07 Apr 2006 14:41:39 -0400 Tom Lane [EMAIL PROTECTED] wrote: Gavin Hamill [EMAIL PROTECTED] writes: OK, having moved from our quad-xeon to an 8-CPU IBM pSeries 650 (8x1.45GHz POWER4 instead of 4 x 3GHz Xeon), our query times have shot up and our website is next to unusable. The IBM

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Gavin Hamill
On Fri, 07 Apr 2006 13:54:21 -0500 Scott Marlowe [EMAIL PROTECTED] wrote: Are the same queries getting the same basic execution plan on both boxes? Turn on logging for slow queries, and explain analyze them on both machines to see if they are. See reply to Tom Lane :) I'd put the old 4 way

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Tom Lane
Gavin Hamill [EMAIL PROTECTED] writes: Scott Marlowe [EMAIL PROTECTED] wrote: My guess is that this is an OS issue. Maybe there are AIX tweaks that will get it up to the same or higher level of performance as your four way xeon. Maybe there aren't. The pSeries isn't much older than our

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread D'Arcy J.M. Cain
On Fri, 7 Apr 2006 20:59:19 +0100 Gavin Hamill [EMAIL PROTECTED] wrote: I'd put the old 4 way Xeon back in production and do some serious testing of this pSeries machine. IBM should be willing to help you, I hope. They probably would if this had been bought new - as it is, we have

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Scott Marlowe
On Fri, 2006-04-07 at 14:59, Gavin Hamill wrote: On Fri, 07 Apr 2006 13:54:21 -0500 Scott Marlowe [EMAIL PROTECTED] wrote: Are the same queries getting the same basic execution plan on both boxes? Turn on logging for slow queries, and explain analyze them on both machines to see if they

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Greg Stark
Gavin Hamill [EMAIL PROTECTED] writes: This is one hell of a moving target and I can't help but think I'm just missing something that's right in front of my nose, too close to see. I'm assuming you compiled postgres yourself? Do you have the output from the configure script? I'm wondering if

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Gábriel Ákos
Gavin Hamill wrote: Bing-bong, passenger announcement.. the panic train is now pulling into platform 8.1.3. Bing-bong. =) OK, having moved from our quad-xeon to an 8-CPU IBM pSeries 650 (8x1.45GHz POWER4 instead of 4 x 3GHz Xeon), our query times have shot up and our website is next to

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Gavin Hamill
On Fri, 7 Apr 2006 16:16:02 -0400 D'Arcy J.M. Cain darcy@druid.net wrote: We also had problems with a high end AIX system and we got no help from IBM. They expected you to put Oracle on and if you used anything else you were on your own. Urk, I thought IBM were supposedly Linux sycophants

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Gavin Hamill
On Fri, 07 Apr 2006 15:24:18 -0500 Scott Marlowe [EMAIL PROTECTED] wrote: See reply to Tom Lane :) I didn't see one go by yet... Could be sitting in the queue. If it's not arrived by now - EXPLAIN ANALYZE doesn't tell me anything :) Let us know if changing the fsync setting helps.

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Gavin Hamill
On Fri, 07 Apr 2006 16:06:02 -0400 Tom Lane [EMAIL PROTECTED] wrote: The pSeries isn't much older than our Xeon machine, and I expected the performance level to be exemplary out of the box.. I'm fairly surprised too. One thing I note from your comparison of settings is that the default

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Luke Lonergan
Title: Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow. Gavin, On 4/7/06 2:24 PM, Gavin Hamill [EMAIL PROTECTED] wrote: I did look into the specs of the system, and the memory bw on the pSeries was /much/ greater than the Xeon - it's one of the things that really pushed me towards

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Tom Lane
Gavin Hamill [EMAIL PROTECTED] writes: There's truss installed which seems to do the same as strace on Linux... and here's a wildly non-scientific glance.. I watched the 'topas' output (top for AIX) , identified a PID that was doing a lot of work, then attached truss to that pid. In addition

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Tom Lane
Luke Lonergan [EMAIL PROTECTED] writes: That said, I find typical memory bandwidth for the P4 in applications is limited at about 2GB/s. See here for more detail: http://www.cs.virginia.edu/stream/standard/Bandwidth.html In fact, looking at the results there, the IBM 650m2 only gets 6GB/s

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Luke Lonergan
Tom, On 4/7/06 3:02 PM, Tom Lane [EMAIL PROTECTED] wrote: On the other hand, we already know that Xeons suck about as badly as can be on that same measure; could the pSeries really be worse? I wouldn't be too surprised, but it sounds like it needs a test. Do we have a test for this? Is

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Tom Lane
Luke Lonergan [EMAIL PROTECTED] writes: On 4/7/06 3:02 PM, Tom Lane [EMAIL PROTECTED] wrote: On the other hand, we already know that Xeons suck about as badly as can be on that same measure; could the pSeries really be worse? I wouldn't be too surprised, but it sounds like it needs a test.

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Gavin Hamill
On Fri, 07 Apr 2006 17:56:49 -0400 Tom Lane [EMAIL PROTECTED] wrote: This is not good. Did the semop storms coincide with visible slowdown? (I'd assume so, but you didn't actually say...) If I'd been able to tell, then I'd tell you =) I'll have another go... Yes, there's a definate

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Tom Lane
Gavin Hamill [EMAIL PROTECTED] writes: On Fri, 07 Apr 2006 17:56:49 -0400 Tom Lane [EMAIL PROTECTED] wrote: This is not good. Did the semop storms coincide with visible slowdown? (I'd assume so, but you didn't actually say...) Yes, there's a definate correlation here.. I attached truss to

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Luke Lonergan
Gavin, On 4/7/06 3:27 PM, Gavin Hamill [EMAIL PROTECTED] wrote: 278774: __semop(15728650, 0x0FFF7E80, 1)= 0 155712: __semop(15728650, 0x0FFF5920, 1)= 0 278774: __semop(15728649, 0x0FFF6F10, 1) 114914: __semop(15728649, 0x0FFF6A40, 1)= 0

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Tom Lane
Luke Lonergan [EMAIL PROTECTED] writes: On 4/7/06 3:27 PM, Gavin Hamill [EMAIL PROTECTED] wrote: 278774: __semop(15728650, 0x0FFF7E80, 1)= 0 155712: __semop(15728650, 0x0FFF5920, 1)= 0 278774: __semop(15728649, 0x0FFF6F10, 1) Seems like you're hitting

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Gavin Hamill
On Fri, 07 Apr 2006 18:52:20 -0400 Tom Lane [EMAIL PROTECTED] wrote: Are you in a position to try your workload using PG CVS tip? There's a nontrivial possibility that we've already fixed this --- a couple months ago I did some work to reduce contention in the lock manager: Well, there's a

Re: [PERFORM] pg 8.1.3, AIX, huge box, painfully slow.

2006-04-07 Thread Gavin Hamill
On Fri, 07 Apr 2006 15:56:52 -0700 Luke Lonergan [EMAIL PROTECTED] wrote: Seems like you're hitting a very small target in RAM with these semop calls. I wonder what part of the code is doing this - Tom would know better how to trace it, but the equivalent of oprofile output would be nice.