Re: [HACKERS] testing ProcArrayLock patches

2011-11-22 Thread Pavan Deolasee
On Tue, Nov 22, 2011 at 4:40 AM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Pavan Deolasee pavan.deola...@gmail.com wrote: It will be a great help if you could spare few minutes to also test the patch to take out the frequently accessed PGPROC members to a different array. We are

Re: [HACKERS] testing ProcArrayLock patches

2011-11-22 Thread Kevin Grittner
Pavan Deolasee pavan.deola...@gmail.com wrote: The numbers are not that bad, but definitely not as good as we saw on some other platforms. Well, this machine is definitely designed to hold up under high concurrency. As I understand it, each core is the memory manager for two 4GB DIMMs, with

Re: [HACKERS] testing ProcArrayLock patches

2011-11-21 Thread Kevin Grittner
Kevin Grittner kevin.gritt...@wicourts.gov wrote: I can run one more set of tests tonight before I have to give it back to the guy who's putting it into production. It sounds like a set like the above except with synchronous_commit = off might be desirable? OK, that's what I did. This

Re: [HACKERS] testing ProcArrayLock patches

2011-11-21 Thread Pavan Deolasee
On Mon, Nov 21, 2011 at 10:44 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Kevin Grittner kevin.gritt...@wicourts.gov wrote: I can run one more set of tests tonight before I have to give it back to the guy who's putting it into production.  It sounds like a set like the above except

Re: [HACKERS] testing ProcArrayLock patches

2011-11-21 Thread Kevin Grittner
Pavan Deolasee pavan.deola...@gmail.com wrote: It will be a great help if you could spare few minutes to also test the patch to take out the frequently accessed PGPROC members to a different array. We are seeing good improvements on HPUX IA platform and the AMD Opteron and it will be

Re: [HACKERS] testing ProcArrayLock patches

2011-11-21 Thread Pavan Deolasee
On Mon, Nov 21, 2011 at 11:01 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Pavan Deolasee pavan.deola...@gmail.com wrote: It will be a great help if you could spare few minutes to also test the patch to take out the frequently accessed PGPROC members to a different array. We are

Re: [HACKERS] testing ProcArrayLock patches

2011-11-21 Thread Kevin Grittner
Pavan Deolasee pavan.deola...@gmail.com wrote: It will be a great help if you could spare few minutes to also test the patch to take out the frequently accessed PGPROC members to a different array. We are seeing good improvements on HPUX IA platform and the AMD Opteron and it will be

Re: [HACKERS] testing ProcArrayLock patches

2011-11-20 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: Hmm. There's obviously something that's different in your environment or configuration from what I tested, but I don't know what it is. The fact that your scale factor is larger than shared_buffers might matter; or Intel vs. AMD. Or maybe you're

Re: [HACKERS] testing ProcArrayLock patches

2011-11-20 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: I was actually thinking it would be interesting to oprofile the read-only test; see if we can figure out where those slowdowns are coming from. CPU: Intel Core/i7, speed 2262 MHz (estimated) Counted CPU_CLK_UNHALTED events (Clock cycles when not

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: Nate Boley's AMD 6128 box (which has 32 cores) and an HP Integrity server (also with 32 cores). [clear improvement with flexlock patch] Hmm. We have a 32-core Intel box (4 x X7560 @ 2.27GHz) with 256 GB RAM. It's about a week from going into

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Robert Haas
On Fri, Nov 18, 2011 at 11:26 AM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Robert Haas robertmh...@gmail.com wrote: Nate Boley's AMD 6128 box (which has 32 cores) and an HP Integrity server (also with 32 cores). [clear improvement with flexlock patch] Hmm.  We have a 32-core Intel

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: Then again, is this a regular pgbench test or is this SELECT-only? SELECT-only Can you by any chance check top or vmstat during the 32-client test and see what percentage you have of user time/system time/idle time? You didn't say whether you

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Kevin Grittner
Kevin Grittner kevin.gritt...@wicourts.gov wrote: We have a 32-core Intel box (4 x X7560 @ 2.27GHz) with 256 GB RAM. In case anyone cares, this is the same box for which I posted STREAM test results a while back. The PostgreSQL tests seem to peak on this 32-core box at 64 clients, while the

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Robert Haas
On Fri, Nov 18, 2011 at 12:03 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Then again, is this a regular pgbench test or is this SELECT-only? SELECT-only Ah, OK. I would not expect flexlocks to help with that; Pavan's patch might, though. Can you by any chance check top or vmstat

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: Kevin Grittner kevin.gritt...@wicourts.gov wrote: Then again, is this a regular pgbench test or is this SELECT-only? SELECT-only Ah, OK. I would not expect flexlocks to help with that; Pavan's patch might, though. OK. Sorry for

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Robert Haas
On Fri, Nov 18, 2011 at 12:45 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: OK.  Sorry for misunderstanding that.  I haven't gotten around to a deep reading of the patch yet.  :-(  I based this on the test script you posted here (with slight modifications for my preferred directory

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: Yeah, I'd just drop -S. Easily done. Make sure to use -c N -j N with pgbench, or you'll probably not be able to saturate it. Yeah, that's part of the script I copied from you. I've also had good luck with wal_writer_delay=20ms, although if

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: Any chance you can run oprofile (on either branch, don't really care) against the 32 client test and post the results? Besides the other changes we discussed, I boosted scale to 150 and ran at READ COMMITTED isolation level (because all threads

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread anara...@anarazel.de
Kevin Grittner kevin.gritt...@wicourts.gov schrieb: Robert Haas robertmh...@gmail.com wrote: Any chance you can run oprofile (on either branch, don't really care) against the 32 client test and post the results? Besides the other changes we discussed, I boosted scale to 150 and ran at

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Kevin Grittner
anara...@anarazel.de and...@anarazel.de wrote: Kevin Grittner kevin.gritt...@wicourts.gov schrieb: samples %image name symbol name 9333944.9651 postgresAllocSetAlloc 8484764.5134 postgresbase_yyparse 7195153.8274 postgresSearchCatCache

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Kevin Grittner
Kevin Grittner kevin.gritt...@wicourts.gov wrote: samples %image name symbol name 4954633.6718 postgreshash_search_with_hash_value When lines like these show up in the annotated version, I'm impressed that we're still finding gains as big as we are: 44613 0.3306

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: I think so. My take was that it was showing 32 of 64 *threads* active -- the hyperthreading funkiness. Is there something in particular you'd like me to check? Not really, just don't understand the number. I'm having trouble resolving the vmstat

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Andres Freund
On Friday, November 18, 2011 08:36:59 PM Kevin Grittner wrote: Kevin Grittner kevin.gritt...@wicourts.gov wrote: samples %image name symbol name 4954633.6718 postgreshash_search_with_hash_value When lines like these show up in the annotated version, I'm

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Kevin Grittner
Andres Freund and...@anarazel.de wrote: When doing line-level profiles I would suggest looking at the instructions. What's the best way to do that? I don't think cache line contention is the most likely candidate here. Simple cache-misses seem far more likely. In combination with

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Andres Freund
On Friday, November 18, 2011 09:16:01 PM Kevin Grittner wrote: Andres Freund and...@anarazel.de wrote: When doing line-level profiles I would suggest looking at the instructions. What's the best way to do that? I think opannotate -a -s produces output with instructions/code intermingled.

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Andres Freund
On Friday, November 18, 2011 11:12:02 PM Andres Freund wrote: On Friday, November 18, 2011 09:16:01 PM Kevin Grittner wrote: Andres Freund and...@anarazel.de wrote: When doing line-level profiles I would suggest looking at the instructions. What's the best way to do that? I think

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Kevin Grittner
Andres Freund and...@anarazel.de wrote: I think opannotate -a -s produces output with instructions/code intermingled. Thanks. I'll check out perf later (thanks for the tips!), but for now, here's the function which was at the top of my oprofile results, annotated with those options. I'm

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Robert Haas
On Fri, Nov 18, 2011 at 2:05 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Robert Haas robertmh...@gmail.com wrote: Any chance you can run oprofile (on either branch, don't really care) against the 32 client test and post the results? [ oprofile results ] Hmm. That looks a lot like

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: Hmm. That looks a lot like a profile with no lock contention at all. Since I see XLogInsert in there, I assume this must be a pgbench write test on unlogged tables? How close am I? Not unless pgbench on HEAD does that by default. Here are the

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Robert Haas
On Fri, Nov 18, 2011 at 6:46 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: tps = 21946.961196 (including connections establishing) tps = 22911.873227 (including connections establishing) For write transactions, that seems pretty respectable. Very.  What do you get without the patch?

Re: [HACKERS] testing ProcArrayLock patches

2011-11-18 Thread Andres Freund
On Saturday, November 19, 2011 12:18:07 AM Kevin Grittner wrote: Andres Freund and...@anarazel.de wrote: I think opannotate -a -s produces output with instructions/code intermingled. Thanks. I'll check out perf later (thanks for the tips!), but for now, here's the function which was at