Re: [HACKERS] Lock partitions

2006-10-18 Thread Mark Wong
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: I see this in the CVS commits for 8.2. Did we determine the proper number of lock partitions? Should it be based on the number of buffers or concurrent sessions allowed? No. NUM_LOCK_PARTITIONS needs to be a compile-time constant for

Re: [HACKERS] Lock partitions

2006-10-18 Thread Tom Lane
Mark Wong [EMAIL PROTECTED] writes: The number of transaction errors increased when I increased the NUM_LOCK_PARTITIONS, which I think is the reason it failed to run when I set it to 16. Hmm, what sort of errors are we talking about? I wonder if you've exposed a bug. Changing

Re: [HACKERS] Lock partitions

2006-10-18 Thread Mark Wong
Tom Lane wrote: Mark Wong [EMAIL PROTECTED] writes: The number of transaction errors increased when I increased the NUM_LOCK_PARTITIONS, which I think is the reason it failed to run when I set it to 16. Hmm, what sort of errors are we talking about? I wonder if you've exposed a bug.

Re: [HACKERS] Lock partitions

2006-10-18 Thread Tom Lane
Mark Wong [EMAIL PROTECTED] writes: Tom Lane wrote: Hmm, what sort of errors are we talking about? ERROR: too many LWLocks taken That really shouldn't happen ... are you sure you did a full recompile after changing NUM_LOCK_PARTITIONS? Actually ... wait a moment. The default value of

Re: [HACKERS] Lock partitions

2006-10-18 Thread Mark Wong
Tom Lane wrote: Mark Wong [EMAIL PROTECTED] writes: Tom Lane wrote: Hmm, what sort of errors are we talking about? ERROR: too many LWLocks taken That really shouldn't happen ... are you sure you did a full recompile after changing NUM_LOCK_PARTITIONS? Actually ... wait a moment. The

Re: [HACKERS] Lock partitions

2006-09-20 Thread Mark Wong
Mark Wong wrote: Tom Lane wrote: Mark Wong [EMAIL PROTECTED] writes: Curious, I'm still seeing the same behavior. Maybe I'll take another snapshot from CVS. Hm, maybe I need to try a bit harder here. Does the not registered error happen immediately/reliably for you, or do you need to run

Re: [HACKERS] Lock partitions

2006-09-20 Thread Tom Lane
Mark Wong [EMAIL PROTECTED] writes: I did a gross test and my kit appears broken between the 8.0 and 8.1 releases. I'll try to narrow down the exact date. I've narrowed it down between cvs pulls from Dec 14, 2005 and Dec 15, 2005. Does the attached diff appear to be a plausible cause?

Re: [HACKERS] Lock partitions

2006-09-20 Thread Mark Wong
Tom Lane wrote: Mark Wong [EMAIL PROTECTED] writes: I did a gross test and my kit appears broken between the 8.0 and 8.1 releases. I'll try to narrow down the exact date. I've narrowed it down between cvs pulls from Dec 14, 2005 and Dec 15, 2005. Does the attached diff appear to be a

Re: [HACKERS] Lock partitions

2006-09-19 Thread Mark Wong
Tom Lane wrote: Mark Wong [EMAIL PROTECTED] writes: Curious, I'm still seeing the same behavior. Maybe I'll take another snapshot from CVS. Hm, maybe I need to try a bit harder here. Does the not registered error happen immediately/reliably for you, or do you need to run the test awhile?

Re: [HACKERS] Lock partitions

2006-09-14 Thread Strong, David
-development Subject: Re: [HACKERS] Lock partitions Strong, David [EMAIL PROTECTED] writes: We have some results for you. We left the buffer partition locks at 128 as this did not seem to be a concern and we're still using 25 backend processes. We ran tests for 4, 8 and 16 lock partitions. For 4 lock

Re: [HACKERS] Lock partitions

2006-09-14 Thread Mark Wong
Tom Lane wrote: Mark Wong [EMAIL PROTECTED] writes: Tom Lane wrote: It would be nice to see some results from the OSDL tests with, say, 4, 8, and 16 lock partitions before we forget about the point though. Anybody know whether OSDL is in a position to run tests for us? Yeah, I can run some

Re: [HACKERS] Lock partitions

2006-09-14 Thread Tom Lane
Mark Wong [EMAIL PROTECTED] writes: Sorry for the delay but looks like there's some data coming in. It also looks like my kit is starting to be a little dated. My stored libpq calls are failing. I'm getting this message: ERROR: record type has not been registered This is a server-side

Re: [HACKERS] Lock partitions

2006-09-14 Thread Mark Wong
Tom Lane wrote: Mark Wong [EMAIL PROTECTED] writes: Sorry for the delay but looks like there's some data coming in. It also looks like my kit is starting to be a little dated. My stored libpq calls are failing. I'm getting this message: ERROR: record type has not been registered This

Re: [HACKERS] Lock partitions

2006-09-14 Thread Tom Lane
Mark Wong [EMAIL PROTECTED] writes: Tom Lane wrote: This is a server-side failure --- could we see how order_status() is defined? What PG version are you testing exactly? I took pgsqsl snapshot from cvs on Sept 11. Due to the length of the file that order_status() is in and of

Re: [HACKERS] Lock partitions

2006-09-14 Thread Mark Wong
Tom Lane wrote: Mark Wong [EMAIL PROTECTED] writes: Tom Lane wrote: This is a server-side failure --- could we see how order_status() is defined? What PG version are you testing exactly? I took pgsqsl snapshot from cvs on Sept 11. Due to the length of the file that order_status() is in

Re: [HACKERS] Lock partitions

2006-09-14 Thread Tom Lane
Mark Wong [EMAIL PROTECTED] writes: But perhaps something much easier, using subversion: mkdir /mnt/dbt2 # for pgdata svn co https://svn.sourceforge.net/svnroot/osdldbt/trunk/dbt2 dbt2 cd dbt2 ./configure --with-postgresql=pgsql_dir configure is not in the svn checkout. I guessed that I

Re: [HACKERS] Lock partitions

2006-09-14 Thread Mark Wong
Tom Lane wrote: Mark Wong [EMAIL PROTECTED] writes: But perhaps something much easier, using subversion: mkdir /mnt/dbt2 # for pgdata svn co https://svn.sourceforge.net/svnroot/osdldbt/trunk/dbt2 dbt2 cd dbt2 ./configure --with-postgresql=pgsql_dir configure is not in the svn checkout. I

Re: [HACKERS] Lock partitions

2006-09-14 Thread Tom Lane
Mark Wong [EMAIL PROTECTED] writes: Oops! 'autoreconf --install' is what I run to generate all that stuff. Ah, better. I see at least part of the problem: CREATE OR REPLACE FUNCTION stock_level (INTEGER, INTEGER, INTEGER) RETURNS INTEGER AS

Re: [HACKERS] Lock partitions

2006-09-14 Thread Mark Wong
Tom Lane wrote: Mark Wong [EMAIL PROTECTED] writes: Oops! 'autoreconf --install' is what I run to generate all that stuff. Ah, better. I see at least part of the problem: CREATE OR REPLACE FUNCTION stock_level (INTEGER, INTEGER, INTEGER) RETURNS INTEGER AS

Re: [HACKERS] Lock partitions

2006-09-14 Thread Tom Lane
Mark Wong [EMAIL PROTECTED] writes: Tom Lane wrote: With that change, I didn't see run_workload report any errors, but maybe I don't know where to look. The error is captured in dbt2/scripts/output/*/client/error.log, where * is the run directory. Hm ... here's what I see in there: Thu

Re: [HACKERS] Lock partitions

2006-09-14 Thread Mark Wong
Tom Lane wrote: Mark Wong [EMAIL PROTECTED] writes: Tom Lane wrote: With that change, I didn't see run_workload report any errors, but maybe I don't know where to look. The error is captured in dbt2/scripts/output/*/client/error.log, where * is the run directory. Hm ... here's what I see

Re: [HACKERS] Lock partitions

2006-09-14 Thread Tom Lane
Mark Wong [EMAIL PROTECTED] writes: Curious, I'm still seeing the same behavior. Maybe I'll take another snapshot from CVS. Hm, maybe I need to try a bit harder here. Does the not registered error happen immediately/reliably for you, or do you need to run the test awhile? As for the

Re: [HACKERS] Lock partitions

2006-09-14 Thread markw
Mark Wong [EMAIL PROTECTED] writes: Curious, I'm still seeing the same behavior. Maybe I'll take another snapshot from CVS. Hm, maybe I need to try a bit harder here. Does the not registered error happen immediately/reliably for you, or do you need to run the test awhile? It appears to

Re: [HACKERS] Lock partitions

2006-09-13 Thread Simon Riggs
On Tue, 2006-09-12 at 12:40 -0400, Tom Lane wrote: Strong, David [EMAIL PROTECTED] writes: When using 16 buffer and 16 lock partitions, we see that BufMapping takes 809 seconds to acquire locks and 174 seconds to release locks. The LockMgr takes 362 seconds to acquire locks and 26 seconds

Re: [HACKERS] Lock partitions

2006-09-13 Thread Strong, David
Lane Cc: Strong, David; PostgreSQL-development Subject: Re: [HACKERS] Lock partitions On Tue, 2006-09-12 at 12:40 -0400, Tom Lane wrote: Strong, David [EMAIL PROTECTED] writes: When using 16 buffer and 16 lock partitions, we see that BufMapping takes 809 seconds to acquire locks and 174

Re: [HACKERS] Lock partitions

2006-09-13 Thread Strong, David
- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Strong, David Sent: Wednesday, September 13, 2006 10:52 AM To: PostgreSQL-development Subject: Re: [HACKERS] Lock partitions Simon, In the 16/16 (16 buffer partitions/16 lock partitions) test, the WALInsertLock lock had 14643080

Re: [HACKERS] Lock partitions

2006-09-13 Thread Tom Lane
Strong, David [EMAIL PROTECTED] writes: We have some results for you. We left the buffer partition locks at 128 as this did not seem to be a concern and we're still using 25 backend processes. We ran tests for 4, 8 and 16 lock partitions. For 4 lock partitions, it took 620 seconds to acquire

Re: [HACKERS] Lock partitions

2006-09-13 Thread Jim Nasby
On Sep 13, 2006, at 2:46 PM, Strong, David wrote: We have some results for you. We left the buffer partition locks at 128 as this did not seem to be a concern and we're still using 25 backend processes. We ran tests for 4, 8 and 16 lock partitions. Isn't having more lock partitions than

Re: [HACKERS] Lock partitions

2006-09-13 Thread Tom Lane
Jim Nasby [EMAIL PROTECTED] writes: Isn't having more lock partitions than buffer partitions pointless? AFAIK they're pretty orthogonal. It's true though that a typical transaction doesn't hold all that many locks, which is why I don't see a need for a large number of lock partitions.

Re: [HACKERS] Lock partitions

2006-09-12 Thread Simon Riggs
On Mon, 2006-09-11 at 11:29 -0400, Tom Lane wrote: Mark Wong [EMAIL PROTECTED] writes: Tom Lane wrote: It would be nice to see some results from the OSDL tests with, say, 4, 8, and 16 lock partitions before we forget about the point though. Anybody know whether OSDL is in a position to

Re: [HACKERS] Lock partitions

2006-09-12 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: On Mon, 2006-09-11 at 11:29 -0400, Tom Lane wrote: Great, thanks. The thing to twiddle is LOG2_NUM_LOCK_PARTITIONS in src/include/storage/lwlock.h. You need a full backend recompile after changing it, but you shouldn't need to initdb, if that helps.

Re: [HACKERS] Lock partitions

2006-09-12 Thread Strong, David
: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Simon Riggs Sent: Tuesday, September 12, 2006 1:37 AM To: Tom Lane Cc: Mark Wong; Bruce Momjian; PostgreSQL-development Subject: Re: [HACKERS] Lock partitions On Mon, 2006-09-11 at 11:29 -0400, Tom Lane wrote: Mark Wong [EMAIL PROTECTED

Re: [HACKERS] Lock partitions

2006-09-12 Thread Tom Lane
Strong, David [EMAIL PROTECTED] writes: When using 16 buffer and 16 lock partitions, we see that BufMapping takes 809 seconds to acquire locks and 174 seconds to release locks. The LockMgr takes 362 seconds to acquire locks and 26 seconds to release locks. When using 128 buffer and 128 lock

Re: [HACKERS] Lock partitions

2006-09-12 Thread Strong, David
to remove the SInvalLock lock to see where time flows to next, but we might. David -Original Message- From: Tom Lane [mailto:[EMAIL PROTECTED] Sent: Tuesday, September 12, 2006 9:40 AM To: Strong, David Cc: PostgreSQL-development Subject: Re: [HACKERS] Lock partitions Strong, David [EMAIL

Re: [HACKERS] Lock partitions

2006-09-11 Thread Tom Lane
Mark Wong [EMAIL PROTECTED] writes: Tom Lane wrote: It would be nice to see some results from the OSDL tests with, say, 4, 8, and 16 lock partitions before we forget about the point though. Anybody know whether OSDL is in a position to run tests for us? Yeah, I can run some dbt2 tests in the

Re: [HACKERS] Lock partitions

2006-09-11 Thread Mark Wong
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: I see this in the CVS commits for 8.2. Did we determine the proper number of lock partitions? Should it be based on the number of buffers or concurrent sessions allowed? No. NUM_LOCK_PARTITIONS needs to be a compile-time constant for

Re: [HACKERS] Lock partitions

2006-09-10 Thread Tom Lane
Bruce Momjian [EMAIL PROTECTED] writes: I see this in the CVS commits for 8.2. Did we determine the proper number of lock partitions? Should it be based on the number of buffers or concurrent sessions allowed? No. NUM_LOCK_PARTITIONS needs to be a compile-time constant for a number of