Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-09 Thread Simon Riggs
On Wed, Jun 8, 2011 at 6:43 PM, Joshua Berkus j...@agliodbs.com wrote: Simon, The point I have made is that I disagree with a feature freeze date fixed ahead of time without regard to the content of the forthcoming release. I've not said I disagree with feature freezes altogether, which

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-09 Thread Robert Haas
On Thu, Jun 9, 2011 at 5:09 AM, Simon Riggs si...@2ndquadrant.com wrote: I have asked that we maintain the Reasonableness we have always had about how the feature freeze date was applied. An example of such reasonableness is that if a feature is a few days late and it is important, then it

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-09 Thread Dave Page
On Thu, Jun 9, 2011 at 2:13 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, Jun 9, 2011 at 5:09 AM, Simon Riggs si...@2ndquadrant.com wrote: I have asked that we maintain the Reasonableness we have always had about how the feature freeze date was applied. An example of such reasonableness

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-09 Thread Pavan Deolasee
Can we make this the last post on this topic please? +1 :) Thanks, Pavan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-08 Thread Jim Nasby
On Jun 7, 2011, at 8:24 AM, Stephen Frost wrote: * Alvaro Herrera (alvhe...@commandprompt.com) wrote: I note that if 2nd Quadrant is interested in having a game-changing platform without having to wait a full year for 9.2, they can obviously distribute a modified version of Postgres that

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-08 Thread Simon Riggs
On Wed, Jun 8, 2011 at 5:19 AM, Bruce Momjian br...@momjian.us wrote: Robert Haas wrote: On Mon, Jun 6, 2011 at 10:49 AM, Simon Riggs si...@2ndquadrant.com wrote: My point was that we have in the past implemented performance changes to increase scalability at the last minute, and also that

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-08 Thread Robert Haas
On Wed, Jun 8, 2011 at 11:39 AM, Jim Nasby j...@nasby.net wrote: On Jun 7, 2011, at 8:24 AM, Stephen Frost wrote: * Alvaro Herrera (alvhe...@commandprompt.com) wrote: I note that if 2nd Quadrant is interested in having a game-changing platform without having to wait a full year for 9.2, they

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-08 Thread Simon Riggs
On Wed, Jun 8, 2011 at 6:02 AM, Tom Lane t...@sss.pgh.pa.us wrote: Bruce Momjian br...@momjian.us writes: Simon is right that we slipped the vxid patch into 8.3 when a Postgres user I talked to at Linuxworld mentioned high vacuum freeze activity and simple calculations showed the many

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-08 Thread Simon Riggs
On Wed, Jun 8, 2011 at 5:33 AM, Bruce Momjian br...@momjian.us wrote: One more thing --- when Tom applied that patch during 8.3 beta it was with everyone's agreement, so the policy should be that if we are going to break the rules, everyone has to agree --- if anyone disagrees, the rules

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-08 Thread Robert Haas
On Wed, Jun 8, 2011 at 12:25 PM, Simon Riggs si...@2ndquadrant.com wrote: As a result of this, I've been insulted, told I have no respect for process and even suggested there was a threat of patch war. Well, you've pretty much said flat out you don't like the process, and you don't agree with

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-08 Thread Simon Riggs
On Wed, Jun 8, 2011 at 5:32 PM, Robert Haas robertmh...@gmail.com wrote: It took me about 3 days to write the patch.  I've now spent the better part of a day on this scheduling discussion.  I would rather have spent that time improving the patch.  Or working on some other patch.  Or getting

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-08 Thread Simon Riggs
On Wed, Jun 8, 2011 at 5:44 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Jun 8, 2011 at 12:25 PM, Simon Riggs si...@2ndquadrant.com wrote: As a result of this, I've been insulted, told I have no respect for process and even suggested there was a threat of patch war. Well, you've

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-08 Thread Joshua Berkus
Simon, The point I have made is that I disagree with a feature freeze date fixed ahead of time without regard to the content of the forthcoming release. I've not said I disagree with feature freezes altogether, which would be utterly ridiculous. Fixed dates are IMHO much less important than

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-08 Thread Joshua D. Drake
On 06/07/2011 11:55 AM, Tom Lane wrote: Simon Riggssi...@2ndquadrant.com writes: Before you arrived, it was quite normal to suggest tuning patches after feature freeze. *Low risk* tuning patches make sense at this stage, yes. Fooling with the lock mechanisms doesn't qualify as low risk in

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-08 Thread Robert Haas
On Wed, Jun 8, 2011 at 1:10 PM, Simon Riggs si...@2ndquadrant.com wrote: Why do you address this to me? Many others have been committing patches against raised issues well after feature freeze. No one other than you has proposed committing anything nearly as invasive as this, and the great

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-08 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Wed, Jun 8, 2011 at 6:02 AM, Tom Lane t...@sss.pgh.pa.us wrote: Just to set the record straight on this ... the vxid patch went in on 2007-09-05: http://archives.postgresql.org/pgsql-committers/2007-09/msg00026.php which was a day shy of a month

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-08 Thread Tom Lane
Joshua Berkus j...@agliodbs.com writes: Simon, The point I have made is that I disagree with a feature freeze date fixed ahead of time without regard to the content of the forthcoming release. I've not said I disagree with feature freezes altogether, which would be utterly ridiculous. Fixed

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Dave Page
On Tue, Jun 7, 2011 at 12:29 AM, Tom Lane t...@sss.pgh.pa.us wrote: Dave Page dp...@pgadmin.org writes: On Mon, Jun 6, 2011 at 8:44 PM, Stephen Frost sfr...@snowman.net wrote: If we're going to start putting in changes like this, I'd suggest that we try and target something like September for

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Stephen Frost
* Alvaro Herrera (alvhe...@commandprompt.com) wrote: I note that if 2nd Quadrant is interested in having a game-changing platform without having to wait a full year for 9.2, they can obviously distribute a modified version of Postgres that integrates Robert's patch. Having thought about this,

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Joshua D. Drake
On 06/06/2011 04:43 PM, Robert Haas wrote: On Mon, Jun 6, 2011 at 6:53 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of vie jun 03 09:17:08 -0400 2011: I've now spent enough time working on this issue now to be convinced that the approach has merit,

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Simon Riggs
On Mon, Jun 6, 2011 at 8:50 PM, Dave Page dp...@pgadmin.org wrote: On Mon, Jun 6, 2011 at 8:40 PM, Stefan Kaltenbrunner ste...@kaltenbrunner.cc wrote: On 06/06/2011 09:24 PM, Dave Page wrote: On Mon, Jun 6, 2011 at 8:12 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: So, to the question

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Robert Haas
On Tue, Jun 7, 2011 at 12:51 PM, Simon Riggs si...@2ndquadrant.com wrote: On Mon, Jun 6, 2011 at 8:50 PM, Dave Page dp...@pgadmin.org wrote: On Mon, Jun 6, 2011 at 8:40 PM, Stefan Kaltenbrunner ste...@kaltenbrunner.cc wrote: On 06/06/2011 09:24 PM, Dave Page wrote: On Mon, Jun 6, 2011 at 8:12

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Robert Haas
On Tue, Jun 7, 2011 at 11:56 AM, Joshua D. Drake j...@commandprompt.com wrote: On 06/06/2011 04:43 PM, Robert Haas wrote: On Mon, Jun 6, 2011 at 6:53 PM, Alvaro Herrera alvhe...@commandprompt.com  wrote: Excerpts from Robert Haas's message of vie jun 03 09:17:08 -0400 2011: I've now spent

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: Please consider this as a serious proposal for tuning in 9.1. Look: it is at least four months too late for anything of the sort in 9.1. We should be fixing bugs, and nothing else, if we ever want to get 9.1 out the door. Performance improvements don't

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Joshua Berkus
iew. The reason we usually skip the summer isn't actually a wholesale lack of people - it's because it's not so good from a publicity perspective, and it's hard to get all the packagers around at the same time. Actually, the summer is *excellent* from a publicity perspective ... at least,

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: ... I think at the next developer meeting we're going to get to hear Tom argue that overlapping the end of beta with the beginning of the next release cycle is a mistake and we should go back to the old system where we yell at everyone to shut up

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Joshua Berkus
Robert, Oh, I get that. I'm just dismayed that we can't have a discussion about the patch without getting sidetracked into a conversation about whether we should throw feature freeze out the window. That's not something you can change. Whatever the patch is, even if it's a psql

9.1 release scheduling (was Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch)

2011-06-07 Thread Tom Lane
Joshua Berkus j...@agliodbs.com writes: Actually, the summer is *excellent* from a publicity perspective ... at least, June and July are. Both of those months are full of US conferences whose PR we can piggyback on to make a splash. August is really the only bad month from a PR

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Robert Haas
On Tue, Jun 7, 2011 at 1:27 PM, Joshua Berkus j...@agliodbs.com wrote: As long as we have solidarity of the committers that this is not allowed, however, this is not a real problem.  And it appears that we do.  In the future, it shouldn't even be necessary to discuss it. Solidarity? Simon -

Re: 9.1 release scheduling (was Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch)

2011-06-07 Thread Thom Brown
On 7 June 2011 19:32, Tom Lane t...@sss.pgh.pa.us wrote: Joshua Berkus j...@agliodbs.com writes: Actually, the summer is *excellent* from a publicity perspective ... at least, June and July are.  Both of those months are full of US conferences whose PR we can piggyback on to make a splash.

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Robert Haas
On Tue, Jun 7, 2011 at 1:21 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: ... I think at the next developer meeting we're going to get to hear Tom argue that overlapping the end of beta with the beginning of the next release cycle is a mistake and we should

Re: 9.1 release scheduling (was Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch)

2011-06-07 Thread Robert Haas
On Tue, Jun 7, 2011 at 1:45 PM, Thom Brown t...@linux.com wrote: Speaking of which, is it now safe to remove the NOT VALID constraints don't dump properly issue from the blocker list since the fix has been committed? I hope so, because I just did that (before noticing this email from you). --

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Tue, Jun 7, 2011 at 1:27 PM, Joshua Berkus j...@agliodbs.com wrote: As long as we have solidarity of the committers that this is not allowed, however, this is not a real problem.  And it appears that we do.  In the future, it shouldn't even be

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Simon Riggs
On Tue, Jun 7, 2011 at 6:33 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Jun 7, 2011 at 1:27 PM, Joshua Berkus j...@agliodbs.com wrote: As long as we have solidarity of the committers that this is not allowed, however, this is not a real problem.  And it appears that we do.  In the

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Stephen Frost
* Simon Riggs (si...@2ndquadrant.com) wrote: Before you arrived, it was quite normal to suggest tuning patches after feature freeze. I haven't been around as long as some, but I think I've been around longer than Robert, and I can say that I don't recall serious performance patches,

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Kevin Grittner
Simon Riggs si...@2ndquadrant.com wrote: Before you arrived, it was quite normal to suggest tuning patches after feature freeze. I've worn a lot of hats in the practical end of this industry, and regardless of which perspective I look at this from, I can't think of anything so destructive to

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Robert Haas
On Tue, Jun 7, 2011 at 2:06 PM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, Jun 7, 2011 at 6:33 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Jun 7, 2011 at 1:27 PM, Joshua Berkus j...@agliodbs.com wrote: As long as we have solidarity of the committers that this is not allowed,

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: Before you arrived, it was quite normal to suggest tuning patches after feature freeze. *Low risk* tuning patches make sense at this stage, yes. Fooling with the lock mechanisms doesn't qualify as low risk in my book. The probability of undetected

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Jignesh Shah
On Mon, Jun 6, 2011 at 11:20 PM, Jignesh Shah jks...@gmail.com wrote: Okay I tried it out with sysbench read scaling test.. Note I had tried that earlier on 9.0 http://jkshah.blogspot.com/2010/11/postgresql-90-simple-select-scaling.html And on that test I found that doing that test on

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I plead guilty to taking my eye off the ball post-beta1. I busted my ass for two months stabilizing other people's code after CF4 was over, and then I moved on to other things. I will try to get my eye back on the ball - but actually I'm not sure

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Simon Riggs
On Tue, Jun 7, 2011 at 7:55 PM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: Before you arrived, it was quite normal to suggest tuning patches after feature freeze. *Low risk* tuning patches make sense at this stage, yes.  Fooling with the lock mechanisms

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: Moving on from that, I have proposed other solutions. Koichi, Jignesh and and then Robert have shown measurements of the huge contention in this area of our software. Robert's patch addresses the problems, as do Koichi's and my latest patch. I would

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Robert Haas
On Tue, Jun 7, 2011 at 3:44 PM, Jignesh Shah jks...@gmail.com wrote: On Mon, Jun 6, 2011 at 11:20 PM, Jignesh Shah jks...@gmail.com wrote: Okay I tried it out with sysbench read scaling test.. Note I had tried that earlier on 9.0

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Simon Riggs
On Tue, Jun 7, 2011 at 9:00 PM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: Moving on from that, I have proposed other solutions. Koichi, Jignesh and and then Robert have shown measurements of the huge contention in this area of our software. Robert's patch

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Robert Haas
On Tue, Jun 7, 2011 at 12:51 PM, Simon Riggs si...@2ndquadrant.com wrote: Stefan/Robert's observation that we perform a VirtualXactLockTableInsert() to no real benefit is a good one. It leads to the following simple patch to remove one lock table hit per transaction. It's a lot smaller impact

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Simon Riggs
On Tue, Jun 7, 2011 at 9:52 PM, Robert Haas robertmh...@gmail.com wrote: If we were to fix ONLY the vxid issue in 9.1 as you were advocating Sensible debate is impossible when you don't read what I've written. --  Simon Riggs   http://www.2ndQuadrant.com/  PostgreSQL

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Robert Haas
On Tue, Jun 7, 2011 at 5:43 PM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, Jun 7, 2011 at 9:52 PM, Robert Haas robertmh...@gmail.com wrote: If we were to fix ONLY the vxid issue in 9.1 as you were advocating Sensible debate is impossible when you don't read what I've written. I've read

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Josh Berkus
On 6/7/11 1:11 PM, Simon Riggs wrote: that appear low risk. I seriously doubt that I would consider *any* meaningful change in the locking area to be low risk. That's a shame. We'll fix it in 9.2 then. I will point out that we bounced Alvaro's FK patch, which *was* submitted in time for CF4,

Re: 9.1 release scheduling (was Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch)

2011-06-07 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mar jun 07 13:53:23 -0400 2011: On Tue, Jun 7, 2011 at 1:45 PM, Thom Brown t...@linux.com wrote: Speaking of which, is it now safe to remove the NOT VALID constraints don't dump properly issue from the blocker list since the fix has been committed?

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Bruce Momjian
Robert Haas wrote: On Mon, Jun 6, 2011 at 10:49 AM, Simon Riggs si...@2ndquadrant.com wrote: My point was that we have in the past implemented performance changes to increase scalability at the last minute, and also that our personal risk perspectives are not always set in stone. Robert

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Bruce Momjian
Bruce Momjian wrote: Simon is right that we slipped the vxid patch into 8.3 when a Postgres user I talked to at Linuxworld mentioned high vacuum freeze activity and simple calculations showed the many read-only queries could cause high xid usage. Fortunately we already had a patch available

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-07 Thread Tom Lane
Bruce Momjian br...@momjian.us writes: Simon is right that we slipped the vxid patch into 8.3 when a Postgres user I talked to at Linuxworld mentioned high vacuum freeze activity and simple calculations showed the many read-only queries could cause high xid usage. Fortunately we already had a

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Heikki Linnakangas
On 06.06.2011 07:12, Robert Haas wrote: I did some further investigation of this. It appears that more than 99% of the lock manager lwlock traffic that remains with this patch applied has locktag_type == LOCKTAG_VIRTUALTRANSACTION. Every SELECT statement runs in a separate transaction, and for

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Simon Riggs
On Sat, Jun 4, 2011 at 5:55 PM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: The approach looks sound to me. It's a fairly isolated patch and we should be considering this for inclusion in 9.1, not wait another year. That suggestion is completely insane.  The

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Heikki Linnakangas
On 06.06.2011 12:40, Simon Riggs wrote: On Sat, Jun 4, 2011 at 5:55 PM, Tom Lanet...@sss.pgh.pa.us wrote: Simon Riggssi...@2ndquadrant.com writes: The approach looks sound to me. It's a fairly isolated patch and we should be considering this for inclusion in 9.1, not wait another year.

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Robert Haas
On Mon, Jun 6, 2011 at 2:54 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Ah, I remember I saw that vxid lock pop up quite high in an oprofile profile recently. I think it was the case of executing a lot of very simple prepared queries. So it would be nice to address that,

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Heikki Linnakangas
On 06.06.2011 07:12, Robert Haas wrote: I did some further investigation of this. It appears that more than 99% of the lock manager lwlock traffic that remains with this patch applied has locktag_type == LOCKTAG_VIRTUALTRANSACTION. Every SELECT statement runs in a separate transaction, and for

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Robert Haas
On Mon, Jun 6, 2011 at 8:02 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 06.06.2011 07:12, Robert Haas wrote: I did some further investigation of this.  It appears that more than 99% of the lock manager lwlock traffic that remains with this patch applied has

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Heikki Linnakangas
On 06.06.2011 14:59, Robert Haas wrote: BTW, how do you identify from oprofile that *vxid* locks were the problem? I didn't think it could produce that level of detail. It can show the call stack of each call, with --callgraph=n option, where you can see what percentage of the calls to

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Simon Riggs
On Mon, Jun 6, 2011 at 11:19 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 06.06.2011 12:40, Simon Riggs wrote: On Sat, Jun 4, 2011 at 5:55 PM, Tom Lanet...@sss.pgh.pa.us  wrote: Simon Riggssi...@2ndquadrant.com  writes: The approach looks sound to me. It's a fairly

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Robert Haas
On Mon, Jun 6, 2011 at 10:49 AM, Simon Riggs si...@2ndquadrant.com wrote: My point was that we have in the past implemented performance changes to increase scalability at the last minute, and also that our personal risk perspectives are not always set in stone. Robert has highlighted the

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: IMHO, it's better to just have a deadline, and stuff either makes it or it doesn't. I realize we haven't always adhered to the principle in the past, but at least IMV that's not a mistake we want to continue repeating. +1 I've said it before, but

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Simon Riggs
On Mon, Jun 6, 2011 at 5:14 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Perhaps the best way to describe the suggestion that this be included in 9.1 isn't that it's an insane suggestion; but that it's a suggestion which, if adopted, would be likely to drive those who are striving

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Josh Berkus
That's an improvement of about ~3.5x. According to the vmstat output, when running without the patch, the CPU state was about 40% idle. With the patch, it dropped down to around 6%. Wow! That's fantastic. Jignesh, are you in a position to test any of Robert's work using DBT or other

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Dimitri Fontaine
Robert Haas robertmh...@gmail.com writes: IMHO, it's better to just have a deadline, Well, that's the fine point we're now talking about. I still think that we should try at making the best release possible. And if that means including changes at beta time because that's when someone got

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Dave Page
On Mon, Jun 6, 2011 at 8:12 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: So, to the question “do we want hard deadlines?” I think the answer is “no”, to “do we need hard deadlines?”, my answer is still “no”, and to the question “does this very change should be considered this late?” my

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Stefan Kaltenbrunner
On 06/06/2011 09:24 PM, Dave Page wrote: On Mon, Jun 6, 2011 at 8:12 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: So, to the question “do we want hard deadlines?” I think the answer is “no”, to “do we need hard deadlines?”, my answer is still “no”, and to the question “does this very

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Stephen Frost
* Dave Page (dp...@pgadmin.org) wrote: Much as I hate to say it (I too want to keep our schedule as predictable and organised as possible), I have to agree. Assuming the patch is good, I think this is something we should push into 9.1. It really could be a game changer. So, with folks putting

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Dave Page
On Mon, Jun 6, 2011 at 8:40 PM, Stefan Kaltenbrunner ste...@kaltenbrunner.cc wrote: On 06/06/2011 09:24 PM, Dave Page wrote: On Mon, Jun 6, 2011 at 8:12 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: So, to the question “do we want hard deadlines?” I think the answer is “no”, to “do we

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Josh Berkus
On 6/6/11 12:12 PM, Dimitri Fontaine wrote: So, to the question “do we want hard deadlines?” I think the answer is “no”, to “do we need hard deadlines?”, my answer is still “no”, and to the question “does this very change should be considered this late?” my answer is yes. I could not disagree

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Andrew Dunstan
On 06/06/2011 03:24 PM, Dave Page wrote: On Mon, Jun 6, 2011 at 8:12 PM, Dimitri Fontainedimi...@2ndquadrant.fr wrote: So, to the question “do we want hard deadlines?” I think the answer is “no”, to “do we need hard deadlines?”, my answer is still “no”, and to the question “does this very

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Dave Page
On Mon, Jun 6, 2011 at 8:44 PM, Stephen Frost sfr...@snowman.net wrote: * Dave Page (dp...@pgadmin.org) wrote: Much as I hate to say it (I too want to keep our schedule as predictable and organised as possible), I have to agree. Assuming the patch is good, I think this is something we should

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Jignesh Shah
On Mon, Jun 6, 2011 at 2:49 PM, Josh Berkus j...@agliodbs.com wrote: That's an improvement of about ~3.5x.  According to the vmstat output, when running without the patch, the CPU state was about 40% idle. With the patch, it dropped down to around 6%. Wow!  That's fantastic. Jignesh, are

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Christopher Browne
On Mon, Jun 6, 2011 at 5:13 PM, Simon Riggs si...@2ndquadrant.com wrote: The cost to us is a few days work and the benefit is a whole year's worth of increased performance for our user base, which has a hardware equivalent well into the millions of dollars. I doubt that this is an accurate

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Robert Haas
On Mon, Jun 6, 2011 at 3:59 PM, Christopher Browne cbbro...@gmail.com wrote: On Mon, Jun 6, 2011 at 5:13 PM, Simon Riggs si...@2ndquadrant.com wrote: The cost to us is a few days work and the benefit is a whole year's worth of increased performance for our user base, which has a hardware

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Kevin Grittner
Stephen Frost sfr...@snowman.net wrote: if people really want this in, then we need to figure out what the new schedule is going to be. I suggest June, 2012. That way we can get a whole bunch more really cool patches in, and the users won't have to wait for 9.2 to get them. -Kevin --

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Simon Riggs
On Mon, Jun 6, 2011 at 8:52 PM, Dave Page dp...@pgadmin.org wrote: On Mon, Jun 6, 2011 at 8:44 PM, Stephen Frost sfr...@snowman.net wrote: * Dave Page (dp...@pgadmin.org) wrote: Much as I hate to say it (I too want to keep our schedule as predictable and organised as possible), I have to

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Alvaro Herrera
Excerpts from Dimitri Fontaine's message of lun jun 06 15:12:54 -0400 2011: So, to the question “do we want hard deadlines?” I think the answer is “no”, to “do we need hard deadlines?”, my answer is still “no”, and to the question “does this very change should be considered this late?” my

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Alvaro Herrera
Excerpts from Robert Haas's message of vie jun 03 09:17:08 -0400 2011: I've now spent enough time working on this issue now to be convinced that the approach has merit, if we can work out the kinks. I'll start with some performance numbers. I hereby recommend that people with patches such as

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Tom Lane
Dave Page dp...@pgadmin.org writes: On Mon, Jun 6, 2011 at 8:44 PM, Stephen Frost sfr...@snowman.net wrote: If we're going to start putting in changes like this, I'd suggest that we try and target something like September for 9.1 to actually be released.  Playing with the lock management isn't

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Robert Haas
On Mon, Jun 6, 2011 at 6:53 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of vie jun 03 09:17:08 -0400 2011: I've now spent enough time working on this issue now to be convinced that the approach has merit, if we can work out the kinks.  I'll start

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Stephen Frost
* Simon Riggs (si...@2ndquadrant.com) wrote: I see no reason to delay from a July release as has long been planned. What open items are genuine blockers? If we need deadlines anywhere its in beta and final release, otherwise we all just sit around shrugging and saying another week I guess.

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-06 Thread Jignesh Shah
On Mon, Jun 6, 2011 at 2:49 PM, Josh Berkus j...@agliodbs.com wrote: That's an improvement of about ~3.5x.  According to the vmstat output, when running without the patch, the CPU state was about 40% idle. With the patch, it dropped down to around 6%. Wow!  That's fantastic. Jignesh, are

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-05 Thread Stefan Kaltenbrunner
On 06/03/2011 03:17 PM, Robert Haas wrote: [...] As you can see, this works out to a bit more than a 4% improvement on this two-core box. I also got access (thanks to Nate Boley) to a 24-core box and ran the same test with scale factor 100 and shared_buffers=8GB. Here are the results of

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-05 Thread Heikki Linnakangas
On 05.06.2011 22:04, Stefan Kaltenbrunner wrote: and one for the -j80 case(also patched). 485798 48.9667 postgres s_lock 60327 6.0808 postgres LWLockAcquire 57049 5.7503 postgres LWLockRelease 18357 1.8503 postgres

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-05 Thread Stefan Kaltenbrunner
On 06/05/2011 09:12 PM, Heikki Linnakangas wrote: On 05.06.2011 22:04, Stefan Kaltenbrunner wrote: and one for the -j80 case(also patched). 485798 48.9667 postgres s_lock 60327 6.0808 postgres LWLockAcquire 57049 5.7503 postgres

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-05 Thread Robert Haas
On Sun, Jun 5, 2011 at 4:01 PM, Stefan Kaltenbrunner ste...@kaltenbrunner.cc wrote: On 06/05/2011 09:12 PM, Heikki Linnakangas wrote: On 05.06.2011 22:04, Stefan Kaltenbrunner wrote: and one for the -j80 case(also patched). 485798   48.9667  postgres                 s_lock 60327     6.0808  

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-05 Thread Robert Haas
On Sun, Jun 5, 2011 at 5:46 PM, Robert Haas robertmh...@gmail.com wrote: Could you compile with LWLOCK_STATS, rerun these tests, total up the blk numbers by LWLockId, and post the results?  (Actually, totalling up the shacq and exacq numbers would be useful as well, if you wouldn't mind.) I

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-05 Thread Robert Haas
On Sun, Jun 5, 2011 at 10:16 PM, Robert Haas robertmh...@gmail.com wrote: I'm definitely interested in investigating what to do about that, but I don't think it's this patch's problem to fix all of our lock manager bottlenecks. I did some further investigation of this. It appears that more

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-04 Thread Simon Riggs
On Fri, Jun 3, 2011 at 2:17 PM, Robert Haas robertmh...@gmail.com wrote: I've now spent enough time working on this issue now to be convinced that the approach has merit, if we can work out the kinks. Yes, the approach has merits and I'm sure we can work out the kinks. As you can see, this

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-04 Thread Simon Riggs
On Sat, Jun 4, 2011 at 2:59 PM, Simon Riggs si...@2ndquadrant.com wrote: As you can see, this works out to a bit more than a 4% improvement on this two-core box.  I also got access (thanks to Nate Boley) to a 24-core box and ran the same test with scale factor 100 and shared_buffers=8GB.  

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-04 Thread Heikki Linnakangas
On 04.06.2011 18:01, Simon Riggs wrote: It's a fairly isolated patch and we should be considering this for inclusion in 9.1, not wait another year. -1 -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-04 Thread Kevin Grittner
Simon Riggs wrote: we should be considering this for inclusion in 9.1, not wait another year. -1 I'm really happy that we're addressing the problems with scaling to a large number of cores, and this patch sounds great. Adding a new feature at this point in the release cycle would be

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-04 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: The approach looks sound to me. It's a fairly isolated patch and we should be considering this for inclusion in 9.1, not wait another year. That suggestion is completely insane. The patch is only WIP and full of bugs, even according to its author.

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-03 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: That's an improvement of about ~3.5x. Outstanding! I don't want to even peek at this until I've posted the two WIP SSI patches (now both listed on the Open Items page), but will definitely take a look after that. -Kevin -- Sent via pgsql-hackers

Re: [HACKERS] reducing the overhead of frequent table locks - now, with WIP patch

2011-06-03 Thread Robert Haas
On Fri, Jun 3, 2011 at 10:13 AM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Robert Haas robertmh...@gmail.com wrote: That's an improvement of about ~3.5x. Outstanding! I don't want to even peek at this until I've posted the two WIP SSI patches (now both listed on the Open Items