Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-03-02 Thread Robert Haas
On Tue, Mar 2, 2010 at 6:54 PM, Bruce Momjian wrote: > Robert Haas wrote: >> > Adding SQL to indicate whether it should be re-planned or not is completely >> > unappealing. If I could change the code, today, I'd just turn off or choose >> > not to use PREPARE/EXECUTE. Today, PREPARE/EXECUTE seems

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-03-02 Thread Bruce Momjian
Robert Haas wrote: > > Adding SQL to indicate whether it should be re-planned or not is completely > > unappealing. If I could change the code, today, I'd just turn off or choose > > not to use PREPARE/EXECUTE. Today, PREPARE/EXECUTE seems like it should > > always be considered slower unless one c

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-03-02 Thread Bruce Momjian
Dimitri Fontaine wrote: > Craig Ringer writes: > > 1) People preparing statements to save on parse+plan time; and > > 2) People preparing statements to get convenenient param placement. > > > > I suspect that most of (1) also want (2), but many of (2) don't care much > > about (1) and are just pre

Polyplanner (was Re: [HACKERS] Avoiding bad prepared-statement plans.)

2010-03-01 Thread Yeb Havinga
How about a totally different approach? What if all queries and plans of all queries, simple and prepared, were pre-planned and cached always, persistent? For prepared statements with >= 1 parameters, histogram and mcv information could be used to search the plan space for interesting plans. M

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-28 Thread Robert Haas
On Sun, Feb 28, 2010 at 2:52 AM, Mark Mielke wrote: > On 02/27/2010 11:20 PM, Craig Ringer wrote: >> >> Essentially, you have: >> >> 1) People preparing statements to save on parse+plan time; and >> 2) People preparing statements to get convenenient param placement. >> >> I suspect that most of (1

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-28 Thread Robert Haas
On Sat, Feb 27, 2010 at 11:22 PM, Tom Lane wrote: > Robert Haas writes: >> On Fri, Feb 26, 2010 at 7:03 PM, Tom Lane wrote: >>> Wouldn't it be better if it just did the right thing automatically? >>> >>> The sort of heuristic I'm envisioning would essentially do "replan every >>> time" for some

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-28 Thread Greg Stark
On Fri, Feb 26, 2010 at 4:01 AM, Robert Haas wrote: >> It's not going to be easier to implement.  Yeah, it would be easy to >> provide a global switch via a GUC setting, but that's not going to be >> helpful, because this is the sort of thing that really needs to be >> managed per-query.  Almost a

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-28 Thread Dimitri Fontaine
Craig Ringer writes: > 1) People preparing statements to save on parse+plan time; and > 2) People preparing statements to get convenenient param placement. > > I suspect that most of (1) also want (2), but many of (2) don't care much > about (1) and are just preparing statements for sql-injection

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-27 Thread Mark Mielke
On 02/27/2010 11:20 PM, Craig Ringer wrote: Essentially, you have: 1) People preparing statements to save on parse+plan time; and 2) People preparing statements to get convenenient param placement. I suspect that most of (1) also want (2), but many of (2) don't care much about (1) and are just

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-27 Thread Tom Lane
Robert Haas writes: > On Fri, Feb 26, 2010 at 7:03 PM, Tom Lane wrote: >> Wouldn't it be better if it just did the right thing automatically? >> >> The sort of heuristic I'm envisioning would essentially do "replan every >> time" for some number of executions, and give up only if it noticed that

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-27 Thread Craig Ringer
On 26/02/2010 11:40 AM, Tom Lane wrote: But putting support for a per-query level of control into the protocol (and then every client library) as well as every PL is going to be painful to implement, and even more painful to use. You mean something like 'EXECUTE REPLAN' and protocol/PL-level e

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-27 Thread Robert Haas
On Fri, Feb 26, 2010 at 7:03 PM, Tom Lane wrote: > Robert Haas writes: >> Basically, what I really want here is some kind of keyword or other >> syntax that I can stick into a PL/pgsql query that requests a replan >> on every execution. > > Wouldn't it be better if it just did the right thing aut

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 07:03 PM, Tom Lane wrote: Robert Haas writes: Basically, what I really want here is some kind of keyword or other syntax that I can stick into a PL/pgsql query that requests a replan on every execution. Wouldn't it be better if it just did the right thing automatically?

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Robert Haas writes: > Basically, what I really want here is some kind of keyword or other > syntax that I can stick into a PL/pgsql query that requests a replan > on every execution. Wouldn't it be better if it just did the right thing automatically? The sort of heuristic I'm envisioning would e

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Robert Haas
On Fri, Feb 26, 2010 at 12:01 PM, Tom Lane wrote: > Robert Haas writes: >> On Fri, Feb 26, 2010 at 11:27 AM, Tom Lane wrote: >>> Also, I think there is a lot of confusion here over two different >>> issues: generic plan versus parameter-specific plan, and bad planner >>> estimates leading to a w

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Mark Mielke writes: > Here are parts that can be done "fixed": > 1) Statement parsing and error checking. > 2) Identification of tables and columns involved in the query. The above two are done in the parser, not the planner. > 3) Query the column statistics for involved columns, to be used in

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Yeb Havinga
Mark Mielke wrote: On 02/26/2010 03:11 PM, Yeb Havinga wrote: Or instead of letting users give the distribution, gather it automatically in some plan statistics catalog? I suspect in most applications queries stay the same for months and maybe years, so after some number of iterations it is po

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 03:11 PM, Yeb Havinga wrote: Tom Lane wrote: Right, but if the parameter is unknown then its distribution is also unknown. In any case that's just nitpicking, because the solution is to create a custom plan for the specific value supplied. Or are you suggesting that we should cre

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 02:57 PM, Tom Lane wrote: Mark Mielke writes: There must be some way to lift the cost of planning out of the plan enumeration and selection phase, such that only plan enumeration and selection is run at execute time. In most cases, plan enumeration and selection, provided that

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Yeb Havinga
Tom Lane wrote: Right, but if the parameter is unknown then its distribution is also unknown. In any case that's just nitpicking, because the solution is to create a custom plan for the specific value supplied. Or are you suggesting that we should create a way for users to say "here is the expe

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Alex Hunsaker
On Fri, Feb 26, 2010 at 09:50, Robert Haas wrote: > On Fri, Feb 26, 2010 at 1:29 AM, Alex Hunsaker wrote: >> Prepared plans + exec plan (new guc/ protocol thing): >>  Use: not quite sure >>  Problems: slow because it would replan every time >>  Solutions: use a prepared plan with the appropriate

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Mark Mielke writes: > There must be some way to lift the cost of planning out of the plan > enumeration and selection phase, such that only plan enumeration and > selection is run at execute time. In most cases, plan enumeration and > selection, provided that all data required to make these dec

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 01:59 PM, Tom Lane wrote: Mark Mielke writes: Just to point out that I agree, and as per my original post, I think the only time prepared statements should be re-planned for the statistics case, is after 'analyze' has run. That sounds like a quicker solution, and a much smalle

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 01:59 PM, Tom Lane wrote: ... It's walking around the problem that the idea of a generic plan is just wrong. The only time a generic plan is right, is when the specific plan would result in the same. I think that's a significant overstatement. There are a large number of cas

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Mark Mielke writes: > Just to point out that I agree, and as per my original post, I think the > only time prepared statements should be re-planned for the statistics > case, is after 'analyze' has run. That sounds like a quicker solution, > and a much smaller gain. After 'analyze' of an object

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 11:27 AM, Tom Lane wrote: Also, I think there is a lot of confusion here over two different issues: generic plan versus parameter-specific plan, and bad planner estimates leading to a wrong plan choice. While the latter is certainly an issue sometimes, there is no reason to believe

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Robert Haas writes: > On Fri, Feb 26, 2010 at 10:07 AM, Tom Lane wrote: >> I think this is basically a planner problem and should be fixed in the >> planner, not by expecting users to make significant changes in >> application logic in order to create an indirect effect. > I would agree if I tho

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Robert Haas writes: > On Fri, Feb 26, 2010 at 11:27 AM, Tom Lane wrote: >> Also, I think there is a lot of confusion here over two different >> issues: generic plan versus parameter-specific plan, and bad planner >> estimates leading to a wrong plan choice.  While the latter is certainly >> an is

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Robert Haas
On Fri, Feb 26, 2010 at 11:27 AM, Tom Lane wrote: > Also, I think there is a lot of confusion here over two different > issues: generic plan versus parameter-specific plan, and bad planner > estimates leading to a wrong plan choice.  While the latter is certainly > an issue sometimes, there is no

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Robert Haas
On Fri, Feb 26, 2010 at 10:07 AM, Tom Lane wrote: > I think this is basically a planner problem and should be fixed in the > planner, not by expecting users to make significant changes in > application logic in order to create an indirect effect. I would agree if I thought that were possible, but

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Robert Haas
On Fri, Feb 26, 2010 at 1:29 AM, Alex Hunsaker wrote: > Prepared plans + exec plan (new guc/ protocol thing): >  Use: not quite sure >  Problems: slow because it would replan every time >  Solutions: use a prepared plan with the appropriate things not > parametrized...? > > [ aka we already have t

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Mark Mielke writes: > Will "guessing" at when the user can afford to wait longer improve the > situation? Maybe or often, but not always. There is no way to eliminate "guessing". The entire point here is that we don't know whether generating a custom plan will provide a win over not doing so, u

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Alex Hunsaker
On Fri, Feb 26, 2010 at 08:07, Tom Lane wrote: > Alex Hunsaker writes: >> Let me see if I can sum up what I was trying to say: >> [ this can be solved by using or avoiding prepared statements ] > > Not really.  The place where that argument really fails is inside > server-side functions: you don'

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Alex Hunsaker writes: > Let me see if I can sum up what I was trying to say: > [ this can be solved by using or avoiding prepared statements ] Not really. The place where that argument really fails is inside server-side functions: you don't get to use query submission protocol there. But even f

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 05:20 AM, Jeroen Vermeulen wrote: Mark Mielke wrote: All the points about ms seem invalid to me. There are many reason why ms could increase, and many of them have nothing to do with plan efficiency. Again, re-planning due to a high ms, or a high ratio of ms, does not indicate t

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Jeroen Vermeulen
Mark Mielke wrote: Re-planning a generic plan with another generic plan may generate zero benefit, with a measurable cost. More on this after... Nobody's talking about doing that any more. I proposed it initially because I didn't know about changes that made it unnecessary. All the points

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
My preference is to deal with the specific value vs generic value issue. For this issue, it can affect performance even if PREPARE/EXECUTE is execute exactly once. In the last case I saw, a certain query was executing once every second, and with a specific value it would take < 1 ms, and with

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-25 Thread Alex Hunsaker
On Thu, Feb 25, 2010 at 22:11, Tom Lane wrote: > Alex Hunsaker writes: > Uh, no, this isn't about saving either parse time or bandwidth. > The discussion is about when to expend more planning time in hopes > of getting better plans. This is what im tripping over: > > Bruce's suggestion that we

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-25 Thread Tom Lane
Alex Hunsaker writes: > I did seem to miss the part where everyone thinks this is a crock... > But I don't remember seeing numbers on parse time or how much > bandwidth this would potentially save. People seem to think it would > be a big savings for just those 2 reasons? Or did I miss some othe

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-25 Thread Alex Hunsaker
On Thu, Feb 25, 2010 at 21:28, Alex Hunsaker wrote: > Not to mention you can already do this more or less client side with a > nice driver. > [ uninformed noise ... ] I did seem to miss the part where everyone thinks this is a crock... But I don't remember seeing numbers on parse time or how much

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-25 Thread Alex Hunsaker
On Thu, Feb 25, 2010 at 20:40, Tom Lane wrote: > It's not going to be easier to implement.  Yeah, it would be easy to > provide a global switch via a GUC setting, but that's not going to be > helpful, because this is the sort of thing that really needs to be > managed per-query.  Almost any nontri

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-25 Thread Tom Lane
Robert Haas writes: > On Thu, Feb 25, 2010 at 10:40 PM, Tom Lane wrote: >> I still like the idea of automatically replanning with the known >> parameter values, and noting whether the result plan was estimated to be >> noticeably cheaper than the generic plan, and giving up on generating >> custo

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-25 Thread Robert Haas
On Thu, Feb 25, 2010 at 10:40 PM, Tom Lane wrote: > Robert Haas writes: >> I actually think there isn't any clean line.  Obscene is in the eye of >> the beholder.  Frankly, I think this discussion is getting off into >> the weeds.  It would be nice, perhaps, to have a feature that will >> detect

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-25 Thread Tom Lane
Robert Haas writes: > I actually think there isn't any clean line. Obscene is in the eye of > the beholder. Frankly, I think this discussion is getting off into > the weeds. It would be nice, perhaps, to have a feature that will > detect when the generic plan is the suxxor and attempt to find a

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-25 Thread Robert Haas
On Thu, Feb 25, 2010 at 9:48 PM, Jeroen Vermeulen wrote: > Robert Haas wrote: >> >> On Wed, Feb 17, 2010 at 5:52 PM, Jeroen Vermeulen wrote: >>> >>> I may have cut this out of my original email for brevity... my impression >>> is >>> that the planner's estimate is likely to err on the side of sca

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-25 Thread Jeroen Vermeulen
Robert Haas wrote: On Wed, Feb 17, 2010 at 5:52 PM, Jeroen Vermeulen wrote: I may have cut this out of my original email for brevity... my impression is that the planner's estimate is likely to err on the side of scalability, not best-case response time; and that this is more likely to happen t

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-21 Thread Robert Haas
On Wed, Feb 17, 2010 at 5:52 PM, Jeroen Vermeulen wrote: > I may have cut this out of my original email for brevity... my impression is > that the planner's estimate is likely to err on the side of scalability, not > best-case response time; and that this is more likely to happen than an > optimis

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-21 Thread Jeroen Vermeulen
Greg Stark wrote: So in principle I agree with this idea. I think a conservative value for the constant would be more like 100x though. If I told you we had an easy way to speed all your queries up by 10% by caching queries but were just choosing not to then I think you would be unhappy. Whereas

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-19 Thread Dimitri Fontaine
"Pierre C" writes: > Yep, but it's a bit awkward and time-consuming, and not quite suited to > ORM-generated requests since you got to generate all the plan names, when > the SQL query itself would be the most convenient "unique > identifier"... The SHA1 proposal seems better to me. Now you stil

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-19 Thread Kenneth Marshall
On Thu, Feb 18, 2010 at 08:31:05PM -0600, David Christensen wrote: > > On Feb 18, 2010, at 2:19 PM, Pierre C wrote: > >> >>> What about catching the error in the application and INSERT'ing into the >>> current preprepare.relation table? The aim would be to do that in dev or >>> in pre-prod environm

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-18 Thread David Christensen
On Feb 18, 2010, at 2:19 PM, Pierre C wrote: What about catching the error in the application and INSERT'ing into the current preprepare.relation table? The aim would be to do that in dev or in pre-prod environments, then copy the table content in production. Yep, but it's a bit awkward

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-18 Thread Pierre C
What about catching the error in the application and INSERT'ing into the current preprepare.relation table? The aim would be to do that in dev or in pre-prod environments, then copy the table content in production. Yep, but it's a bit awkward and time-consuming, and not quite suited to ORM-g

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-18 Thread Dimitri Fontaine
"Pierre C" writes: > On Thu, 18 Feb 2010 16:09:42 +0100, Dimitri Fontaine > wrote: >> http://preprepare.projects.postgresql.org/README.html >> http://packages.debian.org/source/sid/preprepare > > Hey, this thing is nice. Thanks :) > How hard would it be to put a hook in pg so that, instead

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-18 Thread Pierre C
On Thu, 18 Feb 2010 16:09:42 +0100, Dimitri Fontaine wrote: "Pierre C" writes: Problem with prepared statements is they're a chore to use in web apps, especially PHP, since after grabbing a connection from the pool, you don't know if it has prepared plans in it or not. Have you met pre

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-18 Thread Dimitri Fontaine
"Pierre C" writes: > Problem with prepared statements is they're a chore to use in web apps, > especially PHP, since after grabbing a connection from the pool, you don't > know if it has prepared plans in it or not. Have you met preprepare yet? http://preprepare.projects.postgresql.org/README.

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-18 Thread Pierre C
On Tue, 16 Feb 2010 15:22:00 +0100, Greg Stark wrote: There's a second problem though. We don't actually know how long any given query is going to take to plan or execute. We could just remember how long it took to plan and execute last time or how long it took to plan last time and the average

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-16 Thread Greg Stark
On Tue, Feb 16, 2010 at 8:17 PM, Bruce Momjian wrote: >> Incidentally, can you have two active anonymous portals at the same time? > > No, the first one is deleted when the second is created, i.e., our docs > have: > >        An unnamed prepared statement lasts only until the next Parse statement

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-16 Thread Bruce Momjian
Greg Stark wrote: > On Mon, Feb 15, 2010 at 7:11 PM, Bruce Momjian wrote: > > 1. Why do we only do bind-level planning for anonymous wire-level queries? > > > > 2. I realize we did anonymous-only because that was the only way we had > > in the protocol to _signal_ bind-time planning, but didn't we

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-16 Thread Pavel Stehule
> > Well using parameters will always have a better chance of producing a > better plan but that's not the only factor people consider important. > For a lot of users *predictability* is more important than absolute > performance. If my web server could run 10% faster that might be nice > but if it

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-16 Thread Greg Stark
On Mon, Feb 15, 2010 at 7:11 PM, Bruce Momjian wrote: > 1. Why do we only do bind-level planning for anonymous wire-level queries? > > 2. I realize we did anonymous-only because that was the only way we had > in the protocol to _signal_ bind-time planning, but didn't we think of > this when we wer

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-16 Thread Greg Stark
On Mon, Feb 15, 2010 at 7:51 PM, Jeroen Vermeulen wrote: > AFAIC a statement could go to "re-planning mode" if the shortest execution > time for the generic plan takes at least 10x longer than the longest > planning time.  That gives us a decent shot at finding statements where > re-planning is a

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-15 Thread Bruce Momjian
Robert Haas wrote: > > 7. Why is there no option to do parameterized-queries which replan every > > time? > > > > This just seems like an area that has been neglected, or maybe I am > > missing something and our current setup is acceptable. > > No, our current setup is not acceptable, and your que

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-15 Thread Robert Haas
On Mon, Feb 15, 2010 at 2:11 PM, Bruce Momjian wrote: > Pavel Stehule wrote: >> > The problem that we face is that we don't have any very good way to tell >> > whether a fresh planning attempt is likely to yield a plan significantly >> > better than the generic plan. ?I can think of some heuristic

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-15 Thread Jeroen Vermeulen
Tom Lane wrote: Well, no, consider the situation where planning takes 50 ms, the generic plan costs 100ms to execute, but a parameter-specific plan would take 1ms to execute. Planning is very expensive compared to execution but it's still a win to do it. I think that's a fun and worthwhile pr

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-15 Thread Bruce Momjian
Pavel Stehule wrote: > > The problem that we face is that we don't have any very good way to tell > > whether a fresh planning attempt is likely to yield a plan significantly > > better than the generic plan. ?I can think of some heuristics --- for > > example if the query contains LIKE with a para

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-11 Thread Pavel Stehule
2010/2/11 Tom Lane : > Robert Haas writes: >> On Thu, Feb 11, 2010 at 7:48 AM, Bart Samwel wrote: >>> Because that's the >>> underlying assumption of the "ratio" criterion -- that re-planning with >>> filled-in parameters takes about as much time as the initial planning run >>> took. > >> We only

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-11 Thread Tom Lane
Robert Haas writes: > On Thu, Feb 11, 2010 at 7:48 AM, Bart Samwel wrote: >> Because that's the >> underlying assumption of the "ratio" criterion -- that re-planning with >> filled-in parameters takes about as much time as the initial planning run >> took. > We only want to replan when replannin

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-11 Thread Yeb Havinga
Bart Samwel wrote: Perhaps this could be based on a (configurable?) ratio of observed planning time and projected execution time. I mean, if planning it the first time took 30 ms and projected execution time is 1 ms, then by all means NEVER re-plan. IMHO looking at ms is bad for this 'possible

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-11 Thread Robert Haas
On Thu, Feb 11, 2010 at 7:48 AM, Bart Samwel wrote: > On Thu, Feb 11, 2010 at 13:41, Robert Haas wrote: >> >> On Thu, Feb 11, 2010 at 7:39 AM, Bart Samwel wrote: >> > Anyhow, I have no clue how much time the planner takes. Can anybody >> > provide >> > any statistics in that regard? >> >> It dep

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-11 Thread Bart Samwel
On Thu, Feb 11, 2010 at 13:41, Robert Haas wrote: > On Thu, Feb 11, 2010 at 7:39 AM, Bart Samwel wrote: > > Anyhow, I have no clue how much time the planner takes. Can anybody > provide > > any statistics in that regard? > > It depends a great deal on the query, which is one of the things that >

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-11 Thread Robert Haas
On Thu, Feb 11, 2010 at 7:39 AM, Bart Samwel wrote: > On Thu, Feb 11, 2010 at 13:25, Pavel Stehule > wrote: >> >> 2010/2/11 Bart Samwel : >> > Perhaps this could be based on a (configurable?) ratio of observed >> > planning >> > time and projected execution time. I mean, if planning it the first

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-11 Thread Bart Samwel
On Thu, Feb 11, 2010 at 13:25, Pavel Stehule wrote: > 2010/2/11 Bart Samwel : > > Perhaps this could be based on a (configurable?) ratio of observed > planning > > time and projected execution time. I mean, if planning it the first time > > took 30 ms and projected execution time is 1 ms, then by

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-11 Thread Pavel Stehule
2010/2/11 Bart Samwel : > Hi Robert, > > On Tue, Feb 9, 2010 at 17:43, Robert Haas wrote: >> >> On Tue, Feb 9, 2010 at 7:08 AM, Jeroen Vermeulen wrote: >> > = Projected-cost threshold = >> > >> > If a prepared statement takes parameters, and the generic plan has a >> > high >> > projected cost, r

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-11 Thread Bart Samwel
Hi Robert, On Tue, Feb 9, 2010 at 17:43, Robert Haas wrote: > On Tue, Feb 9, 2010 at 7:08 AM, Jeroen Vermeulen wrote: > > = Projected-cost threshold = > > > > If a prepared statement takes parameters, and the generic plan has a high > > projected cost, re-plan each EXECUTE individually with all

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-10 Thread Дмитрий Фефелов
> The only case that I think still has any merit is where you get a > significantly better plan with known parameter values than without. > The projected-cost threshold might be a reasonable approach for > attacking that, ie, if estimated cost of generic plan exceeds X > then take the time to build

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-10 Thread Tom Lane
Bruce Momjian writes: > Can someone explain to me why we only do "delayed binding" for unnamed > prepared queries? It was a way of shoehorning in some driver control over the behavior without the protocol bump that would be involved in adding an actual option to Parse messages.

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-10 Thread Bruce Momjian
Tom Lane wrote: > Robert Haas writes: > > On Tue, Feb 9, 2010 at 7:08 AM, Jeroen Vermeulen wrote: > >> Periodically re-plan prepared statements on EXECUTE. ?This is also a chance > >> for queries that were being re-planned every time to go back to a generic > >> plan. > > > The most common probl

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-10 Thread Bruce Momjian
Kris Jurka wrote: > > The JDBC driver has two methods of disabling permanently planned prepared > statements: > > 1) Use the version two frontend/backend protocol via adding > protocolVersion=2 to your URL. This interpolates all parameters into > the query on the client side. > > 2) Execute

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Kris Jurka
On Tue, 9 Feb 2010, Mark Mielke wrote: In a current commercial app we have that uses JDBC and prepared plans for just about everything, it regularly ends up with execution times of 30+ milliseconds when a complete plan + execute would take less than 1 millisecond. PostgreSQL planning is pr

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Dimitri Fontaine
Jeroen Vermeulen writes: > I think we should be careful not to over-think this. Planning isn't *that* > costly, so apply Amdahl's Law liberally. I'm proposing some easy things we > could do without adding much overhead or maintenance burden; I've been > assuming that getting intimate with the pl

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Tom Lane
Robert Haas writes: > On Tue, Feb 9, 2010 at 7:08 AM, Jeroen Vermeulen wrote: >> Periodically re-plan prepared statements on EXECUTE.  This is also a chance >> for queries that were being re-planned every time to go back to a generic >> plan. > The most common problem here seems to be that (some

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Robert Haas
On Tue, Feb 9, 2010 at 7:08 AM, Jeroen Vermeulen wrote: > = Projected-cost threshold = > > If a prepared statement takes parameters, and the generic plan has a high > projected cost, re-plan each EXECUTE individually with all its parameter > values bound.  It may or may not help, but unless the pl

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Mark Mielke
On 02/09/2010 08:46 AM, Jeroen Vermeulen wrote: This sounds like a really nice to have feature. Maybe it'd also be possible to skip replanning between executes if the current bound values are 'indexwise-equivalent' to the values used at previous planning, i.e. nothing in the statistics indicate

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Richard Huxton
On 09/02/10 14:25, Jeroen Vermeulen wrote: Richard Huxton wrote: = Actual-cost threshold = Also stop using the generic plan if the statement takes a long time to run in practice. Do you mean: 1. Rollback the current query and start again 2. Mark the plan as a bad one and plan again next exec

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Jeroen Vermeulen
Andres Freund wrote: = Actual-cost threshold = Also stop using the generic plan if the statement takes a long time to run in practice. Statistics may have gone bad. It could also be a one-off due to a load peak or something, but that's handled by: That is not that easy. It means that you ha

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Jeroen Vermeulen
Richard Huxton wrote: = Actual-cost threshold = Also stop using the generic plan if the statement takes a long time to run in practice. Do you mean: 1. Rollback the current query and start again 2. Mark the plan as a bad one and plan again next execute If you can figure out how to do #1 then

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Andres Freund
On Tuesday 09 February 2010 13:08:54 Jeroen Vermeulen wrote: > I've been discussing this with Josh, Heikki, and Peter E. over the past > few weeks. > > As Peter observed years ago, prepared statements can perform badly > because their plans are overly generic. Also, statistics change and > someti

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Yeb Havinga
Jeroen Vermeulen wrote: I've been discussing this with Josh, Heikki, and Peter E. over the past few weeks. Is this searchable in the archives? I'm interested in ideas discussed. If a prepared statement takes parameters, and the generic plan has a high projected cost, re-plan each EXECUTE indivi

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Richard Huxton
On 09/02/10 12:08, Jeroen Vermeulen wrote: = Projected-cost threshold = [snip - this is the simple bit. Sounds very sensible. ] = Actual-cost threshold = Also stop using the generic plan if the statement takes a long time to run in practice. Do you mean: 1. Rollback the current query and

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Jeroen Vermeulen
Yeb Havinga wrote: I've been discussing this with Josh, Heikki, and Peter E. over the past few weeks. Is this searchable in the archives? I'm interested in ideas discussed. No, sorry. These were face-to-face discussions at linux.conf.au and FOSDEM. If a prepared statement takes parameters,

[HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Jeroen Vermeulen
I've been discussing this with Josh, Heikki, and Peter E. over the past few weeks. As Peter observed years ago, prepared statements can perform badly because their plans are overly generic. Also, statistics change and sometimes plans should change with them. It would be nice if we could avo