Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-03-02 Thread Bruce Momjian
Dimitri Fontaine wrote: Craig Ringer cr...@postnewspapers.com.au writes: 1) People preparing statements to save on parse+plan time; and 2) People preparing statements to get convenenient param placement. I suspect that most of (1) also want (2), but many of (2) don't care much about (1)

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-03-02 Thread Bruce Momjian
Robert Haas wrote: Adding SQL to indicate whether it should be re-planned or not is completely unappealing. If I could change the code, today, I'd just turn off or choose not to use PREPARE/EXECUTE. Today, PREPARE/EXECUTE seems like it should always be considered slower unless one can

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-03-02 Thread Robert Haas
On Tue, Mar 2, 2010 at 6:54 PM, Bruce Momjian br...@momjian.us wrote: Robert Haas wrote: Adding SQL to indicate whether it should be re-planned or not is completely unappealing. If I could change the code, today, I'd just turn off or choose not to use PREPARE/EXECUTE. Today, PREPARE/EXECUTE

Polyplanner (was Re: [HACKERS] Avoiding bad prepared-statement plans.)

2010-03-01 Thread Yeb Havinga
How about a totally different approach? What if all queries and plans of all queries, simple and prepared, were pre-planned and cached always, persistent? For prepared statements with = 1 parameters, histogram and mcv information could be used to search the plan space for interesting plans.

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-28 Thread Dimitri Fontaine
Craig Ringer cr...@postnewspapers.com.au writes: 1) People preparing statements to save on parse+plan time; and 2) People preparing statements to get convenenient param placement. I suspect that most of (1) also want (2), but many of (2) don't care much about (1) and are just preparing

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-28 Thread Greg Stark
On Fri, Feb 26, 2010 at 4:01 AM, Robert Haas robertmh...@gmail.com wrote: It's not going to be easier to implement.  Yeah, it would be easy to provide a global switch via a GUC setting, but that's not going to be helpful, because this is the sort of thing that really needs to be managed

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-28 Thread Robert Haas
On Sat, Feb 27, 2010 at 11:22 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Fri, Feb 26, 2010 at 7:03 PM, Tom Lane t...@sss.pgh.pa.us wrote: Wouldn't it be better if it just did the right thing automatically? The sort of heuristic I'm envisioning would

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-28 Thread Robert Haas
On Sun, Feb 28, 2010 at 2:52 AM, Mark Mielke m...@mark.mielke.cc wrote: On 02/27/2010 11:20 PM, Craig Ringer wrote: Essentially, you have: 1) People preparing statements to save on parse+plan time; and 2) People preparing statements to get convenenient param placement. I suspect that most

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-27 Thread Robert Haas
On Fri, Feb 26, 2010 at 7:03 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: Basically, what I really want here is some kind of keyword or other syntax that I can stick into a PL/pgsql query that requests a replan on every execution. Wouldn't it be better if

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-27 Thread Craig Ringer
On 26/02/2010 11:40 AM, Tom Lane wrote: But putting support for a per-query level of control into the protocol (and then every client library) as well as every PL is going to be painful to implement, and even more painful to use. You mean something like 'EXECUTE REPLAN' and protocol/PL-level

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-27 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Fri, Feb 26, 2010 at 7:03 PM, Tom Lane t...@sss.pgh.pa.us wrote: Wouldn't it be better if it just did the right thing automatically? The sort of heuristic I'm envisioning would essentially do replan every time for some number of executions, and

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-27 Thread Mark Mielke
On 02/27/2010 11:20 PM, Craig Ringer wrote: Essentially, you have: 1) People preparing statements to save on parse+plan time; and 2) People preparing statements to get convenenient param placement. I suspect that most of (1) also want (2), but many of (2) don't care much about (1) and are

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
My preference is to deal with the specific value vs generic value issue. For this issue, it can affect performance even if PREPARE/EXECUTE is execute exactly once. In the last case I saw, a certain query was executing once every second, and with a specific value it would take 1 ms, and with

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Jeroen Vermeulen
Mark Mielke wrote: Re-planning a generic plan with another generic plan may generate zero benefit, with a measurable cost. More on this after... Nobody's talking about doing that any more. I proposed it initially because I didn't know about changes that made it unnecessary. All the

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 05:20 AM, Jeroen Vermeulen wrote: Mark Mielke wrote: All the points about ms seem invalid to me. There are many reason why ms could increase, and many of them have nothing to do with plan efficiency. Again, re-planning due to a high ms, or a high ratio of ms, does not indicate

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Alex Hunsaker bada...@gmail.com writes: Let me see if I can sum up what I was trying to say: [ this can be solved by using or avoiding prepared statements ] Not really. The place where that argument really fails is inside server-side functions: you don't get to use query submission protocol

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Alex Hunsaker
On Fri, Feb 26, 2010 at 08:07, Tom Lane t...@sss.pgh.pa.us wrote: Alex Hunsaker bada...@gmail.com writes: Let me see if I can sum up what I was trying to say: [ this can be solved by using or avoiding prepared statements ] Not really.  The place where that argument really fails is inside

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Mark Mielke m...@mark.mielke.cc writes: Will guessing at when the user can afford to wait longer improve the situation? Maybe or often, but not always. There is no way to eliminate guessing. The entire point here is that we don't know whether generating a custom plan will provide a win over

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Robert Haas
On Fri, Feb 26, 2010 at 1:29 AM, Alex Hunsaker bada...@gmail.com wrote: Prepared plans + exec plan (new guc/ protocol thing):  Use: not quite sure  Problems: slow because it would replan every time  Solutions: use a prepared plan with the appropriate things not parametrized...? [ aka we

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Robert Haas
On Fri, Feb 26, 2010 at 10:07 AM, Tom Lane t...@sss.pgh.pa.us wrote: I think this is basically a planner problem and should be fixed in the planner, not by expecting users to make significant changes in application logic in order to create an indirect effect. I would agree if I thought that

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Robert Haas
On Fri, Feb 26, 2010 at 11:27 AM, Tom Lane t...@sss.pgh.pa.us wrote: Also, I think there is a lot of confusion here over two different issues: generic plan versus parameter-specific plan, and bad planner estimates leading to a wrong plan choice.  While the latter is certainly an issue

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Fri, Feb 26, 2010 at 11:27 AM, Tom Lane t...@sss.pgh.pa.us wrote: Also, I think there is a lot of confusion here over two different issues: generic plan versus parameter-specific plan, and bad planner estimates leading to a wrong plan choice.  While

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Fri, Feb 26, 2010 at 10:07 AM, Tom Lane t...@sss.pgh.pa.us wrote: I think this is basically a planner problem and should be fixed in the planner, not by expecting users to make significant changes in application logic in order to create an indirect

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 11:27 AM, Tom Lane wrote: Also, I think there is a lot of confusion here over two different issues: generic plan versus parameter-specific plan, and bad planner estimates leading to a wrong plan choice. While the latter is certainly an issue sometimes, there is no reason to

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Mark Mielke m...@mark.mielke.cc writes: Just to point out that I agree, and as per my original post, I think the only time prepared statements should be re-planned for the statistics case, is after 'analyze' has run. That sounds like a quicker solution, and a much smaller gain. After

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 01:59 PM, Tom Lane wrote: ... It's walking around the problem that the idea of a generic plan is just wrong. The only time a generic plan is right, is when the specific plan would result in the same. I think that's a significant overstatement. There are a large number of

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 01:59 PM, Tom Lane wrote: Mark Mielkem...@mark.mielke.cc writes: Just to point out that I agree, and as per my original post, I think the only time prepared statements should be re-planned for the statistics case, is after 'analyze' has run. That sounds like a quicker

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Mark Mielke m...@mark.mielke.cc writes: There must be some way to lift the cost of planning out of the plan enumeration and selection phase, such that only plan enumeration and selection is run at execute time. In most cases, plan enumeration and selection, provided that all data required

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Alex Hunsaker
On Fri, Feb 26, 2010 at 09:50, Robert Haas robertmh...@gmail.com wrote: On Fri, Feb 26, 2010 at 1:29 AM, Alex Hunsaker bada...@gmail.com wrote: Prepared plans + exec plan (new guc/ protocol thing):  Use: not quite sure  Problems: slow because it would replan every time  Solutions: use a

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Yeb Havinga
Tom Lane wrote: Right, but if the parameter is unknown then its distribution is also unknown. In any case that's just nitpicking, because the solution is to create a custom plan for the specific value supplied. Or are you suggesting that we should create a way for users to say here is the

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 02:57 PM, Tom Lane wrote: Mark Mielkem...@mark.mielke.cc writes: There must be some way to lift the cost of planning out of the plan enumeration and selection phase, such that only plan enumeration and selection is run at execute time. In most cases, plan enumeration and

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 03:11 PM, Yeb Havinga wrote: Tom Lane wrote: Right, but if the parameter is unknown then its distribution is also unknown. In any case that's just nitpicking, because the solution is to create a custom plan for the specific value supplied. Or are you suggesting that we should

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Yeb Havinga
Mark Mielke wrote: On 02/26/2010 03:11 PM, Yeb Havinga wrote: Or instead of letting users give the distribution, gather it automatically in some plan statistics catalog? I suspect in most applications queries stay the same for months and maybe years, so after some number of iterations it is

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Mark Mielke m...@mark.mielke.cc writes: Here are parts that can be done fixed: 1) Statement parsing and error checking. 2) Identification of tables and columns involved in the query. The above two are done in the parser, not the planner. 3) Query the column statistics for involved columns,

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Robert Haas
On Fri, Feb 26, 2010 at 12:01 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Fri, Feb 26, 2010 at 11:27 AM, Tom Lane t...@sss.pgh.pa.us wrote: Also, I think there is a lot of confusion here over two different issues: generic plan versus parameter-specific

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: Basically, what I really want here is some kind of keyword or other syntax that I can stick into a PL/pgsql query that requests a replan on every execution. Wouldn't it be better if it just did the right thing automatically? The sort of heuristic I'm

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 07:03 PM, Tom Lane wrote: Robert Haasrobertmh...@gmail.com writes: Basically, what I really want here is some kind of keyword or other syntax that I can stick into a PL/pgsql query that requests a replan on every execution. Wouldn't it be better if it just did the

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-25 Thread Jeroen Vermeulen
Robert Haas wrote: On Wed, Feb 17, 2010 at 5:52 PM, Jeroen Vermeulen j...@xs4all.nl wrote: I may have cut this out of my original email for brevity... my impression is that the planner's estimate is likely to err on the side of scalability, not best-case response time; and that this is more

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-25 Thread Robert Haas
On Thu, Feb 25, 2010 at 9:48 PM, Jeroen Vermeulen j...@xs4all.nl wrote: Robert Haas wrote: On Wed, Feb 17, 2010 at 5:52 PM, Jeroen Vermeulen j...@xs4all.nl wrote: I may have cut this out of my original email for brevity... my impression is that the planner's estimate is likely to err on the

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-25 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I actually think there isn't any clean line. Obscene is in the eye of the beholder. Frankly, I think this discussion is getting off into the weeds. It would be nice, perhaps, to have a feature that will detect when the generic plan is the suxxor and

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-25 Thread Robert Haas
On Thu, Feb 25, 2010 at 10:40 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: I actually think there isn't any clean line.  Obscene is in the eye of the beholder.  Frankly, I think this discussion is getting off into the weeds.  It would be nice, perhaps, to

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-25 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Thu, Feb 25, 2010 at 10:40 PM, Tom Lane t...@sss.pgh.pa.us wrote: I still like the idea of automatically replanning with the known parameter values, and noting whether the result plan was estimated to be noticeably cheaper than the generic plan, and

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-25 Thread Alex Hunsaker
On Thu, Feb 25, 2010 at 20:40, Tom Lane t...@sss.pgh.pa.us wrote: It's not going to be easier to implement.  Yeah, it would be easy to provide a global switch via a GUC setting, but that's not going to be helpful, because this is the sort of thing that really needs to be managed per-query.  

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-25 Thread Alex Hunsaker
On Thu, Feb 25, 2010 at 21:28, Alex Hunsaker bada...@gmail.com wrote: Not to mention you can already do this more or less client side with a nice driver. [ uninformed noise ... ] I did seem to miss the part where everyone thinks this is a crock... But I don't remember seeing numbers on parse

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-25 Thread Tom Lane
Alex Hunsaker bada...@gmail.com writes: I did seem to miss the part where everyone thinks this is a crock... But I don't remember seeing numbers on parse time or how much bandwidth this would potentially save. People seem to think it would be a big savings for just those 2 reasons? Or did I

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-25 Thread Alex Hunsaker
On Thu, Feb 25, 2010 at 22:11, Tom Lane t...@sss.pgh.pa.us wrote: Alex Hunsaker bada...@gmail.com writes: Uh, no, this isn't about saving either parse time or bandwidth. The discussion is about when to expend more planning time in hopes of getting better plans. This is what im tripping over:

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-21 Thread Jeroen Vermeulen
Greg Stark wrote: So in principle I agree with this idea. I think a conservative value for the constant would be more like 100x though. If I told you we had an easy way to speed all your queries up by 10% by caching queries but were just choosing not to then I think you would be unhappy.

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-21 Thread Robert Haas
On Wed, Feb 17, 2010 at 5:52 PM, Jeroen Vermeulen j...@xs4all.nl wrote: I may have cut this out of my original email for brevity... my impression is that the planner's estimate is likely to err on the side of scalability, not best-case response time; and that this is more likely to happen than

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-19 Thread Kenneth Marshall
On Thu, Feb 18, 2010 at 08:31:05PM -0600, David Christensen wrote: On Feb 18, 2010, at 2:19 PM, Pierre C wrote: What about catching the error in the application and INSERT'ing into the current preprepare.relation table? The aim would be to do that in dev or in pre-prod environments, then

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-19 Thread Dimitri Fontaine
Pierre C li...@peufeu.com writes: Yep, but it's a bit awkward and time-consuming, and not quite suited to ORM-generated requests since you got to generate all the plan names, when the SQL query itself would be the most convenient unique identifier... The SHA1 proposal seems better to me. Now

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-18 Thread Pierre C
On Tue, 16 Feb 2010 15:22:00 +0100, Greg Stark gsst...@mit.edu wrote: There's a second problem though. We don't actually know how long any given query is going to take to plan or execute. We could just remember how long it took to plan and execute last time or how long it took to plan last time

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-18 Thread Dimitri Fontaine
Pierre C li...@peufeu.com writes: Problem with prepared statements is they're a chore to use in web apps, especially PHP, since after grabbing a connection from the pool, you don't know if it has prepared plans in it or not. Have you met preprepare yet?

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-18 Thread Pierre C
On Thu, 18 Feb 2010 16:09:42 +0100, Dimitri Fontaine dfonta...@hi-media.com wrote: Pierre C li...@peufeu.com writes: Problem with prepared statements is they're a chore to use in web apps, especially PHP, since after grabbing a connection from the pool, you don't know if it has prepared

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-18 Thread Dimitri Fontaine
Pierre C li...@peufeu.com writes: On Thu, 18 Feb 2010 16:09:42 +0100, Dimitri Fontaine dfonta...@hi-media.com wrote: http://preprepare.projects.postgresql.org/README.html http://packages.debian.org/source/sid/preprepare Hey, this thing is nice. Thanks :) How hard would it be to put a

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-18 Thread Pierre C
What about catching the error in the application and INSERT'ing into the current preprepare.relation table? The aim would be to do that in dev or in pre-prod environments, then copy the table content in production. Yep, but it's a bit awkward and time-consuming, and not quite suited to

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-18 Thread David Christensen
On Feb 18, 2010, at 2:19 PM, Pierre C wrote: What about catching the error in the application and INSERT'ing into the current preprepare.relation table? The aim would be to do that in dev or in pre-prod environments, then copy the table content in production. Yep, but it's a bit

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-16 Thread Greg Stark
On Mon, Feb 15, 2010 at 7:51 PM, Jeroen Vermeulen j...@xs4all.nl wrote: AFAIC a statement could go to re-planning mode if the shortest execution time for the generic plan takes at least 10x longer than the longest planning time.  That gives us a decent shot at finding statements where

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-16 Thread Greg Stark
On Mon, Feb 15, 2010 at 7:11 PM, Bruce Momjian br...@momjian.us wrote: 1. Why do we only do bind-level planning for anonymous wire-level queries? 2. I realize we did anonymous-only because that was the only way we had in the protocol to _signal_ bind-time planning, but didn't we think of this

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-16 Thread Pavel Stehule
Well using parameters will always have a better chance of producing a better plan but that's not the only factor people consider important. For a lot of users *predictability* is more important than absolute performance. If my web server could run 10% faster that might be nice but if it's

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-16 Thread Bruce Momjian
Greg Stark wrote: On Mon, Feb 15, 2010 at 7:11 PM, Bruce Momjian br...@momjian.us wrote: 1. Why do we only do bind-level planning for anonymous wire-level queries? 2. I realize we did anonymous-only because that was the only way we had in the protocol to _signal_ bind-time planning, but

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-16 Thread Greg Stark
On Tue, Feb 16, 2010 at 8:17 PM, Bruce Momjian br...@momjian.us wrote: Incidentally, can you have two active anonymous portals at the same time? No, the first one is deleted when the second is created, i.e., our docs have:        An unnamed prepared statement lasts only until the next Parse

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-15 Thread Bruce Momjian
Pavel Stehule wrote: The problem that we face is that we don't have any very good way to tell whether a fresh planning attempt is likely to yield a plan significantly better than the generic plan. ?I can think of some heuristics --- for example if the query contains LIKE with a

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-15 Thread Jeroen Vermeulen
Tom Lane wrote: Well, no, consider the situation where planning takes 50 ms, the generic plan costs 100ms to execute, but a parameter-specific plan would take 1ms to execute. Planning is very expensive compared to execution but it's still a win to do it. I think that's a fun and worthwhile

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-15 Thread Robert Haas
On Mon, Feb 15, 2010 at 2:11 PM, Bruce Momjian br...@momjian.us wrote: Pavel Stehule wrote: The problem that we face is that we don't have any very good way to tell whether a fresh planning attempt is likely to yield a plan significantly better than the generic plan. ?I can think of some

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-15 Thread Bruce Momjian
Robert Haas wrote: 7. Why is there no option to do parameterized-queries which replan every time? This just seems like an area that has been neglected, or maybe I am missing something and our current setup is acceptable. No, our current setup is not acceptable, and your questions are

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-11 Thread Bart Samwel
Hi Robert, On Tue, Feb 9, 2010 at 17:43, Robert Haas robertmh...@gmail.com wrote: On Tue, Feb 9, 2010 at 7:08 AM, Jeroen Vermeulen j...@xs4all.nl wrote: = Projected-cost threshold = If a prepared statement takes parameters, and the generic plan has a high projected cost, re-plan each

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-11 Thread Pavel Stehule
2010/2/11 Bart Samwel b...@samwel.tk: Hi Robert, On Tue, Feb 9, 2010 at 17:43, Robert Haas robertmh...@gmail.com wrote: On Tue, Feb 9, 2010 at 7:08 AM, Jeroen Vermeulen j...@xs4all.nl wrote: = Projected-cost threshold = If a prepared statement takes parameters, and the generic plan has

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-11 Thread Bart Samwel
On Thu, Feb 11, 2010 at 13:25, Pavel Stehule pavel.steh...@gmail.comwrote: 2010/2/11 Bart Samwel b...@samwel.tk: Perhaps this could be based on a (configurable?) ratio of observed planning time and projected execution time. I mean, if planning it the first time took 30 ms and projected

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-11 Thread Robert Haas
On Thu, Feb 11, 2010 at 7:39 AM, Bart Samwel b...@samwel.tk wrote: On Thu, Feb 11, 2010 at 13:25, Pavel Stehule pavel.steh...@gmail.com wrote: 2010/2/11 Bart Samwel b...@samwel.tk: Perhaps this could be based on a (configurable?) ratio of observed planning time and projected execution

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-11 Thread Bart Samwel
On Thu, Feb 11, 2010 at 13:41, Robert Haas robertmh...@gmail.com wrote: On Thu, Feb 11, 2010 at 7:39 AM, Bart Samwel b...@samwel.tk wrote: Anyhow, I have no clue how much time the planner takes. Can anybody provide any statistics in that regard? It depends a great deal on the query, which

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-11 Thread Robert Haas
On Thu, Feb 11, 2010 at 7:48 AM, Bart Samwel b...@samwel.tk wrote: On Thu, Feb 11, 2010 at 13:41, Robert Haas robertmh...@gmail.com wrote: On Thu, Feb 11, 2010 at 7:39 AM, Bart Samwel b...@samwel.tk wrote: Anyhow, I have no clue how much time the planner takes. Can anybody provide any

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-11 Thread Yeb Havinga
Bart Samwel wrote: Perhaps this could be based on a (configurable?) ratio of observed planning time and projected execution time. I mean, if planning it the first time took 30 ms and projected execution time is 1 ms, then by all means NEVER re-plan. IMHO looking at ms is bad for this 'possible

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-11 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Thu, Feb 11, 2010 at 7:48 AM, Bart Samwel b...@samwel.tk wrote: Because that's the underlying assumption of the ratio criterion -- that re-planning with filled-in parameters takes about as much time as the initial planning run took. We only want

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-11 Thread Pavel Stehule
2010/2/11 Tom Lane t...@sss.pgh.pa.us: Robert Haas robertmh...@gmail.com writes: On Thu, Feb 11, 2010 at 7:48 AM, Bart Samwel b...@samwel.tk wrote: Because that's the underlying assumption of the ratio criterion -- that re-planning with filled-in parameters takes about as much time as the

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-10 Thread Bruce Momjian
Kris Jurka wrote: The JDBC driver has two methods of disabling permanently planned prepared statements: 1) Use the version two frontend/backend protocol via adding protocolVersion=2 to your URL. This interpolates all parameters into the query on the client side. 2) Execute

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-10 Thread Bruce Momjian
Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Tue, Feb 9, 2010 at 7:08 AM, Jeroen Vermeulen j...@xs4all.nl wrote: Periodically re-plan prepared statements on EXECUTE. ?This is also a chance for queries that were being re-planned every time to go back to a generic plan.

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-10 Thread Tom Lane
Bruce Momjian br...@momjian.us writes: Can someone explain to me why we only do delayed binding for unnamed prepared queries? It was a way of shoehorning in some driver control over the behavior without the protocol bump that would be involved in adding an actual option to Parse messages.

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-10 Thread Дмитрий Фефелов
The only case that I think still has any merit is where you get a significantly better plan with known parameter values than without. The projected-cost threshold might be a reasonable approach for attacking that, ie, if estimated cost of generic plan exceeds X then take the time to build a

[HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Jeroen Vermeulen
I've been discussing this with Josh, Heikki, and Peter E. over the past few weeks. As Peter observed years ago, prepared statements can perform badly because their plans are overly generic. Also, statistics change and sometimes plans should change with them. It would be nice if we could

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Jeroen Vermeulen
Yeb Havinga wrote: I've been discussing this with Josh, Heikki, and Peter E. over the past few weeks. Is this searchable in the archives? I'm interested in ideas discussed. No, sorry. These were face-to-face discussions at linux.conf.au and FOSDEM. If a prepared statement takes

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Richard Huxton
On 09/02/10 12:08, Jeroen Vermeulen wrote: = Projected-cost threshold = [snip - this is the simple bit. Sounds very sensible. ] = Actual-cost threshold = Also stop using the generic plan if the statement takes a long time to run in practice. Do you mean: 1. Rollback the current query and

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Yeb Havinga
Jeroen Vermeulen wrote: I've been discussing this with Josh, Heikki, and Peter E. over the past few weeks. Is this searchable in the archives? I'm interested in ideas discussed. If a prepared statement takes parameters, and the generic plan has a high projected cost, re-plan each EXECUTE

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Andres Freund
On Tuesday 09 February 2010 13:08:54 Jeroen Vermeulen wrote: I've been discussing this with Josh, Heikki, and Peter E. over the past few weeks. As Peter observed years ago, prepared statements can perform badly because their plans are overly generic. Also, statistics change and sometimes

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Jeroen Vermeulen
Richard Huxton wrote: = Actual-cost threshold = Also stop using the generic plan if the statement takes a long time to run in practice. Do you mean: 1. Rollback the current query and start again 2. Mark the plan as a bad one and plan again next execute If you can figure out how to do #1

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Jeroen Vermeulen
Andres Freund wrote: = Actual-cost threshold = Also stop using the generic plan if the statement takes a long time to run in practice. Statistics may have gone bad. It could also be a one-off due to a load peak or something, but that's handled by: That is not that easy. It means that you

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Richard Huxton
On 09/02/10 14:25, Jeroen Vermeulen wrote: Richard Huxton wrote: = Actual-cost threshold = Also stop using the generic plan if the statement takes a long time to run in practice. Do you mean: 1. Rollback the current query and start again 2. Mark the plan as a bad one and plan again next

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Mark Mielke
On 02/09/2010 08:46 AM, Jeroen Vermeulen wrote: This sounds like a really nice to have feature. Maybe it'd also be possible to skip replanning between executes if the current bound values are 'indexwise-equivalent' to the values used at previous planning, i.e. nothing in the statistics

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Robert Haas
On Tue, Feb 9, 2010 at 7:08 AM, Jeroen Vermeulen j...@xs4all.nl wrote: = Projected-cost threshold = If a prepared statement takes parameters, and the generic plan has a high projected cost, re-plan each EXECUTE individually with all its parameter values bound.  It may or may not help, but

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Tue, Feb 9, 2010 at 7:08 AM, Jeroen Vermeulen j...@xs4all.nl wrote: Periodically re-plan prepared statements on EXECUTE.  This is also a chance for queries that were being re-planned every time to go back to a generic plan. The most common problem

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Dimitri Fontaine
Jeroen Vermeulen j...@xs4all.nl writes: I think we should be careful not to over-think this. Planning isn't *that* costly, so apply Amdahl's Law liberally. I'm proposing some easy things we could do without adding much overhead or maintenance burden; I've been assuming that getting intimate

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-09 Thread Kris Jurka
On Tue, 9 Feb 2010, Mark Mielke wrote: In a current commercial app we have that uses JDBC and prepared plans for just about everything, it regularly ends up with execution times of 30+ milliseconds when a complete plan + execute would take less than 1 millisecond. PostgreSQL planning is