Re: [HACKERS] upper planner path-ification

2015-06-23 Thread Robert Haas
On Tue, Jun 23, 2015 at 4:41 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: So, unless we don't find out a solution around planner, 2-phase aggregation is like a curry without rice Simon and I spoke with Tom about this upper planner path-ification problem at PGCon, and he indicated that he

Re: [HACKERS] upper planner path-ification

2015-06-23 Thread Kouhei Kaigai
-Original Message- From: David Rowley [mailto:david.row...@2ndquadrant.com] Sent: Tuesday, June 23, 2015 2:06 PM To: Kaigai Kouhei(海外 浩平) Cc: Robert Haas; pgsql-hackers@postgresql.org; Tom Lane Subject: Re: [HACKERS] upper planner path-ification On 23 June 2015 at 13:55, Kouhei

Re: [HACKERS] upper planner path-ification

2015-06-23 Thread Kouhei Kaigai
-Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Robert Haas Sent: Tuesday, June 23, 2015 10:18 PM To: Kaigai Kouhei(海外 浩平) Cc: David Rowley; pgsql-hackers@postgresql.org; Tom Lane Subject: Re: [HACKERS] upper

Re: [HACKERS] upper planner path-ification

2015-06-22 Thread Kouhei Kaigai
-Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Robert Haas Sent: Thursday, May 14, 2015 10:39 AM To: pgsql-hackers@postgresql.org; Tom Lane Subject: [HACKERS] upper planner path-ification Hi, I've been

Re: [HACKERS] upper planner path-ification

2015-06-22 Thread David Rowley
On 23 June 2015 at 13:55, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Once we support to add aggregation path during path consideration, we need to pay attention morphing of the final target-list according to the intermediate path combination, tentatively chosen. For example, if

Re: [HACKERS] upper planner path-ification

2015-05-19 Thread Andrew Gierth
Tom == Tom Lane t...@sss.pgh.pa.us writes: Tom Hm. That's a hangover from when query_planner also gave back a Tom Plan (singular) rather than a set of Paths. I don't see any Tom fundamental reason why we couldn't generalize it to be a list of Tom potentially useful output orderings rather

Re: [HACKERS] upper planner path-ification

2015-05-19 Thread Kyotaro HORIGUCHI
At Tue, 19 May 2015 09:04:00 -0400, Robert Haas robertmh...@gmail.com wrote in CA+TgmobAV3_DS1sXA+PFWkjvX1K-VNiAnMMJrzPfD43g=-4...@mail.gmail.com On Tue, May 19, 2015 at 7:19 AM, Andrew Gierth and...@tao11.riddles.org.uk wrote: Tom == Tom Lane t...@sss.pgh.pa.us writes: Tom Hm. That's a

Re: [HACKERS] upper planner path-ification

2015-05-19 Thread Robert Haas
On Tue, May 19, 2015 at 7:19 AM, Andrew Gierth and...@tao11.riddles.org.uk wrote: Tom == Tom Lane t...@sss.pgh.pa.us writes: Tom Hm. That's a hangover from when query_planner also gave back a Tom Plan (singular) rather than a set of Paths. I don't see any Tom fundamental reason why we

Re: [HACKERS] upper planner path-ification

2015-05-18 Thread Tom Lane
Andrew Gierth and...@tao11.riddles.org.uk writes: Incidentally, the most obvious obstacle to better planning of grouping sets in the sorted cases is not so much how to pick paths in grouping_planner itself, but rather the fact that query_planner wants to be given only one sort order. Is there

Re: [HACKERS] upper planner path-ification

2015-05-18 Thread Andrew Gierth
Tom == Tom Lane t...@sss.pgh.pa.us writes: Hrm, ok. So for the near future, we should leave it more or less as-is? We don't have a timescale yet, but it's our intention to submit a hashagg support patch for grouping sets as soon as time permits. Tom Well, mumble. I keep saying that I

Re: [HACKERS] upper planner path-ification

2015-05-18 Thread Robert Haas
On Sun, May 17, 2015 at 12:11 PM, Tom Lane t...@sss.pgh.pa.us wrote: Rather than adding tlists per se to Paths, I've been vaguely toying with a notion of identifying all the interesting subexpressions in a query (expensive functions, aggregates, etc), giving them indexes 1..n, and then marking

Re: [HACKERS] upper planner path-ification

2015-05-18 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Sun, May 17, 2015 at 12:11 PM, Tom Lane t...@sss.pgh.pa.us wrote: Rather than adding tlists per se to Paths, I've been vaguely toying with a notion of identifying all the interesting subexpressions in a query (expensive functions, aggregates, etc),

Re: [HACKERS] upper planner path-ification

2015-05-18 Thread Simon Riggs
On 18 May 2015 at 14:50, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Sun, May 17, 2015 at 12:11 PM, Tom Lane t...@sss.pgh.pa.us wrote: Rather than adding tlists per se to Paths, I've been vaguely toying with a notion of identifying all the interesting

Re: [HACKERS] upper planner path-ification

2015-05-18 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On 18 May 2015 at 14:50, Tom Lane t...@sss.pgh.pa.us wrote: So for the moment, let's assume that we still rigidly follow the sequence of upper-level steps currently embodied in grouping_planner. (I'm not sure if it even makes sense to consider other

Re: [HACKERS] upper planner path-ification

2015-05-18 Thread Robert Haas
On Mon, May 18, 2015 at 2:50 PM, Tom Lane t...@sss.pgh.pa.us wrote: I don't know, but it seems like this might be pulling in the opposite direction from your previously-stated desire to get subquery_planner to output Paths rather than Plans as soon as possible. Sorry, I didn't mean to suggest

Re: [HACKERS] upper planner path-ification

2015-05-17 Thread Andrew Gierth
Robert == Robert Haas robertmh...@gmail.com writes: Robert I think grouping_planner() is badly in need of some refactoring Robert just to make it shorter. It's over 1000 lines of code, which Robert IMHO is a fairly ridiculous length for a single function. If there's interest, we could do

Re: [HACKERS] upper planner path-ification

2015-05-17 Thread Tom Lane
Andrew Gierth and...@tao11.riddles.org.uk writes: Robert == Robert Haas robertmh...@gmail.com writes: Robert I think grouping_planner() is badly in need of some refactoring Robert just to make it shorter. It's over 1000 lines of code, which Robert IMHO is a fairly ridiculous length for a

Re: [HACKERS] upper planner path-ification

2015-05-17 Thread Andrew Gierth
Tom == Tom Lane t...@sss.pgh.pa.us writes: If there's interest, we could do that specific task as part of adding hashagg support for grouping sets (which would otherwise make it even longer), or as preparatory work for that. Tom I think that refactoring without changing anything about

Re: [HACKERS] upper planner path-ification

2015-05-17 Thread Tom Lane
Andrew Gierth and...@tao11.riddles.org.uk writes: Tom == Tom Lane t...@sss.pgh.pa.us writes: Tom So I'm all for refactoring, but I think it will happen as a natural Tom byproduct of path-ification, and otherwise would be rather forced. Hrm, ok. So for the near future, we should leave it

Re: [HACKERS] upper planner path-ification

2015-05-17 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: So, getting back to this part, what's the value of returning a list of Paths rather than a list of Plans? (1) less work, since we don't have to fill in details not needed for costing purposes; (2) paths carry info that the planner wants but the

Re: [HACKERS] upper planner path-ification

2015-05-16 Thread Robert Haas
On Wed, May 13, 2015 at 10:27 PM, Tom Lane t...@sss.pgh.pa.us wrote: For the reasons I mentioned, I'd like to get to a point where subquery_planner's output is Paths not Plans as soon as possible. But the idea of coarse representation of steps that we aren't trying to be smart about might be

Re: [HACKERS] upper planner path-ification

2015-05-14 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Wed, May 13, 2015 at 10:27 PM, Tom Lane t...@sss.pgh.pa.us wrote: For the reasons I mentioned, I'd like to get to a point where subquery_planner's output is Paths not Plans as soon as possible. But the idea of coarse representation of steps that we

Re: [HACKERS] upper planner path-ification

2015-05-14 Thread Robert Haas
On Thu, May 14, 2015 at 10:54 AM, Tom Lane t...@sss.pgh.pa.us wrote: In any case, the key question if we're to have Paths representing higher-level computations is what do we hang our lists of such Paths off of?. Yeah, I was wondering about that, too. If we have say both GROUP BY and LIMIT,

Re: [HACKERS] upper planner path-ification

2015-05-14 Thread Martijn van Oosterhout
On Thu, May 14, 2015 at 12:19:44PM -0400, Robert Haas wrote: Well, I'm just shooting from the hip here, but it seems to me that the basic pipeline as it exists today is Join - Aggregate - SetOp - Limit - LockRows. I don't think Limit or LockRows can be moved any earlier. SetOps have a lot in

Re: [HACKERS] upper planner path-ification

2015-05-14 Thread Ashutosh Bapat
On Thu, May 14, 2015 at 7:09 AM, Robert Haas robertmh...@gmail.com wrote: Hi, I've been pulling over Tom's occasional remarks about redoing grouping_planner - and maybe further layers of the planner - to work with Paths instead of Plans. I've had difficulty locating all of the relevant

Re: [HACKERS] upper planner path-ification

2015-05-14 Thread Ashutosh Bapat
On Thu, May 14, 2015 at 8:24 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Wed, May 13, 2015 at 10:27 PM, Tom Lane t...@sss.pgh.pa.us wrote: For the reasons I mentioned, I'd like to get to a point where subquery_planner's output is Paths not Plans as

Re: [HACKERS] upper planner path-ification

2015-05-14 Thread Kyotaro HORIGUCHI
Hello, this topic lured me on.. At Wed, 13 May 2015 23:43:57 -0400, Robert Haas robertmh...@gmail.com wrote in ca+tgmoaqa6bcasgcl8toxwmmoom-d7ebesadz4y58cb+tjq...@mail.gmail.com On Wed, May 13, 2015 at 10:27 PM, Tom Lane t...@sss.pgh.pa.us wrote: Both of those are problems all right, but

Re: [HACKERS] upper planner path-ification

2015-05-13 Thread Robert Haas
On Wed, May 13, 2015 at 10:27 PM, Tom Lane t...@sss.pgh.pa.us wrote: Both of those are problems all right, but there is more context here. Thanks for providing the context. I'm inclined to think that it would be useful to solve the first problem even if we didn't solve the second one right

Re: [HACKERS] upper planner path-ification

2015-05-13 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I've been pulling over Tom's occasional remarks about redoing grouping_planner - and maybe further layers of the planner - to work with Paths instead of Plans. ... I think there are two separate problems here. First, there's the problem that