Re: [HACKERS] Path question

2010-10-14 Thread Tom Lane
I wrote: I rather wonder if we don't want two separate execution-time node types anyway, since what Append does seems significantly different from Merge (and MergeAppend would be just a misnomer). I've been working on this patch, and have gotten the executor side split out as a new node type.

Re: [HACKERS] Path question

2010-10-14 Thread Robert Haas
On Thu, Oct 14, 2010 at 11:34 AM, Tom Lane t...@sss.pgh.pa.us wrote: I wrote: I rather wonder if we don't want two separate execution-time node types anyway, since what Append does seems significantly different from Merge (and MergeAppend would be just a misnomer). I've been working on this

Re: [HACKERS] Path question

2010-10-14 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Thu, Oct 14, 2010 at 11:34 AM, Tom Lane t...@sss.pgh.pa.us wrote: Anybody have a strong feeling about what to call these things? At the moment I'm leaning to sticking with MergeAppend, but if we decide to rename it it'd probably be better to do so

Re: [HACKERS] Path question

2010-10-14 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: So I tried out the logic described in this email and, with a few modifications, it seemed to work. Updated patch attached, any review appreciated. Applied with revisions. 3. TGL: Speaking of sorting, it's not entirely clear to me how the patch

Re: [HACKERS] Path question

2010-10-14 Thread Greg Stark
On Thu, Oct 14, 2010 at 8:34 AM, Tom Lane t...@sss.pgh.pa.us wrote: I did run into a problem with my plan to call the new node type Merge: the planner is already using MergePath as the name for the Path struct that is precursor to a MergeJoin.  For the moment I'm calling the new node type

Re: [HACKERS] Path question

2010-10-13 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: Another awkwardness of this patch is that it makes create_append_path() and consequently set_dummy_rel_pathlist() take an additional root argument. While there's nothing terribly unreasonable about this on its face, it's only necessary so that

Re: [HACKERS] Path question

2010-09-29 Thread Robert Haas
On Tue, Sep 28, 2010 at 11:13 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: ...what makes the pathkeys potentially useful is that they match the root pathkeys for the query.  However, without Greg's hacks, the transposed child equivalence classes don't show

Re: [HACKERS] Path question

2010-09-29 Thread Greg Stark
2010/9/23 Robert Haas robertmh...@gmail.com: All of this leaves me wondering why Greg ended up ifdefing out this code in the first place.  There's probably something I'm missing here...  but for now I can't think of a better idea than just removing the #ifdefs and hoping that whatever problem

Re: [HACKERS] Path question

2010-09-29 Thread Robert Haas
On Wed, Sep 29, 2010 at 11:01 AM, Robert Haas robertmh...@gmail.com wrote: Makes sense to me.  It seems that there are actually two halves to this problem: getting the child EMs to be generated in the first place, and then getting them to be examined at the appropriate time. So I tried out the

Re: [HACKERS] Path question

2010-09-28 Thread Robert Haas
2010/9/23 Robert Haas robertmh...@gmail.com: All of this leaves me wondering why Greg ended up ifdefing out this code in the first place.  There's probably something I'm missing here...  but for now I can't think of a better idea than just removing the #ifdefs and hoping that whatever problem

Re: [HACKERS] Path question

2010-09-28 Thread Robert Haas
2010/9/28 Robert Haas robertmh...@gmail.com: 2010/9/23 Robert Haas robertmh...@gmail.com: All of this leaves me wondering why Greg ended up ifdefing out this code in the first place.  There's probably something I'm missing here...  but for now I can't think of a better idea than just removing

Re: [HACKERS] Path question

2010-09-28 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: ...what makes the pathkeys potentially useful is that they match the root pathkeys for the query. However, without Greg's hacks, the transposed child equivalence classes don't show up in the equivalence member lists, so get_eclass_for_sort_expr()

Re: [HACKERS] Path question

2010-09-24 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: FIXME #1 and FIXME #2 were much harder to trigger. In fact, barring significant further lobotimization of the code, I couldn't. It's not that hard if you just tweak equivclass.c to make the order of equivalence-class lists different, viz diff --git

Re: [HACKERS] Path question

2010-09-24 Thread Robert Haas
On Fri, Sep 24, 2010 at 5:05 PM, Tom Lane t...@sss.pgh.pa.us wrote: It's not that hard if you just tweak equivclass.c to make the order of equivalence-class lists different, viz [...] Since the order of equivalence-class member lists isn't supposed to be semantically significant, I claim that

Re: [HACKERS] Path question

2010-09-23 Thread Robert Haas
On Tue, Sep 21, 2010 at 12:29 AM, David Fetter da...@fetter.org wrote: On Mon, Sep 20, 2010 at 10:57:00PM -0400, Robert Haas wrote: 2010/9/3 Hans-Jürgen Schönig h...@cybertec.at: On Sep 2, 2010, at 1:20 AM, Robert Haas wrote: I agree. Explicit partitioning may open up some additional

Re: [HACKERS] Path question

2010-09-23 Thread Hans-Jürgen Schönig
On Sep 23, 2010, at 3:29 PM, Robert Haas wrote: On Tue, Sep 21, 2010 at 12:29 AM, David Fetter da...@fetter.org wrote: On Mon, Sep 20, 2010 at 10:57:00PM -0400, Robert Haas wrote: 2010/9/3 Hans-Jürgen Schönig h...@cybertec.at: On Sep 2, 2010, at 1:20 AM, Robert Haas wrote: I agree. Explicit

Re: [HACKERS] Path question

2010-09-23 Thread Robert Haas
2010/9/23 Hans-Jürgen Schönig h...@cybertec.at: sorry for not getting back to you sooner. i am currently on the road for some days. we got the top 3 things fixed already. however, some code seems to be relying on a sorted list somewhere(???). we are in the process of sorting out most of the

Re: [HACKERS] Path question

2010-09-23 Thread Robert Haas
2010/9/20 Robert Haas robertmh...@gmail.com: 4. TGL: In any case, I'm amazed that it's not failing regression tests all over the place with those critical tests in make_sort_from_pathkeys lobotomized by random #ifdef FIXMEs.  Perhaps we need some more regression tests  Obviously, we need

Re: [HACKERS] Path question

2010-09-20 Thread Robert Haas
2010/9/3 Hans-Jürgen Schönig h...@cybertec.at: On Sep 2, 2010, at 1:20 AM, Robert Haas wrote: I agree. Explicit partitioning may open up some additional optimization possibilities in certain cases, but Merge Append is more general and extremely valuable in its own right. we have revised

Re: [HACKERS] Path question

2010-09-20 Thread David Fetter
On Mon, Sep 20, 2010 at 10:57:00PM -0400, Robert Haas wrote: 2010/9/3 Hans-Jürgen Schönig h...@cybertec.at: On Sep 2, 2010, at 1:20 AM, Robert Haas wrote: I agree. Explicit partitioning may open up some additional optimization possibilities in certain cases, but Merge Append is more

Re: [HACKERS] Path question

2010-09-03 Thread Hans-Jürgen Schönig
On Sep 2, 2010, at 1:20 AM, Robert Haas wrote: On Sep 1, 2010, at 10:21 AM, Greg Stark gsst...@mit.edu wrote: For what it's worth I disagree with Tom. I think this is a situation where we need *both* types of solution. Ideally we will be able to use a plain Append node for cases where we

[HACKERS] Path question

2010-09-01 Thread Boszormenyi Zoltan
Hi, we are experimenting with modifying table partitioning so the ORDER BY clause can be pushed down to child nodes on the grounds that: 1. smaller datasets are faster to sort, e.g. two datasets that almost spill out to disk are faster to sort in memory and later merge them than the union

Re: [HACKERS] Path question

2010-09-01 Thread Tom Lane
Boszormenyi Zoltan z...@cybertec.at writes: we are experimenting with modifying table partitioning so the ORDER BY clause can be pushed down to child nodes on the grounds that: This is really premature, and anything you do along those lines now will probably never get committed. The problem

Re: [HACKERS] Path question

2010-09-01 Thread Greg Stark
2010/9/1 Boszormenyi Zoltan z...@cybertec.at: we are experimenting with modifying table partitioning so the ORDER BY clause can be pushed down to child nodes on the grounds that: 1. smaller datasets are faster to sort, e.g. two datasets that almost    spill out to disk are faster to sort in

Re: [HACKERS] Path question

2010-09-01 Thread PostgreSQL - Hans-Jürgen Schönig
On Sep 1, 2010, at 4:10 PM, Tom Lane wrote: Boszormenyi Zoltan z...@cybertec.at writes: we are experimenting with modifying table partitioning so the ORDER BY clause can be pushed down to child nodes on the grounds that: This is really premature, and anything you do along those lines now

Re: [HACKERS] Path question

2010-09-01 Thread Tom Lane
=?iso-8859-1?Q?PostgreSQL_-_Hans-J=FCrgen_Sch=F6nig?= postg...@cybertec.at writes: On Sep 1, 2010, at 4:10 PM, Tom Lane wrote: This is really premature, and anything you do along those lines now will probably never get committed. well, why non-overlapping? the idea is to make append smart

Re: [HACKERS] Path question

2010-09-01 Thread PostgreSQL - Hans-Jürgen Schönig
hello tom, yeah, we have followed quite a lot of discussion as well ... and yes, no patches. as far as this problem is concerned: we are working on a patch and did some prototyping inside the planner already (attached). the code we have is pretty limited atm (such as checking for a sort clause

Re: [HACKERS] Path question

2010-09-01 Thread Robert Haas
On Sep 1, 2010, at 10:21 AM, Greg Stark gsst...@mit.edu wrote: For what it's worth I disagree with Tom. I think this is a situation where we need *both* types of solution. Ideally we will be able to use a plain Append node for cases where we know the relative ordering of the data in different