Re: Ordered Partitioned Table Scans

2019-04-07 Thread Amit Langote
On 2019/04/06 18:06, Julien Rouhaud wrote: > On Sat, Apr 6, 2019 at 2:45 AM David Rowley > wrote: >> >> On Sat, 6 Apr 2019 at 12:26, Tom Lane wrote: >>> Pushed with some hacking, mostly trying to improve the comments. >> >> Great! Many thanks for improving those and pushing it. >> >> Many thanks

Re: Ordered Partitioned Table Scans

2019-04-06 Thread Julien Rouhaud
On Sat, Apr 6, 2019 at 2:45 AM David Rowley wrote: > > On Sat, 6 Apr 2019 at 12:26, Tom Lane wrote: > > Pushed with some hacking, mostly trying to improve the comments. > > Great! Many thanks for improving those and pushing it. > > Many thanks to Julien, Antonin for their detailed reviews on

Re: Ordered Partitioned Table Scans

2019-04-05 Thread David Rowley
On Sat, 6 Apr 2019 at 12:26, Tom Lane wrote: > Pushed with some hacking, mostly trying to improve the comments. Great! Many thanks for improving those and pushing it. Many thanks to Julien, Antonin for their detailed reviews on this. Thanks Amit for your input on this as well. Much appreciated.

Re: Ordered Partitioned Table Scans

2019-04-05 Thread Tom Lane
David Rowley writes: > On Wed, 3 Apr 2019 at 14:01, Amit Langote > wrote: >> I don't have any more comments. > Great. Many thanks for having a look at this. Going by [1], Julien > also seems pretty happy, so I'm going to change this over to ready for > committer. Pushed with some hacking,

Re: Ordered Partitioned Table Scans

2019-04-03 Thread Julien Rouhaud
Hi, On Wed, Apr 3, 2019 at 7:47 AM David Rowley wrote: > > On Wed, 3 Apr 2019 at 14:01, Amit Langote > wrote: > > I don't have any more comments. > > Great. Many thanks for having a look at this. Going by [1], Julien > also seems pretty happy, so I'm going to change this over to ready for >

Re: Ordered Partitioned Table Scans

2019-04-02 Thread David Rowley
On Wed, 3 Apr 2019 at 14:01, Amit Langote wrote: > I don't have any more comments. Great. Many thanks for having a look at this. Going by [1], Julien also seems pretty happy, so I'm going to change this over to ready for committer. [1]

Re: Ordered Partitioned Table Scans

2019-04-02 Thread Amit Langote
On 2019/04/03 3:10, David Rowley wrote: > On Wed, 3 Apr 2019 at 01:26, Amit Langote wrote: >> +#include "nodes/pathnodes.h" >> ... >> +partitions_are_ordered(struct RelOptInfo *partrel) >> >> Maybe, "struct" is unnecessary? > > I just left it there so that the signature matched the header file.

Re: Ordered Partitioned Table Scans

2019-04-02 Thread David Rowley
On Wed, 3 Apr 2019 at 01:26, Amit Langote wrote: > +#include "nodes/pathnodes.h" > ... > +partitions_are_ordered(struct RelOptInfo *partrel) > > Maybe, "struct" is unnecessary? I just left it there so that the signature matched the header file. Looking around for examples I see

Re: Ordered Partitioned Table Scans

2019-04-02 Thread Amit Langote
Hi David, On Tue, Apr 2, 2019 at 8:49 PM David Rowley wrote: > I ended up rewording the entire thing and working on the header > comment for the function too. I think previously it wasn't that well > defined what "ordered" meant. I added a mention that we expect that > NULLs, if possible must

Re: Ordered Partitioned Table Scans

2019-04-02 Thread David Rowley
On Tue, 2 Apr 2019 at 14:26, Amit Langote wrote: > How about extending the sentence about when the optimization is disabled > as follows: > > "If we find a Datum in a partition that's greater than one previously > already seen, then values could become out of order and we'd have to > disable the

Re: Ordered Partitioned Table Scans

2019-04-01 Thread Amit Langote
Hi, On 2019/03/29 7:59, David Rowley wrote: > On Fri, 29 Mar 2019 at 00:00, Amit Langote > wrote: >> >> On 2019/03/28 8:04, David Rowley wrote: >>> If it's *always* scanned last then it's fine for ORDER BY partkey >>> NULLS LAST. If they have ORDER BY partkey NULLS FIRST then we won't >>> match

Re: Ordered Partitioned Table Scans

2019-03-28 Thread David Rowley
On Fri, 29 Mar 2019 at 00:00, Amit Langote wrote: > > On 2019/03/28 8:04, David Rowley wrote: > > If it's *always* scanned last then it's fine for ORDER BY partkey > > NULLS LAST. If they have ORDER BY partkey NULLS FIRST then we won't > > match on the pathkeys. > > Sorry, I had meant to say

Re: Ordered Partitioned Table Scans

2019-03-28 Thread Amit Langote
Hi David, On 2019/03/28 8:04, David Rowley wrote: > On Wed, 27 Mar 2019 at 21:24, Amit Langote > wrote: >> Noticed a typo. >> >> + * multiple subpaths then we can't make guarantees about the >> + * order tuples in those subpaths, so we must leave the >> >> order

Re: Ordered Partitioned Table Scans

2019-03-27 Thread David Rowley
On Thu, 28 Mar 2019 at 15:40, David Rowley wrote: > Thanks for the review. I've attached a patch that mostly just moved > the code around. Here's one that fixes up the compiler warning from the last one. Thanks CF bot... -- David Rowley http://www.2ndQuadrant.com/

Re: Ordered Partitioned Table Scans

2019-03-27 Thread David Rowley
On Thu, 28 Mar 2019 at 14:34, Amit Langote wrote: > > On 2019/03/28 7:29, David Rowley wrote: > > On Wed, 27 Mar 2019 at 19:48, Amit Langote > > It does need to know how many partitions the partitioned table has, > > which it gets from partrel->nparts, so yeah, RelOptInfo is probably > > needed.

Re: Ordered Partitioned Table Scans

2019-03-27 Thread Amit Langote
Hi, On 2019/03/28 7:29, David Rowley wrote: > On Wed, 27 Mar 2019 at 19:48, Amit Langote > wrote: >> Sorry if this was discussed before, but why does this patch add any new >> code to partprune.c? AFAICT, there's no functionality changes to the >> pruning code. > > You're right. It probably

Re: Ordered Partitioned Table Scans

2019-03-27 Thread David Rowley
On Wed, 27 Mar 2019 at 21:24, Amit Langote wrote: > Noticed a typo. > > + * multiple subpaths then we can't make guarantees about the > + * order tuples in those subpaths, so we must leave the > > order of tuples? I'll fix that. Thanks. > Again, sorry if this was

Re: Ordered Partitioned Table Scans

2019-03-27 Thread David Rowley
On Wed, 27 Mar 2019 at 19:48, Amit Langote wrote: > Sorry if this was discussed before, but why does this patch add any new > code to partprune.c? AFAICT, there's no functionality changes to the > pruning code. You're right. It probably shouldn't be there. There's a bit of a lack of a good

Re: Ordered Partitioned Table Scans

2019-03-27 Thread Amit Langote
On 2019/03/27 15:48, Amit Langote wrote: > Hi David, > > Sorry if this was discussed before, but why does this patch add any new > code to partprune.c? AFAICT, there's no functionality changes to the > pruning code. > > Both > > +bool > +partkey_is_bool_constant_for_query(RelOptInfo *partrel,

Re: Ordered Partitioned Table Scans

2019-03-27 Thread Amit Langote
Hi David, Sorry if this was discussed before, but why does this patch add any new code to partprune.c? AFAICT, there's no functionality changes to the pruning code. Both +bool +partkey_is_bool_constant_for_query(RelOptInfo *partrel, int partkeycol) and +static bool

Re: Ordered Partitioned Table Scans

2019-03-26 Thread David Rowley
Thanks for having another look. On Wed, 27 Mar 2019 at 00:22, Julien Rouhaud wrote: > A few, mostly nitpicking, comments: > > + if (rel->part_scheme != NULL && IS_SIMPLE_REL(rel) && > + partitions_are_ordered(root, rel)) > > shouldn't the test be IS_PARTITIONED_REL(rel) instead of

Re: Ordered Partitioned Table Scans

2019-03-26 Thread Julien Rouhaud
On Tue, Mar 26, 2019 at 3:13 AM David Rowley wrote: > > On Tue, 26 Mar 2019 at 09:02, Julien Rouhaud wrote: > > FTR this patch doesn't apply since single child [Merge]Append > > suppression (8edd0e7946) has been pushed. > > Thanks for letting me know. I've attached v14 based on current master.

Re: Ordered Partitioned Table Scans

2019-03-25 Thread David Rowley
On Tue, 26 Mar 2019 at 09:02, Julien Rouhaud wrote: > FTR this patch doesn't apply since single child [Merge]Append > suppression (8edd0e7946) has been pushed. Thanks for letting me know. I've attached v14 based on current master. -- David Rowley http://www.2ndQuadrant.com/

Re: Ordered Partitioned Table Scans

2019-03-25 Thread Julien Rouhaud
On Sun, Mar 24, 2019 at 11:06 AM David Rowley wrote: > > On Sat, 23 Mar 2019 at 19:42, Tom Lane wrote: > > > > David Rowley writes: > > > On Sat, 23 Mar 2019 at 05:40, Tom Lane wrote: > > >> BTW, another thing we could possibly do to answer this objection is to > > >> give the ordered-Append

Re: Ordered Partitioned Table Scans

2019-03-24 Thread David Rowley
On Sat, 23 Mar 2019 at 19:42, Tom Lane wrote: > > David Rowley writes: > > On Sat, 23 Mar 2019 at 05:40, Tom Lane wrote: > >> BTW, another thing we could possibly do to answer this objection is to > >> give the ordered-Append node an artificially pessimistic startup cost, > >> such as the sum

Re: Ordered Partitioned Table Scans

2019-03-24 Thread David Rowley
On Sun, 24 Mar 2019 at 05:16, Julien Rouhaud wrote: > ISTM that a query like > SELECT * FROM nested ORDER BY 1, 2; > could simply append all the partitions in the right order (or generate > a tree of ordered appends), but: > > QUERY PLAN >

Re: Ordered Partitioned Table Scans

2019-03-23 Thread Julien Rouhaud
On Wed, Dec 19, 2018 at 3:01 PM David Rowley wrote: > > On Thu, 20 Dec 2018 at 01:58, Julien Rouhaud wrote: > > > > The multi-level partitioning case is another > > thing that would need to be handled for instance (and that's the main > > reason I couldn't submit a new patch when I was working

Re: Ordered Partitioned Table Scans

2019-03-23 Thread Tom Lane
David Rowley writes: > On Sat, 23 Mar 2019 at 05:40, Tom Lane wrote: >> BTW, another thing we could possibly do to answer this objection is to >> give the ordered-Append node an artificially pessimistic startup cost, >> such as the sum or the max of its children's startup costs. That's >>

Re: Ordered Partitioned Table Scans

2019-03-22 Thread David Rowley
On Sat, 23 Mar 2019 at 05:40, Tom Lane wrote: > BTW, another thing we could possibly do to answer this objection is to > give the ordered-Append node an artificially pessimistic startup cost, > such as the sum or the max of its children's startup costs. That's > pretty ugly and unprincipled, but

Re: Ordered Partitioned Table Scans

2019-03-22 Thread David Rowley
On Sat, 23 Mar 2019 at 04:56, Tom Lane wrote: > > David Rowley writes: > > Append has the additional > > saving of not having to determine to perform a sort on the top row > > from each subpath. > > Uh, what? sorted-Append and MergeAppend would need pre-sorts on > exactly the same set of

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Julien Rouhaud
On Fri, Mar 22, 2019 at 7:19 PM Robert Haas wrote: > > On Fri, Mar 22, 2019 at 12:40 PM Tom Lane wrote: > > Robert Haas writes: > > > On Fri, Mar 22, 2019 at 11:56 AM Tom Lane wrote: > > >> In cases where, say, the first child requires no sort but also doesn't > > >> emit very many rows, while

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Robert Haas
On Fri, Mar 22, 2019 at 12:40 PM Tom Lane wrote: > Robert Haas writes: > > On Fri, Mar 22, 2019 at 11:56 AM Tom Lane wrote: > >> In cases where, say, the first child requires no sort but also doesn't > >> emit very many rows, while the second child requires an expensive sort, > >> the planner

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Tom Lane
Robert Haas writes: > On Fri, Mar 22, 2019 at 11:56 AM Tom Lane wrote: >> In cases where, say, the first child requires no sort but also doesn't >> emit very many rows, while the second child requires an expensive sort, >> the planner will have a ridiculously optimistic opinion of the cost of >>

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Robert Haas
On Fri, Mar 22, 2019 at 12:21 PM Tom Lane wrote: > Once again: this objection is not a "death sentence for this patch". > I simply wish to suppress the option to generate an ordered Append > when some of the inputs would require an added sort step. As long > as we have pre-ordered paths for all

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Tom Lane
Robert Haas writes: > On Fri, Mar 22, 2019 at 11:56 AM Tom Lane wrote: >> In cases where, say, the first child requires no sort but also doesn't >> emit very many rows, while the second child requires an expensive sort, >> the planner will have a ridiculously optimistic opinion of the cost of >>

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Robert Haas
On Fri, Mar 22, 2019 at 11:56 AM Tom Lane wrote: > In cases where, say, the first child requires no sort but also doesn't > emit very many rows, while the second child requires an expensive sort, > the planner will have a ridiculously optimistic opinion of the cost of > fetching slightly more

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Tom Lane
David Rowley writes: > Thanks for explaining. I see where you're coming from now. I think > this point would carry more weight if using the Append instead of the > MergeAppend were worse in some cases as we could end up using an > inferior plan accidentally. However, that's not the case. The

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Simon Riggs
On Fri, 22 Mar 2019 at 11:39, Tom Lane wrote: > Simon Riggs writes: > > I agree that the issue of mixing sorts at various points will make > nonsense > > of the startup cost/total cost results. > > Right. > > > I don't see LIMIT costing being broken as a reason to restrict this > >

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Tom Lane
Simon Riggs writes: > I agree that the issue of mixing sorts at various points will make nonsense > of the startup cost/total cost results. Right. > I don't see LIMIT costing being broken as a reason to restrict this > optimization. I would ask that we allow improvements to the important use >

Re: Ordered Partitioned Table Scans

2019-03-22 Thread David Rowley
On Sat, 23 Mar 2019 at 04:12, Tom Lane wrote: > The reason why I'm skeptical about LIMIT with a plan of the form > append-some-sorts-together is that there are going to be large > discontinuities in the cost-vs-number-of-rows-returned graph, > ie, every time you finish one child plan and start

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Simon Riggs
On Fri, 22 Mar 2019 at 11:12, Tom Lane wrote: > David Rowley writes: > > On Sat, 9 Mar 2019 at 10:52, Tom Lane wrote: > >>> This can be a huge win for queries of the form "ORDER BY partkey LIMIT > >>> x". Even if the first subpath(s) aren't natively ordered, not all of > >>> the sorts should

Re: Ordered Partitioned Table Scans

2019-03-22 Thread Tom Lane
David Rowley writes: > On Sat, 9 Mar 2019 at 10:52, Tom Lane wrote: >>> This can be a huge win for queries of the form "ORDER BY partkey LIMIT >>> x". Even if the first subpath(s) aren't natively ordered, not all of >>> the sorts should actually be performed. >> [ shrug... ] We've got no

Re: Ordered Partitioned Table Scans

2019-03-22 Thread David Rowley
On Sat, 9 Mar 2019 at 10:52, Tom Lane wrote: > > Julien Rouhaud writes: > > On Fri, Mar 8, 2019 at 9:15 PM Tom Lane wrote: > >> I think you should remove all that > >> and restrict this optimization to the case where all the subpaths are > >> natively ordered --- if we have to insert Sorts,

Re: Ordered Partitioned Table Scans

2019-03-15 Thread David Rowley
On Sat, 16 Mar 2019 at 04:22, David Rowley wrote: > I've attached an updated patch which fixes the conflict with 0a9d7e1f6d8 ... and here's the one that I should have sent. (renamed to v12 to prevent confusion) -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL

Re: Ordered Partitioned Table Scans

2019-03-15 Thread Robert Haas
On Fri, Mar 8, 2019 at 3:15 PM Tom Lane wrote: > I took a quick look through this and I'm not very happy with it. > It seems to me that the premise ought to be "just use an Append > if we can prove the output would be ordered anyway", but that's not > what we actually have here: instead you're

Re: Ordered Partitioned Table Scans

2019-03-15 Thread David Rowley
I've attached an updated patch which fixes the conflict with 0a9d7e1f6d8 -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services v11-0001-Allow-Append-to-be-used-in-place-of-MergeAppend-.patch Description: Binary data

Re: Ordered Partitioned Table Scans

2019-03-08 Thread David Rowley
On Sat, 9 Mar 2019 at 09:14, Tom Lane wrote: > I took a quick look through this and I'm not very happy with it. > It seems to me that the premise ought to be "just use an Append > if we can prove the output would be ordered anyway", but that's not > what we actually have here: instead you're

Re: Ordered Partitioned Table Scans

2019-03-08 Thread David Rowley
On Sat, 9 Mar 2019 at 10:52, Tom Lane wrote: > > Julien Rouhaud writes: > > On Fri, Mar 8, 2019 at 9:15 PM Tom Lane wrote: > >> I think you should remove all that > >> and restrict this optimization to the case where all the subpaths are > >> natively ordered --- if we have to insert Sorts,

Re: Ordered Partitioned Table Scans

2019-03-08 Thread Tom Lane
Julien Rouhaud writes: > On Fri, Mar 8, 2019 at 9:15 PM Tom Lane wrote: >> I think you should remove all that >> and restrict this optimization to the case where all the subpaths are >> natively ordered --- if we have to insert Sorts, it's hardly going to move >> the needle to worry about

Re: Ordered Partitioned Table Scans

2019-03-08 Thread Julien Rouhaud
On Fri, Mar 8, 2019 at 9:15 PM Tom Lane wrote: > > David Rowley writes: > > [ v9-0001-Allow-Append-to-be-used-in-place-of-MergeAppend-f.patch ] > > I think you should remove all that > and restrict this optimization to the case where all the subpaths are > natively ordered --- if we have to

Re: Ordered Partitioned Table Scans

2019-03-08 Thread Tom Lane
David Rowley writes: > [ v9-0001-Allow-Append-to-be-used-in-place-of-MergeAppend-f.patch ] I took a quick look through this and I'm not very happy with it. It seems to me that the premise ought to be "just use an Append if we can prove the output would be ordered anyway", but that's not what we

Re: Ordered Partitioned Table Scans

2019-03-05 Thread David Rowley
On Wed, 6 Mar 2019 at 07:17, Robert Haas wrote: > > On Wed, Dec 19, 2018 at 5:08 PM David Rowley > wrote: > > With my idea for using live_parts, we'll process the partitions > > looking for interleaved values on each query, after pruning takes > > place. In this case, we'll see the partitions

Re: Ordered Partitioned Table Scans

2019-03-05 Thread Robert Haas
On Wed, Dec 19, 2018 at 5:08 PM David Rowley wrote: > With my idea for using live_parts, we'll process the partitions > looking for interleaved values on each query, after pruning takes > place. In this case, we'll see the partitions are naturally ordered. I > don't really foresee any issues with

Re: Ordered Partitioned Table Scans

2019-03-04 Thread David Rowley
Thanks a lot for taking the time to look at this. On Tue, 5 Mar 2019 at 03:03, Antonin Houska wrote: > As for the latest version (v8-0001-...) I've only caught a small typo: "When > the first subpath needs sorted ...". It was probably meant "... needs sort > ...". That was a sort of short way

Re: Ordered Partitioned Table Scans

2019-03-04 Thread Antonin Houska
David Rowley wrote: > On Mon, 5 Nov 2018 at 10:46, David Rowley > wrote: > > On 1 November 2018 at 22:05, Antonin Houska wrote: > > > I think these conditions are too restrictive: > > > > > > /* > > > * Determine if these pathkeys match the partition order, or > > > reverse

Re: Ordered Partitioned Table Scans

2019-02-23 Thread David Rowley
On Thu, 31 Jan 2019 at 16:29, David Rowley wrote: > I've attached a rebased patch which fixes up the recent conflicts. No > other changes were made. Rebased version due to a new call to create_append_path() added in ab5fcf2b0. No other changes made. -- David Rowley

Re: Ordered Partitioned Table Scans

2019-02-03 Thread Michael Paquier
On Thu, Jan 31, 2019 at 04:29:56PM +1300, David Rowley wrote: > I've attached a rebased patch which fixes up the recent conflicts. No > other changes were made. Moved to next CF, waiting for review. -- Michael signature.asc Description: PGP signature

Re: Ordered Partitioned Table Scans

2019-01-30 Thread David Rowley
I've attached a rebased patch which fixes up the recent conflicts. No other changes were made. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services v7-0001-Allow-Append-to-be-used-in-place-of-MergeAppend-f.patch Description:

Re: Ordered Partitioned Table Scans

2019-01-06 Thread Julien Rouhaud
On Sun, Jan 6, 2019 at 4:24 AM David Rowley wrote: > > On Thu, 20 Dec 2018 at 18:20, Julien Rouhaud wrote: > > > > > > No, what I'm proposing is to store if the partitions are naturally > > ordered or not, *and* recheck after pruning if that property could > > have changed (so if some partitions

Re: Ordered Partitioned Table Scans

2019-01-05 Thread David Rowley
On Thu, 20 Dec 2018 at 18:20, Julien Rouhaud wrote: > > On Wed, Dec 19, 2018 at 11:08 PM David Rowley > wrote: > > create table listp (a int) partition by list (a); > > create table listp12 partition of listp for values in(1,2); > > create table listp03 partition of listp for vlaues in(0,3); > >

Re: Ordered Partitioned Table Scans

2018-12-19 Thread Julien Rouhaud
On Wed, Dec 19, 2018 at 11:08 PM David Rowley wrote: > > On Thu, 20 Dec 2018 at 09:48, Julien Rouhaud wrote: > > On Wed, Dec 19, 2018 at 3:01 PM David Rowley > > wrote: > > > If so, I'd question why the default partition > > > is so special? Pruning of any of the other partitions could turn a

Re: Ordered Partitioned Table Scans

2018-12-19 Thread David Rowley
On Thu, 20 Dec 2018 at 09:48, Julien Rouhaud wrote: > On Wed, Dec 19, 2018 at 3:01 PM David Rowley > wrote: > > If so, I'd question why the default partition > > is so special? Pruning of any of the other partitions could turn a > > naturally unordered LIST partitioned table into a naturally

Re: Ordered Partitioned Table Scans

2018-12-19 Thread Julien Rouhaud
On Wed, Dec 19, 2018 at 3:01 PM David Rowley wrote: > > On Thu, 20 Dec 2018 at 01:58, Julien Rouhaud wrote: > > I'm objecting to processing for all partitions, but processing for > just non-pruned partitions seems fine to me. If there are 10k > partitions and we pruned none of them, then

Re: Ordered Partitioned Table Scans

2018-12-19 Thread David Rowley
On Thu, 20 Dec 2018 at 01:58, Julien Rouhaud wrote: > I don't see why we should drop this flag. If we know that the > partitions are naturally ordered, they'll still be ordered after some > partitions have been prune, so we can skip later checks if we already > have the information. The only

Re: Ordered Partitioned Table Scans

2018-12-19 Thread Julien Rouhaud
On Wed, Dec 19, 2018 at 1:18 PM David Rowley wrote: > > On Wed, 19 Dec 2018 at 23:25, Julien Rouhaud wrote: > > > > I see. But since for now the optimisation will only be done > > considering all partitions, I still think that it's better to store a > > bool flag in the PartitionDesc to

Re: Ordered Partitioned Table Scans

2018-12-19 Thread David Rowley
On Wed, 19 Dec 2018 at 23:25, Julien Rouhaud wrote: > On Wed, Dec 19, 2018 at 10:51 AM David Rowley > wrote: > > The reason I'm keen to leave this alone today is that determining > > which partitions are pruned requires looking at each partition's > > RelOptInfo and checking if it's marked as a

Re: Ordered Partitioned Table Scans

2018-12-19 Thread Julien Rouhaud
On Wed, Dec 19, 2018 at 10:51 AM David Rowley wrote: > > On Wed, 19 Dec 2018 at 20:40, Julien Rouhaud wrote: > > > If I understand correctly, the new behavior is controlled by > > partitions_are_ordered(), but it only checks for declared partitions, > > not partitions that survived pruning. Did

Re: Ordered Partitioned Table Scans

2018-12-19 Thread David Rowley
On Wed, 19 Dec 2018 at 20:40, Julien Rouhaud wrote: > I started to look at v5. Thanks for giving this a look over. > If I understand correctly, the new behavior is controlled by > partitions_are_ordered(), but it only checks for declared partitions, > not partitions that survived pruning. Did

Re: Ordered Partitioned Table Scans

2018-11-22 Thread David Rowley
On Mon, 5 Nov 2018 at 10:46, David Rowley wrote: > On 1 November 2018 at 22:05, Antonin Houska wrote: > > I think these conditions are too restrictive: > > > > /* > > * Determine if these pathkeys match the partition order, or reverse > > * partition order. It can't

Re: Ordered Partitioned Table Scans

2018-11-04 Thread David Rowley
On 1 November 2018 at 22:05, Antonin Houska wrote: > I think these conditions are too restrictive: > > /* > * Determine if these pathkeys match the partition order, or reverse > * partition order. It can't match both, so only go to the trouble of > * checking

Re: Ordered Partitioned Table Scans

2018-11-01 Thread Antonin Houska
David Rowley wrote: > On 1 November 2018 at 04:01, Antonin Houska wrote: > > * As for the logic, I found generate_mergeappend_paths() to be the most > > interesting part: > > > > Imagine table partitioned by "i", so "partition_pathkeys" is {i}. > > > > partition 1: > > > > i | j > > --+-- > >

Re: Ordered Partitioned Table Scans

2018-10-31 Thread David Rowley
On 1 November 2018 at 04:01, Antonin Houska wrote: > * As for the logic, I found generate_mergeappend_paths() to be the most > interesting part: > > Imagine table partitioned by "i", so "partition_pathkeys" is {i}. > > partition 1: > > i | j > --+-- > 0 | 0 > 1 | 1 > 0 | 1 > 1 | 0 > > partition

Re: Ordered Partitioned Table Scans

2018-10-31 Thread Antonin Houska
David Rowley wrote: > On 31 October 2018 at 13:05, David Rowley > wrote: > >>> On 28 October 2018 at 03:49, Julien Rouhaud wrote: > >> I've registered as a reviewer. I still didn't have a deep look at > >> the patch yet, but thanks a lot for working on it! > > > > Thanks for signing up to

Re: Ordered Partitioned Table Scans

2018-10-31 Thread David Rowley
On 31 October 2018 at 13:05, David Rowley wrote: >>> On 28 October 2018 at 03:49, Julien Rouhaud wrote: >> I've registered as a reviewer. I still didn't have a deep look at >> the patch yet, but thanks a lot for working on it! > > Thanks for signing up to review. I need to send another

Re: Ordered Partitioned Table Scans

2018-10-30 Thread David Rowley
On 31 October 2018 at 12:24, Julien Rouhaud wrote: > On Mon, Oct 29, 2018 at 1:44 AM David Rowley > wrote: >> >> On 28 October 2018 at 03:49, Julien Rouhaud wrote: >> > I just had a look at your patch. I see that you implemented only a >> > subset of the possible optimizations (only the case

Re: Ordered Partitioned Table Scans

2018-10-30 Thread Julien Rouhaud
On Mon, Oct 29, 2018 at 1:44 AM David Rowley wrote: > > On 28 October 2018 at 03:49, Julien Rouhaud wrote: > > I just had a look at your patch. I see that you implemented only a > > subset of the possible optimizations (only the case for range > > partitionoing without subpartitions). This has

Re: Ordered Partitioned Table Scans

2018-10-29 Thread David Rowley
On 29 October 2018 at 13:44, David Rowley wrote: > v2 of the patch is attached. I've not had time yet to give it a > throughout post write review, but on first look it seems okay. Added to the November 'fest. https://commitfest.postgresql.org/20/1850/ -- David Rowley

Re: Ordered Partitioned Table Scans

2018-10-28 Thread David Rowley
Thanks for looking at this. On 28 October 2018 at 03:49, Julien Rouhaud wrote: > I just had a look at your patch. I see that you implemented only a > subset of the possible optimizations (only the case for range > partitionoing without subpartitions). This has been previously > discussed, but

Re: Ordered Partitioned Table Scans

2018-10-27 Thread Julien Rouhaud
Hi, On Fri, Oct 26, 2018 at 1:01 PM Julien Rouhaud wrote: > On Fri, Oct 26, 2018 at 6:40 AM David Rowley > wrote: > > > > On 26 October 2018 at 16:52, Amit Langote > > wrote: > > > I recall Ronan Dunklau and Julien Rouhaud had proposed a patch for this > > > last year, but the

Re: Ordered Partitioned Table Scans

2018-10-26 Thread Julien Rouhaud
Hi, On Fri, Oct 26, 2018 at 6:40 AM David Rowley wrote: > > On 26 October 2018 at 16:52, Amit Langote > wrote: > > I recall Ronan Dunklau and Julien Rouhaud had proposed a patch for this > > last year, but the partitioning-related planning code hadn't advanced then > > as much as it has today,

Re: Ordered Partitioned Table Scans

2018-10-25 Thread David Rowley
On 26 October 2018 at 16:52, Amit Langote wrote: > I recall Ronan Dunklau and Julien Rouhaud had proposed a patch for this > last year, but the partitioning-related planning code hadn't advanced then > as much as it has today, so they sort of postponed working on it. > Eventually their patch was

Re: Ordered Partitioned Table Scans

2018-10-25 Thread Amit Langote
On 2018/10/26 11:50, David Rowley wrote: > RANGE partitioning of time-series data is quite a common range to use > partitioning, and such tables tend to grow fairly large. I thought > since we always store RANGE partitioned tables in the PartitionDesc in > ascending range order that it might be

Ordered Partitioned Table Scans

2018-10-25 Thread David Rowley
RANGE partitioning of time-series data is quite a common range to use partitioning, and such tables tend to grow fairly large. I thought since we always store RANGE partitioned tables in the PartitionDesc in ascending range order that it might be useful to make use of this and when the required