RE: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-11-03 Thread Phil Florent
Hi, Thanks for your work, our prototype runs OK. PostgreSQL 11 and its now fully functional partitioning feature is our validated choice to replace a well-known proprietary RDBMS in 100+ public hospitals for our dss application. Best regards Phil De : Amit

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-08-08 Thread Amit Langote
On 2018/08/09 13:00, Tom Lane wrote: > Amit Langote writes: >> One reason why we should adapt such a test case is that, in the future, we >> may arrange for make_partitionedrel_pruneinfo(), whose code we just fixed, >> to not be called if we know that run-time pruning is not needed. It seems >>

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-08-08 Thread Tom Lane
Amit Langote writes: > One reason why we should adapt such a test case is that, in the future, we > may arrange for make_partitionedrel_pruneinfo(), whose code we just fixed, > to not be called if we know that run-time pruning is not needed. It seems > that that's true for the test added by the

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-08-08 Thread Amit Langote
On 2018/08/09 0:48, Tom Lane wrote: > David Rowley writes: >> On 8 August 2018 at 17:28, Amit Langote >> wrote: >>> Attached is a patch which modifies the if test to compare relids instead >>> of RelOptInfo pointers. > >> Thanks for investigating and writing a patch. I agree with the fix. > >

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-08-08 Thread Tom Lane
David Rowley writes: > On 8 August 2018 at 17:28, Amit Langote wrote: >> Attached is a patch which modifies the if test to compare relids instead >> of RelOptInfo pointers. > Thanks for investigating and writing a patch. I agree with the fix. I changed this to compare the relid sets not just

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-08-08 Thread David Rowley
On 8 August 2018 at 17:28, Amit Langote wrote: > Attached is a patch which modifies the if test to compare relids instead > of RelOptInfo pointers. Thanks for investigating and writing a patch. I agree with the fix. It's probably worth writing a test that performs run-time pruning from an

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-08-07 Thread Amit Langote
On 2018/08/08 8:09, Tom Lane wrote: > Rushabh Lathia writes: >> Consider the below case: > > I initially thought the rule might be messing stuff up, but you can get > the same result without the rule by writing out the transformed query > by hand: > > regression=# explain UPDATE pt_p1 SET a = 3

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-08-07 Thread Tom Lane
Rushabh Lathia writes: > Consider the below case: I initially thought the rule might be messing stuff up, but you can get the same result without the rule by writing out the transformed query by hand: regression=# explain UPDATE pt_p1 SET a = 3 from pt WHERE pt.a = 2 and pt.a = pt_p1.a;

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-08-07 Thread Rushabh Lathia
Hi, Consider the below case: CREATE TABLE pt (a INT, b INT, c INT) PARTITION BY RANGE(a); CREATE TABLE pt_p1 PARTITION OF pt FOR VALUES FROM (1) to (6) PARTITION BY RANGE (b); CREATE TABLE pt_p1_p1 PARTITION OF pt_p1 FOR VALUES FROM (11) to (44); CREATE TABLE pt_p1_p2 PARTITION OF pt_p1 FOR

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-08-02 Thread Alvaro Herrera
On 2018-Aug-01, Tom Lane wrote: > David Rowley writes: > > On 20 July 2018 at 01:03, David Rowley wrote: > >> I've attached a patch intended for master which is just v2 based on > >> post 5220bb7533. > > I've pushed the v3 patch with a lot of editorial work (e.g. cleaning > up comments you

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-08-01 Thread David Rowley
On 2 August 2018 at 11:48, Tom Lane wrote: > I've pushed the v3 patch with a lot of editorial work (e.g. cleaning > up comments you hadn't). Thanks for doing that. > >> In [1] I mentioned that I think that bug should be fixed as part of >> this bug fix too. > > I didn't include this change

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-08-01 Thread Tom Lane
David Rowley writes: > On 20 July 2018 at 01:03, David Rowley wrote: >> I've attached a patch intended for master which is just v2 based on >> post 5220bb7533. I've pushed the v3 patch with a lot of editorial work (e.g. cleaning up comments you hadn't). I still want to think about getting rid

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-08-01 Thread Tom Lane
[ getting back to this thread at last ] Amit Langote writes: > On 2018/07/21 0:17, David Rowley wrote: >> You could work around that by having some array that points to the >> target partitioned table of each hierarchy, but I don't see why that's >> better than having the additional struct. >

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-08-01 Thread David Rowley
On 20 July 2018 at 01:03, David Rowley wrote: > I've attached a patch intended for master which is just v2 based on > post 5220bb7533. In [1] I mentioned that I think that bug should be fixed as part of this bug fix too. It just seems a little strange to fix that one separately when without the

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-07-25 Thread Amit Langote
On 2018/07/21 0:17, David Rowley wrote: > On 20 July 2018 at 21:44, Amit Langote wrote: >> But I don't think the result of make_partition_pruneinfo itself has to be >> List of PartitionedRelPruneInfo nested under PartitionPruneInfo. I gather >> that each PartitionPruneInfo corresponds to each

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-07-20 Thread David Rowley
On 20 July 2018 at 21:44, Amit Langote wrote: > But I don't think the result of make_partition_pruneinfo itself has to be > List of PartitionedRelPruneInfo nested under PartitionPruneInfo. I gather > that each PartitionPruneInfo corresponds to each root partitioned table > and a

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-07-20 Thread Amit Langote
On 2018/07/19 22:03, David Rowley wrote: > v3-0001-Fix-run-time-partition-pruning-for-UNION-ALL-pare.patch Thanks for updating the patch. I studied this patch today and concluded that it's going a bit too far by carrying the nested partition pruning info structures from the planner all the way

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-07-19 Thread David Rowley
On 18 July 2018 at 06:01, Alvaro Herrera wrote: > On 2018-Jul-16, David Rowley wrote: > >> On 16 July 2018 at 12:55, David Rowley wrote: >> > Thinking about this some more, I don't quite see any reason that the >> > partitioned_rels for a single hierarchy couldn't just be a Bitmapset >> >

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-07-19 Thread David Rowley
On 17 July 2018 at 12:21, David Rowley wrote: > On 16 July 2018 at 06:55, Tom Lane wrote: >> I started to look at this patch. I think this is basically the right >> direction to go in, but I'm not terribly happy with the details of the >> data structure design. > > I've made an attempt at

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-07-17 Thread Alvaro Herrera
On 2018-Jul-16, David Rowley wrote: > On 16 July 2018 at 12:55, David Rowley wrote: > > Thinking about this some more, I don't quite see any reason that the > > partitioned_rels for a single hierarchy couldn't just be a Bitmapset > > instead of an IntList. > > Of course, this is not possible

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-07-16 Thread David Rowley
On 16 July 2018 at 06:55, Tom Lane wrote: > I started to look at this patch. I think this is basically the right > direction to go in, but I'm not terribly happy with the details of the > data structure design. I've made an attempt at addressing the issues that I understood. I've not done

RE: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-07-16 Thread Phil Florent
I get it. Thank you for this precision. Regards Phil De : David Rowley Envoyé : lundi 16 juillet 2018 07:48 À : Phil Florent Cc : Tom Lane; Robert Haas; Amit Langote; PostgreSQL Hackers Objet : Re: Internal error XX000 with enable_partition_pruning=on, pg 11

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-07-15 Thread David Rowley
On 16 July 2018 at 16:56, Phil Florent wrote: > I should post that in the general section but I am confused by the > sentence "A parent partition is always going to have a lower relid than > its children" > It's a little confusing since RelOptInfo has a relid field and so does RangeTblEntry.

RE: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-07-15 Thread Phil Florent
Hi, I should post that in the general section but I am confused by the sentence "A parent partition is always going to have a lower relid than its children" In the code, relid has many meanings and not only "oid of a class" as in

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-07-15 Thread David Rowley
On 16 July 2018 at 12:55, David Rowley wrote: > Thinking about this some more, I don't quite see any reason that the > partitioned_rels for a single hierarchy couldn't just be a Bitmapset > instead of an IntList. Of course, this is not possible since we can't pass a List of Bitmapsets to the

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-07-15 Thread David Rowley
On 16 July 2018 at 06:55, Tom Lane wrote: > Also, in the Plan > representation, I'd suggest avoiding situations where a data structure > is sometimes a List of Lists of PartitionedRelPruneInfo and sometimes > just a single-level List. Saving one ListCell isn't worth the code > complexity and

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-07-15 Thread David Rowley
On 16 July 2018 at 06:55, Tom Lane wrote: > First off, given that we're now going to have a Plan data structure > that accurately reflects the partition hierarchy, I wonder whether we > couldn't get rid of the fiddling around with lists of ints and lists of > lists of ints; aren't those basically

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-07-15 Thread Tom Lane
David Rowley writes: > Anyway... Patch attached. This is method 3 of the 3 methods I thought > to fix this, so if this is not suitable then I'm out of ideas. I started to look at this patch. I think this is basically the right direction to go in, but I'm not terribly happy with the details of

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-07-12 Thread Alvaro Herrera
On 2018-Jul-12, Tom Lane wrote: > Andres Freund writes: > > On 2018-06-29 18:17:08 -0400, Tom Lane wrote: > >> I'm on vacation and won't have time to look at this until week after > >> next. If you don't mind putting the topic on hold that long, I'll > >> be happy to take responsibility for it.

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-07-12 Thread Tom Lane
Andres Freund writes: > On 2018-06-29 18:17:08 -0400, Tom Lane wrote: >> I'm on vacation and won't have time to look at this until week after >> next. If you don't mind putting the topic on hold that long, I'll >> be happy to take responsibility for it. > Is that still the plan? Do you forsee

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-07-12 Thread Andres Freund
Hi Tom, On 2018-06-29 18:17:08 -0400, Tom Lane wrote: > Alvaro Herrera writes: > > Since Tom has been revamping this code lately, I think it's a good > > idea to wait for his input. > > I'm on vacation and won't have time to look at this until week after > next. If you don't mind putting the

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-06-29 Thread Tom Lane
Alvaro Herrera writes: > Since Tom has been revamping this code lately, I think it's a good > idea to wait for his input. I'm on vacation and won't have time to look at this until week after next. If you don't mind putting the topic on hold that long, I'll be happy to take responsibility for

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-06-29 Thread Alvaro Herrera
On 2018-Jun-27, David Rowley wrote: > I've only just completed reading back through all the code and I think > it's correct. I ended up changing add_paths_to_append_rel() so that > instead of performing concatenation on partitioned_rels from two UNION > ALL children, it creates a List of lists.

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-06-19 Thread David Rowley
On 20 June 2018 at 02:28, Robert Haas wrote: > On Sun, Jun 17, 2018 at 10:59 PM, David Rowley >> Robert, do you have any objections to the proposed patch? > > I don't have time to study this right now, but I think the main > possible objection is around performance. If not flattening the >

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-06-19 Thread Robert Haas
On Sun, Jun 17, 2018 at 10:59 PM, David Rowley wrote: > Thanks for looking. > > Robert, do you have any objections to the proposed patch? I don't have time to study this right now, but I think the main possible objection is around performance. If not flattening the Append is the best way to

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-06-17 Thread David Rowley
On 18 June 2018 at 14:36, Amit Langote wrote: > On 2018/06/15 20:41, David Rowley wrote: >> If the top level Append is the UNION ALL one, then there won't be any >> partitioned_rels. If that's what you mean by no-op then, yeah. There >> are no duplicate locks already obtained in the parent with

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-06-17 Thread Amit Langote
On 2018/06/15 20:41, David Rowley wrote: > On 15 June 2018 at 20:37, Amit Langote wrote: >> select * from partitioned_table_a >> union all >> select * from partitioned_table_b >> >> The only thing that changes with the patch is that >> ExecLockNonLeafAppendTables is called *twice* for the two

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-06-15 Thread David Rowley
On 15 June 2018 at 20:37, Amit Langote wrote: > select * from partitioned_table_a > union all > select * from partitioned_table_b > > The only thing that changes with the patch is that > ExecLockNonLeafAppendTables is called *twice* for the two nested Appends > corresponding to

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-06-15 Thread Amit Langote
On 2018/06/11 16:49, David Rowley wrote: > On 11 June 2018 at 12:19, Tom Lane wrote: >> David Rowley writes: >>> On 10 June 2018 at 04:48, Tom Lane wrote: So, IIUC, the issue is that for partitioning cases Append expects *all* its children to be partitions of the *same* partitioned

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-06-11 Thread David Rowley
On 11 June 2018 at 12:19, Tom Lane wrote: > David Rowley writes: >> On 10 June 2018 at 04:48, Tom Lane wrote: >>> So, IIUC, the issue is that for partitioning cases Append expects *all* >>> its children to be partitions of the *same* partitioned table? That >>> is, you could also break it with

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-06-10 Thread Tom Lane
David Rowley writes: > On 10 June 2018 at 04:48, Tom Lane wrote: >> So, IIUC, the issue is that for partitioning cases Append expects *all* >> its children to be partitions of the *same* partitioned table? That >> is, you could also break it with >> >> select * from partitioned_table_a >>

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-06-09 Thread Tom Lane
David Rowley writes: > So it looks like I've assumed that the Append path's partitioned_rels > will only ever be set for partitioned tables, but it can, in fact, be > set for UNION ALL parents too when the union children are partitioned > tables. > As a discussion topic, I've attached a patch

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-06-08 Thread David Rowley
On 9 June 2018 at 06:50, David Rowley wrote: > It looks like this was 499be013de6, which was one of mine. > > A more simple case to reproduce is: > > drop table listp; > create table listp (a int, b int) partition by list(a); > create table listp1 partition of listp for values in (1); > select *

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-06-08 Thread David Rowley
On 9 June 2018 at 04:57, Tom Lane wrote: > Phil Florent writes: >> explain analyze select * from v where v.k1 > date '2017-01-01'; >> ERREUR: XX000: did not find all requested child rels in append_rel_list >> EMPLACEMENT : find_appinfos_by_relids, prepunion.c : 2643 > > Reproduced here, thanks

Re: Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-06-08 Thread Tom Lane
Phil Florent writes: > explain analyze select * from v where v.k1 > date '2017-01-01'; > ERREUR: XX000: did not find all requested child rels in append_rel_list > EMPLACEMENT : find_appinfos_by_relids, prepunion.c : 2643 Reproduced here, thanks for the report! This is very timely since we were

Internal error XX000 with enable_partition_pruning=on, pg 11 beta1 on Debian

2018-06-08 Thread Phil Florent
Hi, I obtained an XX000 error testing my DSS application with PostgreSQL 11 beta 1. Here is a simplified version of my test, no data in the tables : -- 11 select version(); version