Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-25 Thread Alvaro Herrera
Alvaro Herrera wrote: > I think this is the right fix for this problem. I wonder about > exploring other callers of RelationGetIndexList to see who else could be > confused ... CLUSTER was also affected (and ALTER TABLE .. CLUSTER ON). Patched both and added simple tests. Couldn't detect any ot

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-24 Thread Alvaro Herrera
I think this is the right fix for this problem. I wonder about exploring other callers of RelationGetIndexList to see who else could be confused ... -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services >From 53506fd3a

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-24 Thread Alvaro Herrera
David Rowley wrote: > 10. You've added code to get_relation_info() to handle partitioned > indexes, but that code is skipped due to: > > /* > * Make list of indexes. Ignore indexes on system catalogs if told to. > * Don't bother with indexes for an inheritance parent, either. > */ > if (inhparen

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-19 Thread Alvaro Herrera
Hi Jesper Jesper Pedersen wrote: > I get > > pg_restore: creating INDEX "pg_catalog.pg_authid_oid_index" > pg_restore: [archiver (db)] Error while PROCESSING TOC: > pg_restore: [archiver (db)] Error from TOC entry 5493; 1259 2677 INDEX > pg_authid_oid_index jpedersen > pg_restore: [archiver (db)

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-19 Thread Jesper Pedersen
Hi Alvaro, On 01/18/2018 04:43 PM, Alvaro Herrera wrote: Here's a patch to add pg_dump tests. This verifies that the CREATE INDEX on parent and partition appear, as well as ALTER INDEX .. ATTACH PARTITION. While doing this, I noticed a small bug: the ALTER INDEX would not be dumped when only o

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-18 Thread Alvaro Herrera
Here's a patch to add pg_dump tests. This verifies that the CREATE INDEX on parent and partition appear, as well as ALTER INDEX .. ATTACH PARTITION. While doing this, I noticed a small bug: the ALTER INDEX would not be dumped when only one of the schemas are specified to pg_dump (schema of parent

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-17 Thread Robert Haas
On Tue, Jan 16, 2018 at 6:08 PM, David Rowley wrote: > On 17 January 2018 at 03:58, Alvaro Herrera wrote: >>> 9. Error details claim that p2_a_idx is not a partition of p. >>> Shouldn't it say table "p2" is not a partition of "p"? >> >> You missed the "on" in the DETAIL: >> DETAIL: Index "p2_a

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-16 Thread David Rowley
On 17 January 2018 at 03:58, Alvaro Herrera wrote: >> 9. Error details claim that p2_a_idx is not a partition of p. >> Shouldn't it say table "p2" is not a partition of "p"? > > You missed the "on" in the DETAIL: > DETAIL: Index "p2_a_idx" is not on a partition of table "p". > You could argue t

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-16 Thread Alvaro Herrera
David Rowley wrote: > I've just made another pass over the patch and have a few more comments. Thanks! I'm a bit tied up in a few other things ATM, so an updated version will have to wait a couple of days. > 1. Expression varattnos don't seem to get translated correctly. Yesterday while creati

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-15 Thread David Rowley
On 12 January 2018 at 07:52, Alvaro Herrera wrote: > The delta patch turned out to have at least one stupid bug, and a few > non-stupid bugs. Here's an updated version that should behave > correctly, and addresses all reported problems. Thanks for updating the patch. I've just made another pass

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-12 Thread Peter Eisentraut
On 1/11/18 13:52, Alvaro Herrera wrote: > The delta patch turned out to have at least one stupid bug, and a few > non-stupid bugs. Here's an updated version that should behave > correctly, and addresses all reported problems. It seems that CompareIndexInfo() still doesn't compare indexes' operato

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-10 Thread David Rowley
On 11 January 2018 at 04:37, Alvaro Herrera wrote: > In prior incarnations of the patch, I had an if-test to prevent > attaching invalid indexes, but I decided to remove it at some point -- > mainly thinking of attaching a partition for which a CREATE INDEX > CONCURRENTLY was running which already

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-10 Thread Alvaro Herrera
David Rowley wrote: > Hi Álvaro, Hi David, Thanks for the review. Attached is a delta patch that fixes most things, except your item 14 below. Before getting into the details of the items you list, in this version I also fixed a couple of issues noted by Jaime Casanova; namely a) ALTER INDEX

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-08 Thread David Rowley
On 9 January 2018 at 09:54, Alvaro Herrera wrote: > I removed the pg_index.indparentidx column that previous versions add, > and replaced it with pg_inherits rows. This makes the code a little bit > bulkier in a couple of places, but nothing terrible. As a benefit, > there's no extra index in pg

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-08 Thread Jesper Pedersen
Hi Alvaro, On 01/08/2018 03:36 PM, Alvaro Herrera wrote: Jesper Pedersen wrote: Maybe a warning for existing indexes on the same column(s), but with a different type, should be emitted during ATTACH, e.g. [detach one partition, replace index with a different one, attach partition] Of cou

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-08 Thread Alvaro Herrera
Jesper Pedersen wrote: > Maybe a warning for existing indexes on the same column(s), but with a > different type, should be emitted during ATTACH, e.g. > [detach one partition, replace index with a different one, attach > partition] > Of course, this could also fall under index maintenance and n

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-08 Thread Jesper Pedersen
Hi Alvaro, On 01/04/2018 09:30 AM, Alvaro Herrera wrote: v11 fixes the dependency problem I mentioned in https://postgr.es/m/20171229203818.pqxf2cyl4g2wre6h@alvherre.pgsql and adds some test to cover that stuff. Thanks, make check-world passes. Maybe a warning for existing indexes on the sam

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-05 Thread Alvaro Herrera
Robert Haas wrote: > On Fri, Jan 5, 2018 at 4:57 PM, Peter Eisentraut > wrote: > > On 1/4/18 23:08, David Rowley wrote: > >> I admit to not having had a chance to look at any code with this yet, > >> but I'm just thinking about a case like the following. > >> > >> CREATE TABLE part (a INT, b INT)

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-05 Thread Robert Haas
On Fri, Jan 5, 2018 at 4:57 PM, Peter Eisentraut wrote: > On 1/4/18 23:08, David Rowley wrote: >> On 5 January 2018 at 11:01, Alvaro Herrera wrote: >>> (The more I think of this, the more I believe that pg_inherits is a >>> better answer. Opinions?) >> >> I admit to not having had a chance to lo

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-05 Thread Robert Haas
On Thu, Jan 4, 2018 at 5:01 PM, Alvaro Herrera wrote: > Tangentially: I didn't like very much that I added a new index to > pg_index to support this feature. I thought maybe it'd be better to > change the index on indrelid to be on (indrelid,indparentidx) instead, > but that doesn't seem great ei

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-05 Thread Peter Eisentraut
On 1/4/18 23:08, David Rowley wrote: > On 5 January 2018 at 11:01, Alvaro Herrera wrote: >> (The more I think of this, the more I believe that pg_inherits is a >> better answer. Opinions?) > > I admit to not having had a chance to look at any code with this yet, > but I'm just thinking about a c

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-04 Thread David Rowley
On 5 January 2018 at 11:01, Alvaro Herrera wrote: > (The more I think of this, the more I believe that pg_inherits is a > better answer. Opinions?) I admit to not having had a chance to look at any code with this yet, but I'm just thinking about a case like the following. CREATE TABLE part (a I

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-04 Thread Alvaro Herrera
Peter Eisentraut wrote: > On 1/4/18 12:00, Robert Haas wrote: > >> The catalog representations of partitioned tables and partitioned > >> indexes are completely different, which may or may not be desirable. > > > > How so? > > If someone wants to write a query, show me all the partitions of this

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-04 Thread Peter Eisentraut
On 1/4/18 12:00, Robert Haas wrote: >> The catalog representations of partitioned tables and partitioned >> indexes are completely different, which may or may not be desirable. > > How so? If someone wants to write a query, show me all the partitions of this table versus show me all the partition

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-04 Thread Alvaro Herrera
Peter Eisentraut wrote: > CompareIndexInfo() doesn't compare indexes' operator classes and collations. Hmm ... will look into these. > As mentioned elsewhere already, the tests fail because \di shows the > owner, which can vary between sites. Already fixed in v10 which I posted this morning. -

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-04 Thread Alvaro Herrera
Robert Haas wrote: > On Thu, Jan 4, 2018 at 11:44 AM, Peter Eisentraut > wrote: > > I'm not sure why this feature of automatically picking up matching > > indexes even exists. Is it for some specific workflows or upgrade > > scenarios? It's kind of a surprising feature in a way. > > It allows y

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-04 Thread Robert Haas
On Thu, Jan 4, 2018 at 11:44 AM, Peter Eisentraut wrote: > I'm not sure why this feature of automatically picking up matching > indexes even exists. Is it for some specific workflows or upgrade > scenarios? It's kind of a surprising feature in a way. It allows you to avoid building a new indexe

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-04 Thread Peter Eisentraut
On 12/29/17 12:59, Alvaro Herrera wrote: >> Maybe we need a new "auto internal" deptype with a mix of semantics of >> the other two deptypes. It seems a bit ugly and I'm not sure it'd work >> either ... I'll try to code it tomorrow. > > This seems to work pretty well, much to my surprise. I was

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-03 Thread Alvaro Herrera
Peter Eisentraut wrote: > On 12/29/17 15:38, Alvaro Herrera wrote: > > I just realized there's a further problem in the area: when a partition > > is detached from its parent, its indexes are not made independent of the > > indexes on parent. So they can't be dropped on their own (booh!); and > >

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-03 Thread Peter Eisentraut
On 12/29/17 15:38, Alvaro Herrera wrote: > I just realized there's a further problem in the area: when a partition > is detached from its parent, its indexes are not made independent of the > indexes on parent. So they can't be dropped on their own (booh!); and > dropping the index on the former p

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-02 Thread Alvaro Herrera
Jesper Pedersen wrote: > On 12/22/2017 10:10 AM, Alvaro Herrera wrote: > > If you have wording suggestions for the doc changes, please send them > > along. > > Maybe we should make the default index name more explicit under the 'name' > parameter as attached. I'm -0.2 on documenting this. In g

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-02 Thread Jesper Pedersen
Hi Alvaro, On 12/29/2017 12:59 PM, Alvaro Herrera wrote: This seems to work pretty well, much to my surprise. I was a bit scared of adding a new deptype, but actually the only affected code is findDependentObjects() and the semantics of the new type is a subset of the existing DEPTYPE_INTERNAL,

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-02 Thread Jesper Pedersen
Hi Alvaro, On 12/22/2017 10:10 AM, Alvaro Herrera wrote: I believe these are all fixed by the attached delta patch. Thanks. If you have wording suggestions for the doc changes, please send them along. Maybe we should make the default index name more explicit under the 'name' parameter a

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-29 Thread Alvaro Herrera
I just realized there's a further problem in the area: when a partition is detached from its parent, its indexes are not made independent of the indexes on parent. So they can't be dropped on their own (booh!); and dropping the index on the former parent partitioned table drops the index on the fo

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-28 Thread Alvaro Herrera
Amit Langote wrote: > 2. Something strange about how dependencies are managed: > > create table p (a char) partition by list (a); > create table pa partition of p for values in ('a');; > create table pb partition of p for values in ('b'); > create index on p (a); > drop table pa; > ERROR: canno

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-25 Thread Amit Langote
On 2017/12/25 13:52, Amit Langote wrote: > Hi Alvaro, > > On 2017/12/23 0:10, Alvaro Herrera wrote: >> I believe these are all fixed by the attached delta patch. > > @@ -1676,7 +1694,12 @@ psql_completion(const char *text, int start, int end) > "UNION SELECT 'A

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-24 Thread Amit Langote
Hi Alvaro, On 2017/12/23 0:10, Alvaro Herrera wrote: > I believe these are all fixed by the attached delta patch. @@ -1676,7 +1694,12 @@ psql_completion(const char *text, int start, int end) "UNION SELECT 'ALL IN TABLESPACE'"); /* ALTER INDEX */ else

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-22 Thread Alvaro Herrera
Hello Jesper, Jesper Pedersen wrote: > Passes check-world here too w/ TAP + cassert. Great, thanks for checking. > index.c: > [and a few other comments] I believe these are all fixed by the attached delta patch. If you have wording suggestions for the doc changes, please send them along. Th

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-21 Thread Jesper Pedersen
Hi Alvaro, On 12/20/2017 04:25 PM, Alvaro Herrera wrote: I modified the regression test so that a partitioning hierarchy would be left behind after the test is run, which is useful to test pg_upgrade and pg_dump -- this caught one small bug. That and some reading of the diff resulted in v8, att

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-20 Thread Alvaro Herrera
Just to show what I'm talking about, here are a few prototype patches. Beware, these are all very lightly tested/reviewed. Input is welcome, particularly if it comes in the guise of patches to regression tests showing cases that misbehave. 0001 is a fixup for the v6 patch I posted upthread; it's

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-20 Thread Alvaro Herrera
Robert Haas wrote: > Sounds great! I made the comment during my talk at PGCONF.EU that > partitioned tables in PostgreSQL are really just a bunch of tables > glued together, but that over time "we'll make the glue better", and I > think the improvements on which you are working will go a long way

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-20 Thread Robert Haas
On Wed, Dec 20, 2017 at 12:01 PM, Alvaro Herrera wrote: > I have two patches to rebase on top of this, which I hope to post later > today: > > 1) let these indexes be unique (i.e. allow unique and PK constraints) > 2) allow foreign keys on partitioned tables > > I have a further patch to allow FOR

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-20 Thread Alvaro Herrera
Great, thanks for the input. pg_dump behaves as described upthread -- thanks David and Robert for the input. I did this by injecting a fake "INDEX ATTACH" object in pg_dump's model, which depends on the index-on-parent-table; which in turn depends on the index-on-partition. Because of the depend

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-18 Thread David Rowley
On 19 December 2017 at 05:08, Robert Haas wrote: > On Mon, Dec 18, 2017 at 11:02 AM, Alvaro Herrera > wrote: >> After this discussion, this is how I see things working: >> >> 1. pg_dump >>a) creates indexes on partitions normally >>b) once all existing indexes are done, index on parent is

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-18 Thread Robert Haas
On Mon, Dec 18, 2017 at 11:02 AM, Alvaro Herrera wrote: > After this discussion, this is how I see things working: > > 1. pg_dump >a) creates indexes on partitions normally >b) once all existing indexes are done, index on parent is created, > with ONLY. No cascading occurs, no index

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-18 Thread Alvaro Herrera
After this discussion, this is how I see things working: 1. pg_dump a) creates indexes on partitions normally b) once all existing indexes are done, index on parent is created, with ONLY. No cascading occurs, no indexes are attached. c) ATTACH is run for each existing partition ind

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-17 Thread Robert Haas
On Sun, Dec 17, 2017 at 10:05 PM, David Rowley wrote: > I think you feel quite strongly about not having the code select a > random matching index, so if we want to stick to that rule, then we'll > need to create a set of new leaf indexes rather than select a random > one. I feel quite strongly a

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-17 Thread David Rowley
On 18 December 2017 at 15:59, Robert Haas wrote: > On Sun, Dec 17, 2017 at 9:38 PM, David Rowley > wrote: >> On 18 December 2017 at 15:04, Robert Haas wrote: >>> On Sun, Dec 17, 2017 at 5:29 AM, David Rowley >>> wrote: I'm now not that clear on what the behaviour is if the ONLY keyword is

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-17 Thread Robert Haas
On Sun, Dec 17, 2017 at 9:38 PM, David Rowley wrote: > On 18 December 2017 at 15:04, Robert Haas wrote: >> On Sun, Dec 17, 2017 at 5:29 AM, David Rowley >> wrote: >>> I'm now not that clear on what the behaviour is if the ONLY keyword is >>> not specified on the CREATE INDEX for the partitioned

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-17 Thread David Rowley
On 18 December 2017 at 15:04, Robert Haas wrote: > On Sun, Dec 17, 2017 at 5:29 AM, David Rowley > wrote: >> I'm now not that clear on what the behaviour is if the ONLY keyword is >> not specified on the CREATE INDEX for the partitioned index. Does that >> go and create each leaf partition index

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-17 Thread Robert Haas
On Sun, Dec 17, 2017 at 5:29 AM, David Rowley wrote: >> If ALTER INDEX .. ATTACH is already taking AEL on the parent, then I >> think it might as well try to validate while it's at it. But if not >> then we might want to go with #2. > > I'm now not that clear on what the behaviour is if the ONLY

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-17 Thread Robert Haas
On Sun, Dec 17, 2017 at 10:50 AM, Alvaro Herrera wrote: >> If ALTER INDEX .. ATTACH is already taking AEL on the parent, then I >> think it might as well try to validate while it's at it. But if not >> then we might want to go with #2. > > The problem I have with it is that restoring a dump conta

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-17 Thread Alvaro Herrera
Robert Haas wrote: > On Fri, Dec 15, 2017 at 5:18 PM, Alvaro Herrera > wrote: > > We have two options for marking valid: > > > > 1. after each ALTER INDEX ATTACH, verify whether the set of partitions > > that contain the index is complete; if so, mark it valid, otherwise do > > nothing. This suc

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-17 Thread David Rowley
On 17 December 2017 at 16:22, Robert Haas wrote: > On Fri, Dec 15, 2017 at 5:18 PM, Alvaro Herrera > wrote: >> We have two options for marking valid: >> >> 1. after each ALTER INDEX ATTACH, verify whether the set of partitions >> that contain the index is complete; if so, mark it valid, otherwis

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-16 Thread Robert Haas
On Fri, Dec 15, 2017 at 5:18 PM, Alvaro Herrera wrote: > We have two options for marking valid: > > 1. after each ALTER INDEX ATTACH, verify whether the set of partitions > that contain the index is complete; if so, mark it valid, otherwise do > nothing. This sucks because we have to check that o

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-15 Thread Alvaro Herrera
Robert Haas wrote: > On Fri, Dec 15, 2017 at 4:02 PM, Alvaro Herrera > wrote: > > 3. Robert's: use CREATE INDEX ON ONLY , which creates a shell > >index on parent only (no recursion), followed by CREATE INDEX ON > >. DETACH is not provided. If you ATTACH an index for a > >partition

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-15 Thread Alvaro Herrera
Robert Haas wrote: > On Thu, Nov 30, 2017 at 7:02 AM, Alvaro Herrera > wrote: > > Great question. So you're thinking that the planner might have an > > interest in knowing what indexes are defined at the parent table level > > for planning purposes; but for that to actually have any effect we wo

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-15 Thread Robert Haas
On Fri, Dec 15, 2017 at 4:02 PM, Alvaro Herrera wrote: > 3. Robert's: use CREATE INDEX ON ONLY , which creates a shell >index on parent only (no recursion), followed by CREATE INDEX ON >. DETACH is not provided. If you ATTACH an index for a >partition that already has one index attac

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-15 Thread Alvaro Herrera
Alvaro Herrera wrote: > 3. Robert's: use CREATE INDEX ON ONLY , which creates a shell >index on parent only (no recursion), followed by CREATE INDEX ON >. DETACH is not provided. If you ATTACH an index for a >partition that already has one index attached, then (1) the newly >atta

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-15 Thread Alvaro Herrera
Hmm, so I'm now unsure what the actual proposals for handling pg_dump are. We seem to have the following three proposals: 1. Alvaro: use CREATE INDEX ON ONLY (not recursive ), followed by CREATE INDEX ON , followed by ALTER INDEX ATTACH PARTITION . I provide an ALTER INDEX DETACH PART

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-12 Thread Robert Haas
On Mon, Dec 11, 2017 at 10:04 PM, Alvaro Herrera wrote: > David Rowley wrote: > >> ATTACH/REPLACE sounds fine. My objection was more about the >> DETACH/ATTACH method to replace an index. > > So what happens if you do ALTER INDEX .. ATTACH and you already have > another index in that partition tha

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-12 Thread Alvaro Herrera
David Rowley wrote: > ATTACH/REPLACE sounds fine. My objection was more about the > DETACH/ATTACH method to replace an index. So what happens if you do ALTER INDEX .. ATTACH and you already have another index in that partition that is attached to the same parent in the index? With my code, what

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-07 Thread David Rowley
On 8 December 2017 at 11:23, Robert Haas wrote: > On Thu, Dec 7, 2017 at 4:57 PM, David Rowley > wrote: >> That's true, but is it worth inventing/maintaining an ATTACH syntax >> for that? It's not a very common case that multiple matching indexes >> existing. If it is worth it, do we need DETACH

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-07 Thread Robert Haas
On Thu, Dec 7, 2017 at 4:57 PM, David Rowley wrote: >> Sure, that would fix the problem I'm concerned about, but creating the >> parent index first, as a shell, and then creating each child and >> attaching it to the parent *also* fixes the problem I'm concerned >> about, and without dragging any

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-07 Thread David Rowley
On 8 December 2017 at 10:51, Robert Haas wrote: > On Thu, Dec 7, 2017 at 4:43 PM, David Rowley > wrote: >> And yeah, this does nothing for making sure we choose the correct >> index if more than one matching index exists on the leaf partition, >> but perhaps we can dump a series of >> >> ALTER IN

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-07 Thread Robert Haas
On Thu, Dec 7, 2017 at 4:43 PM, David Rowley wrote: > And yeah, this does nothing for making sure we choose the correct > index if more than one matching index exists on the leaf partition, > but perhaps we can dump a series of > > ALTER INDEX p_a_idx REPLACE INDEX FOR p1 WITH p1_a_idx; > ALTER IN

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-07 Thread David Rowley
On 8 December 2017 at 10:07, Robert Haas wrote: > I do agree with you that an index which is currently enforcing a > unique constraint has to remain continuously valid -- or if it > unavoidably doesn't, for example if we allowed an unlogged unique > index on a logged table, then we'd have to do so

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-07 Thread Robert Haas
On Tue, Dec 5, 2017 at 7:42 PM, David Rowley wrote: > On 6 December 2017 at 11:35, Robert Haas wrote: >> What are we giving up by explicitly attaching >> the correct index? > > The part I don't like about the ATTACH and DETACH of partitioned index > is that it seems to be trying to just follow th

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-05 Thread David Rowley
On 6 December 2017 at 13:42, David Rowley wrote: > On 6 December 2017 at 11:35, Robert Haas wrote: >> What are we giving up by explicitly attaching >> the correct index? > > The part I don't like about the ATTACH and DETACH of partitioned index > is that it seems to be trying to just follow the s

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-05 Thread Ashutosh Bapat
On Wed, Dec 6, 2017 at 9:42 AM, David Rowley wrote: > On 6 December 2017 at 11:35, Robert Haas wrote: >> What are we giving up by explicitly attaching >> the correct index? > > The part I don't like about the ATTACH and DETACH of partitioned index > is that it seems to be trying to just follow th

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-05 Thread David Rowley
On 6 December 2017 at 11:35, Robert Haas wrote: > What are we giving up by explicitly attaching > the correct index? The part I don't like about the ATTACH and DETACH of partitioned index is that it seems to be trying to just follow the syntax we use to remove a partition from a partitioned table

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-05 Thread David Rowley
On 6 December 2017 at 09:58, Alvaro Herrera wrote: > David Rowley wrote: >> On 6 December 2017 at 04:29, Robert Haas wrote: >> > How does that proposal keep pg_dump from latching onto the wrong >> > index, if there's more than one index on the same columns? >> >> I'm not hugely concerned about th

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-05 Thread Robert Haas
On Tue, Dec 5, 2017 at 3:53 PM, David Rowley wrote: > I'm not hugely concerned about that. It's not a new problem and it's > not a problem that I recall seeing anyone complain about, at least not > to the extent that we've ever bothered to fix it. > > The existing problem is with FOREIGN KEY const

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-05 Thread Alvaro Herrera
David Rowley wrote: > On 6 December 2017 at 04:29, Robert Haas wrote: > > On Mon, Dec 4, 2017 at 6:57 PM, David Rowley > >> I proposed that this worked a different way in [1]. I think the way I > >> mention is cleaner as it means there's no extra reason for a > >> partitioned index to be indisvali

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-05 Thread David Rowley
On 6 December 2017 at 04:29, Robert Haas wrote: > On Mon, Dec 4, 2017 at 6:57 PM, David Rowley >> I proposed that this worked a different way in [1]. I think the way I >> mention is cleaner as it means there's no extra reason for a >> partitioned index to be indisvalid=false than there is for any

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-05 Thread Robert Haas
On Mon, Dec 4, 2017 at 6:57 PM, David Rowley wrote: > On 2 December 2017 at 03:39, Robert Haas wrote: >> On Thu, Nov 30, 2017 at 11:39 PM, David Rowley >> wrote: >>> I feel like we could do better here with little extra effort. The >>> DETACH index feature does not really seem required for this

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-04 Thread David Rowley
On 2 December 2017 at 03:39, Robert Haas wrote: > On Thu, Nov 30, 2017 at 11:39 PM, David Rowley > wrote: >> I feel like we could do better here with little extra effort. The >> DETACH index feature does not really seem required for this patch. > > Because of the dump/restore considerations menti

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-04 Thread David Rowley
On 2 December 2017 at 11:10, Alvaro Herrera wrote: > What you're saying is that I've written code for A+B, and you're > "interested in C (which is incompatible with B), so can we have A+C and > drop B". But in reality, there exists (unwritten) D that solves the > incompatiblity between B and C.

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-02 Thread Alvaro Herrera
David Rowley wrote: > So, then this patch is only really intended as a syntax shortcut for > mass adding of indexes to each partition? This patch is intended to serve as a basis on which to construct further features, just like every other patch we apply. > I feel like we could do better here wi

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-01 Thread Robert Haas
On Thu, Nov 30, 2017 at 11:39 PM, David Rowley wrote: > I feel like we could do better here with little extra effort. The > DETACH index feature does not really seem required for this patch. Because of the dump/restore considerations mentioned in http://postgr.es/m/ca+tgmobuhghg9v8saswkhbbfywg5a0

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-30 Thread David Rowley
On 1 December 2017 at 01:02, Alvaro Herrera wrote: > we currently (I mean after my > patch) don't mark partitioned-table-level indexes as valid or not valid > depending on whether all its children exist, so trying to use that in > the planner without having a flag could cause invalid plans to be >

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-30 Thread Robert Haas
On Thu, Nov 30, 2017 at 7:02 AM, Alvaro Herrera wrote: > Great question. So you're thinking that the planner might have an > interest in knowing what indexes are defined at the parent table level > for planning purposes; but for that to actually have any effect we would > need to change the plann

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-30 Thread Alvaro Herrera
Michael Paquier wrote: > On Wed, Nov 15, 2017 at 2:49 AM, Alvaro Herrera > wrote: > > Here's the remaining bits, rebased. > > At least patch 3 has conflicts with HEAD. I am moving this patch to > next CF per a lack of reviews, switching status to waiting on author. Thanks, will submit a new ver

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-30 Thread Alvaro Herrera
David Rowley wrote: > I wonder if it should be this patches job to alter the code in > get_relation_info() which causes the indexes not to be loaded for > partitioned tables: > > /* > * Make list of indexes. Ignore indexes on system catalogs if told to. > * Don't bother with indexes for an inhe

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-29 Thread Michael Paquier
On Wed, Nov 15, 2017 at 2:49 AM, Alvaro Herrera wrote: > Here's the remaining bits, rebased. At least patch 3 has conflicts with HEAD. I am moving this patch to next CF per a lack of reviews, switching status to waiting on author. -- Michael

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-20 Thread Ashutosh Bapat
On Fri, Nov 17, 2017 at 6:24 PM, David Rowley wrote: > > I'm kind of thinking this patch should change that, even if the patch is not > making use of the indexes, you could argue that something using > set_rel_pathlist_hook might want to do something there, although, there's > likely a bunch of co

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-17 Thread David Rowley
On 15 November 2017 at 06:49, Alvaro Herrera wrote: > Here's the remaining bits, rebased. I wonder if it should be this patches job to alter the code in get_relation_info() which causes the indexes not to be loaded for partitioned tables: /* * Make list of indexes. Ignore indexes on system c

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-17 Thread David Rowley
On 15 November 2017 at 06:49, Alvaro Herrera wrote: > Here's the remaining bits, rebased. Hi, I've not had time for a thorough look at this, but on a quick scan I noticed that CompareIndexInfo() missed checking if the Index AM matches the AM of the partitioned index. Testing with: create ta

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-14 Thread David Rowley
On 15 November 2017 at 04:23, Alvaro Herrera wrote: > David Rowley wrote: >> I'd have thought some sort of: ALTER INDEX name_of_partitioned_index >> REPLACE INDEX FOR partitioned_tablename WITH >> name_of_new_matching_bloat_free_index; >> >> ... or something along those lines, and just have an ato

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-14 Thread Alvaro Herrera
Hi Jesper, Thanks for reviewing. Jesper Pedersen wrote: > I have been looking at the "CREATE INDEX ... ONLY" syntax, and I think it > could cause some confusion due to the "Note" described in create_index.sgml. > > An alternative, maybe crazy, could be to treat a partitioned index as one; > e.g

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-14 Thread Robert Haas
On Tue, Nov 14, 2017 at 1:43 PM, Simon Riggs wrote: > I don't see any comments from you or Tom about patch 0001, which was > simple refactoring and not much to complain about. We both commented that getting rid of copy_partition_data could introduce memory leaks. > Perhaps there is some confusio

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-14 Thread Jesper Pedersen
Hi, On 11/14/2017 12:49 PM, Alvaro Herrera wrote: Thanks, pushed. Here's the remaining bits, rebased. First of all, thanks for working on this. I have been looking at the "CREATE INDEX ... ONLY" syntax, and I think it could cause some confusion due to the "Note" described in create_index

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-14 Thread Simon Riggs
On 14 November 2017 at 13:12, Robert Haas wrote: > On Tue, Nov 14, 2017 at 12:49 PM, Alvaro Herrera > wrote: >> Here's the remaining bits, rebased. > > It's true that Tom and I reviewed patch 0001, as your proposed commit > message states. But it's also true that we both said that it probably >

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-14 Thread Robert Haas
On Tue, Nov 14, 2017 at 12:49 PM, Alvaro Herrera wrote: > Here's the remaining bits, rebased. It's true that Tom and I reviewed patch 0001, as your proposed commit message states. But it's also true that we both said that it probably wasn't a good idea. -- Robert Haas EnterpriseDB: http://www.

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-14 Thread Alvaro Herrera
Alvaro Herrera wrote: > Simon Riggs wrote: > > On 13 November 2017 at 12:55, Alvaro Herrera > > wrote: > > > Somehow I managed to include an unrelated patch as attachment. Here's > > > another attempt (on which I also lightly touched ddl.sgml with relevant > > > changes). > > > > Looks good. So

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-14 Thread Alvaro Herrera
David Rowley wrote: > But if you can DETACH a partition from requiring an index to back up > this partitioned index, then the partitioned index is not valid while > the leaf table's index is missing. Yeah. I don't see this as too much of a problem myself. At present, from the planner's POV each

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-14 Thread Alvaro Herrera
Simon Riggs wrote: > On 13 November 2017 at 12:55, Alvaro Herrera wrote: > > Somehow I managed to include an unrelated patch as attachment. Here's > > another attempt (on which I also lightly touched ddl.sgml with relevant > > changes). > > Looks good. Some minor comments below. > > 0001- Simpl

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-14 Thread David Rowley
On 15 November 2017 at 01:30, Alvaro Herrera wrote: > David Rowley wrote: >> hmm, but surely the all those indexes must already exist if the >> partitioned index exists. Won't we be disallowing DROP INDEX of the >> leaf partition indexes if that index is marked as being part of the >> partitioned

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-14 Thread Alvaro Herrera
David Rowley wrote: > On 15 November 2017 at 01:09, Alvaro Herrera wrote: > > if a > > partition exists which *doesn't* have the index, restoring things this > > way would create the index in that partition too, which is unwanted > > because the end state is different to what was in the dumped dat

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-11-14 Thread David Rowley
On 15 November 2017 at 01:09, Alvaro Herrera wrote: > if a > partition exists which *doesn't* have the index, restoring things this > way would create the index in that partition too, which is unwanted > because the end state is different to what was in the dumped database. hmm, but surely the al

  1   2   >