Re: pg_dump is broken for partition tablespaces

2019-04-25 Thread Alvaro Herrera
I have pushed this now, after putting back a few of the tests I had proposed earlier, as well as a couple of sentences in the docs to hopefully make it clearer how it works. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training &

Re: pg_dump is broken for partition tablespaces

2019-04-24 Thread Alvaro Herrera
On 2019-Apr-23, Alvaro Herrera wrote: > I'm not sure yet that 100% of the patch is gone, but yes much of it > would go away thankfully. Of course, the part that fixes the bug that indexes move tablespace when recreated by a rewriting ALTER TABLE is still necessary. Included in the attached

Re: pg_dump is broken for partition tablespaces

2019-04-23 Thread Amit Langote
On 2019/04/23 20:03, David Rowley wrote: > On Tue, 23 Apr 2019 at 18:18, Amit Langote > wrote: >> >> If partitions needed a >> map in the old database, this patch means that they will *continue* to >> need it in the new database. > > That's incorrect. Not completely though, because... > My

Re: pg_dump is broken for partition tablespaces

2019-04-23 Thread Alvaro Herrera
On 2019-Apr-24, David Rowley wrote: > On Wed, 24 Apr 2019 at 10:26, Alvaro Herrera wrote: > > If creating a partition, there is the additional rule that parent's > > tablespace overrides default_tablespace: > > > > a2. if there's a TABLESPACE clause, use that. > > b2. otherwise, if the parent

Re: pg_dump is broken for partition tablespaces

2019-04-23 Thread David Rowley
On Wed, 24 Apr 2019 at 10:26, Alvaro Herrera wrote: > If creating a partition, there is the additional rule that parent's > tablespace overrides default_tablespace: > > a2. if there's a TABLESPACE clause, use that. > b2. otherwise, if the parent has a tablespace, use that. > c2. otherwise, if

Re: pg_dump is broken for partition tablespaces

2019-04-23 Thread Alvaro Herrera
Thanks for taking the time to think through this. On 2019-Apr-22, Robert Haas wrote: > If we know what the feature is supposed to do, then it should be > possible to look at each relevant piece of code and decides whether it > implements the specification correctly or not. But if we don't have

Re: pg_dump is broken for partition tablespaces

2019-04-23 Thread David Rowley
On Tue, 23 Apr 2019 at 18:18, Amit Langote wrote: > > If partitions needed a > map in the old database, this patch means that they will *continue* to > need it in the new database. That's incorrect. My point was about dropped columns being removed after a dump / reload. Only binary upgrade

Re: pg_dump is broken for partition tablespaces

2019-04-23 Thread Amit Langote
On 2019/04/23 14:45, David Rowley wrote: > On Tue, 23 Apr 2019 at 13:49, Amit Langote > wrote: >> >> On 2019/04/23 7:51, Alvaro Herrera wrote: >>> To me, it sounds >>> unintuitive to accept partitions that don't exactly match the order of >>> the parent table; but it's been supported all along.

Re: pg_dump is broken for partition tablespaces

2019-04-22 Thread David Rowley
On Tue, 23 Apr 2019 at 13:49, Amit Langote wrote: > > On 2019/04/23 7:51, Alvaro Herrera wrote: > > To me, it sounds > > unintuitive to accept partitions that don't exactly match the order of > > the parent table; but it's been supported all along. > > You might know it already, but even though

Re: pg_dump is broken for partition tablespaces

2019-04-22 Thread Amit Langote
On 2019/04/23 7:51, Alvaro Herrera wrote: > On 2019-Mar-06, Tom Lane wrote: >> David Rowley writes: >>> As far as I can see, the biggest fundamental difference with doing >>> things this way will be that the column order of partitions will be >>> preserved, where before it would inherit the order

Re: pg_dump is broken for partition tablespaces

2019-04-22 Thread Tom Lane
Alvaro Herrera writes: > Now that I re-read this complaint once again, I wonder if a mismatching > column order in partitions isn't a thing we ought to preserve anyhow. > Robert, Amit -- is it by design that pg_dump loses the original column > order for partitions, when not in binary-upgrade

Re: pg_dump is broken for partition tablespaces

2019-04-22 Thread Alvaro Herrera
On 2019-Mar-06, Tom Lane wrote: > David Rowley writes: > > As far as I can see, the biggest fundamental difference with doing > > things this way will be that the column order of partitions will be > > preserved, where before it would inherit the order of the partitioned > > table. I'm a little

Re: pg_dump is broken for partition tablespaces

2019-04-22 Thread Robert Haas
On Mon, Apr 22, 2019 at 4:43 PM Alvaro Herrera wrote: > Well, frequently when people discuss ideas on this list, others discuss > and provide further ideas to try help to find a working solution, rather > than throw every roadblock they can think of (though roadblocks are > indeed thrown now and

Re: pg_dump is broken for partition tablespaces

2019-04-22 Thread Alvaro Herrera
On 2019-Apr-22, Robert Haas wrote: > On Mon, Apr 22, 2019 at 3:08 PM Alvaro Herrera > wrote: > > On 2019-Apr-22, Andres Freund wrote: > > > Why is the obvious answer is to not just remove the whole tablespace > > > inheritance behaviour? > > > > Because it was requested by many, and there were

Re: pg_dump is broken for partition tablespaces

2019-04-22 Thread Robert Haas
On Mon, Apr 22, 2019 at 3:08 PM Alvaro Herrera wrote: > On 2019-Apr-22, Andres Freund wrote: > > Why is the obvious answer is to not just remove the whole tablespace > > inheritance behaviour? > > Because it was requested by many, and there were plenty of people > surprised that things didn't

Re: pg_dump is broken for partition tablespaces

2019-04-22 Thread Tom Lane
Alvaro Herrera writes: > On 2019-Apr-22, Andres Freund wrote: >> Why is the obvious answer is to not just remove the whole tablespace >> inheritance behaviour? > Because it was requested by many, and there were plenty of people > surprised that things didn't work that way. There are lots of

Re: pg_dump is broken for partition tablespaces

2019-04-22 Thread Alvaro Herrera
On 2019-Apr-22, Tom Lane wrote: > Yeah, that's where I'm at as well. Alvaro's proposal could be made > to work perhaps, but I think it would still end up with some odd > corner-case behaviors. One example is that "TABLESPACE X" would > be allowed if the database's default tablespace is Y, but

Re: pg_dump is broken for partition tablespaces

2019-04-22 Thread Alvaro Herrera
On 2019-Apr-22, Andres Freund wrote: > Why is the obvious answer is to not just remove the whole tablespace > inheritance behaviour? Because it was requested by many, and there were plenty of people surprised that things didn't work that way. -- Álvaro Herrera

Re: pg_dump is broken for partition tablespaces

2019-04-22 Thread Tom Lane
Andres Freund writes: > On 2019-04-22 14:16:28 -0400, Alvaro Herrera wrote: >> I think we can get out of this whole class of problems by forbidding the >> TABLESPACE clause for partitioned rels from mentioning the database >> tablespace -- that is, users either mention some *other* tablespace, or

Re: pg_dump is broken for partition tablespaces

2019-04-22 Thread Andres Freund
Hi, On 2019-04-22 14:16:28 -0400, Alvaro Herrera wrote: > On 2019-Apr-22, Robert Haas wrote: > > > PostgreSQL has historically and very deliberately *not made a > > distinction* between "this object is in the default tablespace" and > > "this object is in tablespace X which happens to be the

Re: pg_dump is broken for partition tablespaces

2019-04-22 Thread Alvaro Herrera
On 2019-Apr-22, Robert Haas wrote: > PostgreSQL has historically and very deliberately *not made a > distinction* between "this object is in the default tablespace" and > "this object is in tablespace X which happens to be the default." I > think that it's too late to invent such a distinction

Re: pg_dump is broken for partition tablespaces

2019-04-22 Thread Robert Haas
On Wed, Apr 17, 2019 at 6:06 PM Alvaro Herrera wrote: > > > 3. Partitioned relations can have the database tablespace in > > >pg_class.reltablespace, as opposed to storage-bearing relations which > > >cannot. This is useful to be able to put partitions in the database > > >tablespace

Re: pg_dump is broken for partition tablespaces

2019-04-18 Thread Alvaro Herrera
On 2019-Apr-17, Alvaro Herrera wrote: > On 2019-Apr-17, Tom Lane wrote: > > > Alvaro Herrera writes: > > > 1. pg_dump now uses regular CREATE TABLE followed by ALTER TABLE / ATTACH > > >PARTITION when creating partitions, rather than CREATE TABLE > > >PARTITION OF. pg_dump

Re: pg_dump is broken for partition tablespaces

2019-04-17 Thread Alvaro Herrera
On 2019-Apr-17, Tom Lane wrote: > Alvaro Herrera writes: > > 1. pg_dump now uses regular CREATE TABLE followed by ALTER TABLE / ATTACH > >PARTITION when creating partitions, rather than CREATE TABLE > >PARTITION OF. pg_dump --binary-upgrade was already doing that, so > >this part

Re: pg_dump is broken for partition tablespaces

2019-04-17 Thread Tom Lane
Alvaro Herrera writes: > 1. pg_dump now uses regular CREATE TABLE followed by ALTER TABLE / ATTACH >PARTITION when creating partitions, rather than CREATE TABLE >PARTITION OF. pg_dump --binary-upgrade was already doing that, so >this part mostly removes some code. In order to make

Re: pg_dump is broken for partition tablespaces

2019-04-17 Thread Alvaro Herrera
On 2019-Apr-17, David Rowley wrote: > On Mon, 15 Apr 2019 at 15:26, Alvaro Herrera wrote: > > > > On 2019-Apr-15, David Rowley wrote: > > > > > To be honest, if I'd done a better job of thinking through the > > > implications of this tablespace inheritance in ca4103025d, then I'd > > > probably

Re: pg_dump is broken for partition tablespaces

2019-04-16 Thread David Rowley
On Mon, 15 Apr 2019 at 15:26, Alvaro Herrera wrote: > > On 2019-Apr-15, David Rowley wrote: > > > To be honest, if I'd done a better job of thinking through the > > implications of this tablespace inheritance in ca4103025d, then I'd > > probably have not bothered submitting a patch for it. We

Re: pg_dump is broken for partition tablespaces

2019-04-14 Thread Alvaro Herrera
On 2019-Apr-15, David Rowley wrote: > To be honest, if I'd done a better job of thinking through the > implications of this tablespace inheritance in ca4103025d, then I'd > probably have not bothered submitting a patch for it. We could easily > revert that, but we'd still be left with the same

Re: pg_dump is broken for partition tablespaces

2019-04-14 Thread David Rowley
On Mon, 15 Apr 2019 at 05:32, Alvaro Herrera wrote: > > On 2019-Apr-14, Andres Freund wrote: > > > On 2019-04-14 10:38:05 -0400, Tom Lane wrote: > > > It's entirely possible BTW that this whole business of inheriting > > > tablespace from the partitioned table is broken and should be thrown > > >

Re: pg_dump is broken for partition tablespaces

2019-04-14 Thread Alvaro Herrera
On 2019-Apr-14, Andres Freund wrote: > On 2019-04-14 10:38:05 -0400, Tom Lane wrote: > > It's entirely possible BTW that this whole business of inheriting > > tablespace from the partitioned table is broken and should be thrown > > out. I certainly don't see any compelling reason for partitions

Re: pg_dump is broken for partition tablespaces

2019-04-14 Thread Andres Freund
Hi, On 2019-04-14 10:38:05 -0400, Tom Lane wrote: > It's entirely possible BTW that this whole business of inheriting > tablespace from the partitioned table is broken and should be thrown > out. I certainly don't see any compelling reason for partitions to > act differently from regular tables

Re: pg_dump is broken for partition tablespaces

2019-04-14 Thread Tom Lane
David Rowley writes: > On Mon, 15 Apr 2019 at 02:16, Tom Lane wrote: >> Well, it's not really nice perhaps, but you cannot just put in some >> other concrete tablespace OID instead. What a zero there means is >> "use the database's default tablespace", and the point of it is that >> it still

Re: pg_dump is broken for partition tablespaces

2019-04-14 Thread David Rowley
On Mon, 15 Apr 2019 at 02:16, Tom Lane wrote: > > David Rowley writes: > > I'd say the fact that we populate reltablespace with 0 is a bug as > > it's not going to do what they want after a dump/restore. > > Well, it's not really nice perhaps, but you cannot just put in some > other concrete

Re: pg_dump is broken for partition tablespaces

2019-04-14 Thread Tom Lane
David Rowley writes: > I'd say the fact that we populate reltablespace with 0 is a bug as > it's not going to do what they want after a dump/restore. Well, it's not really nice perhaps, but you cannot just put in some other concrete tablespace OID instead. What a zero there means is "use the

Re: pg_dump is broken for partition tablespaces

2019-04-14 Thread David Rowley
On Sat, 13 Apr 2019 at 11:36, Alvaro Herrera wrote: > Here's a patch to fix the reported problems. It's somewhat invasive, > and I've spent a long time staring at it, so I very much appreciate eyes > on it. I think it's a bit strange that don't store the pg_default's oid in reltablespace for

Re: pg_dump is broken for partition tablespaces

2019-04-12 Thread Alvaro Herrera
On 2019-Mar-06, David Rowley wrote: > Over on [1] Andres pointed out that the pg_dump support for the new to > PG12 tablespace inheritance feature is broken. This is the feature > added in ca4103025dfe26 to allow a partitioned table to have a > tablespace that acts as the default tablespace for

Re: pg_dump is broken for partition tablespaces

2019-04-11 Thread Alvaro Herrera
On 2019-Apr-10, Alvaro Herrera wrote: > This is because ruleutils.c attaches a TABLESPACE clause when asked to > dump an index definition; and tablecmds.c uses ruleutils to deparse the > index definition into something that can be replayed via CREATE INDEX > commands (or ALTER TABLE ADD

Re: pg_dump is broken for partition tablespaces

2019-04-10 Thread Alvaro Herrera
On 2019-Apr-10, Alvaro Herrera wrote: > but the test immediately does this: > > alter table at_partitioned alter column b type numeric using b::numeric; > > and watch what happens! (1663 is pg_default) > > alvherre=# select relname, reltablespace from pg_class where relname like >

Re: pg_dump is broken for partition tablespaces

2019-04-10 Thread Alvaro Herrera
On 2019-Apr-10, Andres Freund wrote: > Hi, > > On 2019-04-10 09:28:21 -0400, Alvaro Herrera wrote: > > So I think that apart from David's patch, we should just document all > > these things carefully. > > Yea, I think that's the most important part. > > I'm not convinced that we should have

Re: pg_dump is broken for partition tablespaces

2019-04-10 Thread Andres Freund
Hi, On 2019-04-10 09:28:21 -0400, Alvaro Herrera wrote: > So I think that apart from David's patch, we should just document all > these things carefully. Yea, I think that's the most important part. I'm not convinced that we should have any inheriting behaviour btw - it seems like there's a lot

Re: pg_dump is broken for partition tablespaces

2019-04-09 Thread David Rowley
On Wed, 10 Apr 2019 at 11:05, Alvaro Herrera wrote: > > On 2019-Apr-09, Alvaro Herrera wrote: > > > There is one deficiency that needs to be solved in order for this to > > work fully: currently there is no way to reset "reltablespace" to 0. > > Therefore I propose to add > ALTER TABLE tb ...

Re: pg_dump is broken for partition tablespaces

2019-04-09 Thread Alvaro Herrera
On 2019-Apr-09, Alvaro Herrera wrote: > There is one deficiency that needs to be solved in order for this to > work fully: currently there is no way to reset "reltablespace" to 0. Therefore I propose to add ALTER TABLE tb ... RESET TABLESPACE; which sets reltablespace to 0, and it would work

Re: pg_dump is broken for partition tablespaces

2019-04-09 Thread Alvaro Herrera
On 2019-Mar-06, Andres Freund wrote: > I don't think the argument that the user intended to explicitly set a > tablespace holds much water if it was just set via default_tablespace, > rather than an explicit TABLESPACE. I think iff you really want > something like this feature, you'd have to

Re: pg_dump is broken for partition tablespaces

2019-04-09 Thread Alvaro Herrera
On 2019-Apr-09, Alvaro Herrera wrote: > However, in order to fix the pg_dump behavior for the partitioned rel, > we would need to emit the tablespace differently, i.e. not use SET > default_tablespace, but instead attach the tablespace clause to the > CREATE TABLE line. > > I'll go have a look

Re: pg_dump is broken for partition tablespaces

2019-04-09 Thread Alvaro Herrera
On 2019-Mar-07, David Rowley wrote: > On Thu, 7 Mar 2019 at 11:37, Andres Freund wrote: > > > > On 2019-03-07 11:31:15 +1300, David Rowley wrote: > > > Do you think it's fine to reword the docs to make this point more > > > clear, or do you see this as a fundamental problem with the patch? > > >

Re: pg_dump is broken for partition tablespaces

2019-04-09 Thread Alvaro Herrera
On 2019-Mar-06, Andres Freund wrote: > > I also find it far from clear that: > > > > > > The tablespace_name is > > the name > > of the tablespace in which the new table is to be created. > > If not specified, > >is consulted, or > >if the table is

Re: pg_dump is broken for partition tablespaces

2019-03-06 Thread Andres Freund
On 2019-03-07 15:25:34 +1300, David Rowley wrote: > On Thu, 7 Mar 2019 at 11:37, Andres Freund wrote: > > > > On 2019-03-07 11:31:15 +1300, David Rowley wrote: > > > Do you think it's fine to reword the docs to make this point more > > > clear, or do you see this as a fundamental problem with the

Re: pg_dump is broken for partition tablespaces

2019-03-06 Thread David Rowley
On Thu, 7 Mar 2019 at 11:37, Andres Freund wrote: > > On 2019-03-07 11:31:15 +1300, David Rowley wrote: > > Do you think it's fine to reword the docs to make this point more > > clear, or do you see this as a fundamental problem with the patch? > > Hm, both? I mean I wouldn't necessarily

Re: pg_dump is broken for partition tablespaces

2019-03-06 Thread Andres Freund
Hi, On 2019-03-07 11:31:15 +1300, David Rowley wrote: > On Thu, 7 Mar 2019 at 05:17, Andres Freund wrote: > > I'm also concerned that the the current catalog representation isn't > > right. As I said: > > > > > I also find it far from clear that: > > > > > > > > > The

Re: pg_dump is broken for partition tablespaces

2019-03-06 Thread David Rowley
On Thu, 7 Mar 2019 at 05:17, Andres Freund wrote: > I'm also concerned that the the current catalog representation isn't > right. As I said: > > > I also find it far from clear that: > > > > > > The tablespace_name is > > the name > > of the tablespace in which the new

Re: pg_dump is broken for partition tablespaces

2019-03-06 Thread David Rowley
On Thu, 7 Mar 2019 at 03:36, Tom Lane wrote: > > David Rowley writes: > > As far as I can see, the biggest fundamental difference with doing > > things this way will be that the column order of partitions will be > > preserved, where before it would inherit the order of the partitioned > >

Re: pg_dump is broken for partition tablespaces

2019-03-06 Thread Andres Freund
Hi, On 2019-03-06 19:45:06 +1300, David Rowley wrote: > Over on [1] Andres pointed out that the pg_dump support for the new to > PG12 tablespace inheritance feature is broken. This is the feature > added in ca4103025dfe26 to allow a partitioned table to have a > tablespace that acts as the

Re: pg_dump is broken for partition tablespaces

2019-03-06 Thread Tom Lane
David Rowley writes: > As far as I can see, the biggest fundamental difference with doing > things this way will be that the column order of partitions will be > preserved, where before it would inherit the order of the partitioned > table. I'm a little unsure if doing this column reordering was

Re: pg_dump is broken for partition tablespaces

2019-03-06 Thread David Rowley
On Wed, 6 Mar 2019 at 20:19, Michael Paquier wrote: > > On Wed, Mar 06, 2019 at 07:45:06PM +1300, David Rowley wrote: > > Can anyone see any fundamental reason that we should not create a > > partitioned table by doing CREATE TABLE followed by ATTACH PARTITION? > > If not, I'll write a patch

Re: pg_dump is broken for partition tablespaces

2019-03-05 Thread Michael Paquier
On Wed, Mar 06, 2019 at 07:45:06PM +1300, David Rowley wrote: > Partitioned indexes have this similar inherit tablespace from parent > feature, so ca4103025dfe26 was intended to align the behaviour of the > two. Partitioned indexes happen not to suffer from the same issue as > the indexes are