Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-09-04 Thread Bruce Momjian
On Thu, Sep 4, 2014 at 03:24:05PM -0600, Noah Yetter wrote: > Doing the upgrade with an installation built from REL9_3_STABLE at > commit 52eed3d4267faf671dae0450d99982cb9ba1ac52 was successful. > > The view that I saw get re-created as a table doesn't have any circular > references, or indeed an

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-09-04 Thread Noah Yetter
Doing the upgrade with an installation built from REL9_3_STABLE at commit 52eed3d4267faf671dae0450d99982cb9ba1ac52 was successful. The view that I saw get re-created as a table doesn't have any circular references, or indeed any references to other views, nor do any other views reference it. But

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-09-04 Thread Bruce Momjian
On Thu, Sep 4, 2014 at 03:48:17PM -0400, Robert Haas wrote: > On Thu, Sep 4, 2014 at 3:35 PM, Bruce Momjian wrote: > >> At any rate, I've additionally observed that the relation which is blowing > >> up > >> pg_upgrade is a VIEW in the source cluster but gets created as a TABLE in > >> the > >>

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-09-04 Thread Robert Haas
On Thu, Sep 4, 2014 at 3:35 PM, Bruce Momjian wrote: >> At any rate, I've additionally observed that the relation which is blowing up >> pg_upgrade is a VIEW in the source cluster but gets created as a TABLE in the >> upgraded cluster, which may better explain why it had no toast table before >>

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-09-04 Thread Bruce Momjian
On Thu, Sep 4, 2014 at 01:14:01PM -0600, Noah Yetter wrote: > Isn't that exactly what the release note says?  > "where the new server creates a TOAST table but the old version did not"  > vs.  > "where the new cluster needs a TOAST table that the old cluster didn't" Sorry, yes, I got confused. W

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-09-04 Thread Noah Yetter
Isn't that exactly what the release note says? "where the new server creates a TOAST table but the old version did not" vs. "where the new cluster needs a TOAST table that the old cluster didn't" At any rate, I've additionally observed that the relation which is blowing up pg_upgrade is a VIEW in

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-09-04 Thread David G Johnston
On Thu, Sep 4, 2014 at 2:39 PM, Bruce Momjian [via PostgreSQL] < ml-node+s1045698n5817828...@n5.nabble.com> wrote: > On Thu, Sep 4, 2014 at 11:37:27AM -0600, Noah Yetter wrote: > > > The 9.3.5 release notes contain... > > > > > > • Fix pg_upgrade for cases where the new server creates a TOAST t

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-09-04 Thread Bruce Momjian
On Thu, Sep 4, 2014 at 11:37:27AM -0600, Noah Yetter wrote: > The 9.3.5 release notes contain... > > > • Fix pg_upgrade for cases where the new server creates a TOAST table but > the > old version did not (Bruce Momjian) > > This rare situation would manifest as "relation OID mismatc

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-09-04 Thread Noah Yetter
The 9.3.5 release notes contain... - Fix pg_upgrade for cases where the new server creates a TOAST table but the old version did not (Bruce Momjian) This rare situation would manifest as "relation OID mismatch" errors. ...which I thought was this bug, hence my confusion. If anyon

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-09-03 Thread Bruce Momjian
On Wed, Sep 3, 2014 at 05:12:30PM -0600, Noah Yetter wrote: > I'm not sure it's fixed.  I am attempting a pg_upgrade from 9.2.8 to 9.3.5 and > it dies like so: > > (...many relations restoring successfully snipped...) > pg_restore: creating SEQUENCE address_address_id_seq > pg_restore: [archiver

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-09-03 Thread David G Johnston
Based upon the dates the noted patch is not in 9.3.5; which was released a couple of weeks previous to it being committed. David J. nyetter wrote > I'm not sure it's fixed. I am attempting a pg_upgrade from 9.2.8 to 9.3.5 > and it dies like so: > > (...many relations restoring successfully sni

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-09-03 Thread Noah Yetter
I'm not sure it's fixed. I am attempting a pg_upgrade from 9.2.8 to 9.3.5 and it dies like so: (...many relations restoring successfully snipped...) pg_restore: creating SEQUENCE address_address_id_seq pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-08-07 Thread Bruce Momjian
On Tue, Aug 5, 2014 at 07:31:21PM -0400, Bruce Momjian wrote: > On Thu, Jul 10, 2014 at 06:38:26PM -0400, Bruce Momjian wrote: > > On Thu, Jul 10, 2014 at 06:17:14PM -0400, Bruce Momjian wrote: > > > Well, we are going to need to call internal C functions, often bypassing > > > their typical call

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-08-05 Thread Bruce Momjian
On Thu, Jul 10, 2014 at 06:38:26PM -0400, Bruce Momjian wrote: > On Thu, Jul 10, 2014 at 06:17:14PM -0400, Bruce Momjian wrote: > > Well, we are going to need to call internal C functions, often bypassing > > their typical call sites and the assumption about locking, etc. Perhaps > > this could be

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-16 Thread Robert Haas
On Wed, Jul 16, 2014 at 10:19 AM, Bruce Momjian wrote: > On Mon, Jul 14, 2014 at 09:40:33PM +0200, Andres Freund wrote: >> On 2014-07-11 09:55:34 -0400, Bruce Momjian wrote: >> > On Fri, Jul 11, 2014 at 09:48:06AM -0400, Bruce Momjian wrote: >> > > > Uh, why does this need to be in ALTER TABLE? C

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-16 Thread Andres Freund
On 2014-07-16 10:19:05 -0400, Bruce Momjian wrote: > > > What would be nice is if I could mark just the tables that will need > > > toast tables created in that later phase (those tables that didn't have > > > a toast table in the old cluster, but need one in the new cluster). > > > However, I can

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-16 Thread Bruce Momjian
On Mon, Jul 14, 2014 at 09:40:33PM +0200, Andres Freund wrote: > On 2014-07-11 09:55:34 -0400, Bruce Momjian wrote: > > On Fri, Jul 11, 2014 at 09:48:06AM -0400, Bruce Momjian wrote: > > > > Uh, why does this need to be in ALTER TABLE? Can't this be part of > > > > table creation done by pg_dump?

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-14 Thread Andres Freund
On 2014-07-11 09:55:34 -0400, Bruce Momjian wrote: > On Fri, Jul 11, 2014 at 09:48:06AM -0400, Bruce Momjian wrote: > > > Uh, why does this need to be in ALTER TABLE? Can't this be part of > > > table creation done by pg_dump? > > > > Uh, I think you need to read the thread. We have to delay the

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-14 Thread Bruce Momjian
On Mon, Jul 14, 2014 at 11:26:19AM -0400, Robert Haas wrote: > On Fri, Jul 11, 2014 at 9:55 AM, Bruce Momjian wrote: > > On Fri, Jul 11, 2014 at 09:48:06AM -0400, Bruce Momjian wrote: > >> > Uh, why does this need to be in ALTER TABLE? Can't this be part of > >> > table creation done by pg_dump?

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-14 Thread Fabrízio de Royes Mello
On Mon, Jul 14, 2014 at 12:26 PM, Robert Haas wrote: > > On Fri, Jul 11, 2014 at 9:55 AM, Bruce Momjian wrote: > > On Fri, Jul 11, 2014 at 09:48:06AM -0400, Bruce Momjian wrote: > >> > Uh, why does this need to be in ALTER TABLE? Can't this be part of > >> > table creation done by pg_dump? > >>

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-14 Thread Robert Haas
On Fri, Jul 11, 2014 at 9:55 AM, Bruce Momjian wrote: > On Fri, Jul 11, 2014 at 09:48:06AM -0400, Bruce Momjian wrote: >> > Uh, why does this need to be in ALTER TABLE? Can't this be part of >> > table creation done by pg_dump? >> >> Uh, I think you need to read the thread. We have to delay the

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-11 Thread Bruce Momjian
On Fri, Jul 11, 2014 at 09:48:06AM -0400, Bruce Momjian wrote: > > Uh, why does this need to be in ALTER TABLE? Can't this be part of > > table creation done by pg_dump? > > Uh, I think you need to read the thread. We have to delay the toast > creation part so we don't use an oid that will later

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-11 Thread Bruce Momjian
On Fri, Jul 11, 2014 at 12:18:40AM -0400, Alvaro Herrera wrote: > Bruce Momjian wrote: > > On Thu, Jul 10, 2014 at 06:38:26PM -0400, Bruce Momjian wrote: > > > > I have thought some more on this. I thought I would need to open > > > pg_class in C and do complex backend stuff, but I now realize I

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-10 Thread Alvaro Herrera
Bruce Momjian wrote: > On Thu, Jul 10, 2014 at 06:38:26PM -0400, Bruce Momjian wrote: > > I have thought some more on this. I thought I would need to open > > pg_class in C and do complex backend stuff, but I now realize I can do > > it from libpq, and just call ALTER TABLE and I think that alway

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-10 Thread Bruce Momjian
On Thu, Jul 10, 2014 at 06:38:26PM -0400, Bruce Momjian wrote: > On Thu, Jul 10, 2014 at 06:17:14PM -0400, Bruce Momjian wrote: > > Well, we are going to need to call internal C functions, often bypassing > > their typical call sites and the assumption about locking, etc. Perhaps > > this could be

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-10 Thread Bruce Momjian
On Thu, Jul 10, 2014 at 06:17:14PM -0400, Bruce Momjian wrote: > Well, we are going to need to call internal C functions, often bypassing > their typical call sites and the assumption about locking, etc. Perhaps > this could be done from a plpgsql function. We could add and drop a > dummy column

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-10 Thread Bruce Momjian
On Thu, Jul 10, 2014 at 11:11:19PM +0200, Andres Freund wrote: > On 2014-07-10 16:33:40 -0400, Bruce Momjian wrote: > > On Thu, Jul 10, 2014 at 10:46:30AM -0400, Robert Haas wrote: > > > > Agreed. I am now thinking we could harness the code that already exists > > > > to optionally add a TOAST tab

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-10 Thread Andres Freund
On 2014-07-10 16:33:40 -0400, Bruce Momjian wrote: > On Thu, Jul 10, 2014 at 10:46:30AM -0400, Robert Haas wrote: > > > Agreed. I am now thinking we could harness the code that already exists > > > to optionally add a TOAST table as part of ALTER TABLE ADD COLUMN. We > > > would just need an entr

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-10 Thread Bruce Momjian
On Thu, Jul 10, 2014 at 10:46:30AM -0400, Robert Haas wrote: > > Agreed. I am now thinking we could harness the code that already exists > > to optionally add a TOAST table as part of ALTER TABLE ADD COLUMN. We > > would just need an entry point to call it from pg_upgrade, either via an > > SQL c

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-10 Thread Robert Haas
On Wed, Jul 9, 2014 at 12:09 PM, Bruce Momjian wrote: >> To me, that sounds vastly more complicated and error-prone than >> forcing the TOAST tables to be added in a second pass as Andres >> suggested. >> >> But I just work here. > > Agreed. I am now thinking we could harness the code that alread

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-09 Thread Bruce Momjian
On Wed, Jul 9, 2014 at 10:13:17AM -0400, Robert Haas wrote: > >> Uh, I guess we could write some code that iterates over all tables and > >> finds the tables that should have TOAST tables, but don't (because > >> binary-upgrade backend mode suppressed their creation), and adds them. > >> > >> Howe

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-09 Thread Robert Haas
>> Uh, I guess we could write some code that iterates over all tables and >> finds the tables that should have TOAST tables, but don't (because >> binary-upgrade backend mode suppressed their creation), and adds them. >> >> However, that would be a lot of code and might be risky to backpatch. >> Th

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-07 Thread Bruce Momjian
On Mon, Jul 7, 2014 at 01:44:59PM -0400, Bruce Momjian wrote: > On Mon, Jul 7, 2014 at 11:24:51AM -0400, Robert Haas wrote: > > > As far as the reusing of oids, we don't set the oid counter until after > > > the restore, so any new unmatched toast table would given a very low > > > oid. Since we

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-07 Thread Bruce Momjian
On Mon, Jul 7, 2014 at 11:24:51AM -0400, Robert Haas wrote: > > As far as the reusing of oids, we don't set the oid counter until after > > the restore, so any new unmatched toast table would given a very low > > oid. Since we restore in oid order, for an oid to be assigned that was > > used in t

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-07 Thread Bruce Momjian
On Fri, Jul 4, 2014 at 11:12:58PM -0400, Bruce Momjian wrote: > On Fri, Jul 4, 2014 at 12:01:37AM -0400, Bruce Momjian wrote: > > > The most robust, but not trivial, approach seems to be to prevent toast > > > table creation if there wasn't a set_next_toast_pg_class_oid(). Then, > > > after all r

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-07 Thread Robert Haas
On Fri, Jul 4, 2014 at 11:12 PM, Bruce Momjian wrote: > On Fri, Jul 4, 2014 at 12:01:37AM -0400, Bruce Momjian wrote: >> > The most robust, but not trivial, approach seems to be to prevent toast >> > table creation if there wasn't a set_next_toast_pg_class_oid(). Then, >> > after all relations ar

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-04 Thread Bruce Momjian
On Fri, Jul 4, 2014 at 12:01:37AM -0400, Bruce Momjian wrote: > > The most robust, but not trivial, approach seems to be to prevent toast > > table creation if there wasn't a set_next_toast_pg_class_oid(). Then, > > after all relations are created, iterate over all pg_class entries that > > possib

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-03 Thread Bruce Momjian
On Thu, Jul 3, 2014 at 11:55:40PM +0200, Andres Freund wrote: > I don't think it's just that simple unfortunately. If pg_class entries > get created that didn't exist on the old server there's a chance for oid > conflicts. Consider > > SELECT binary_upgrade.set_next_heap_pg_class_oid('17094'::pg_

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-03 Thread Bruce Momjian
On Thu, Jul 3, 2014 at 05:09:41PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > I have had trouble getting a table schema that is causing problems, but > > received a report via EDB support recently that had a simple schema > > (anonymized): > > ... > > needs_toast_table() computes the lengt

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-03 Thread Andres Freund
On 2014-07-03 17:09:41 -0400, Tom Lane wrote: > Bruce Momjian writes: > > I have had trouble getting a table schema that is causing problems, but > > received a report via EDB support recently that had a simple schema > > (anonymized): > > ... > > needs_toast_table() computes the length of this ta

Re: [HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-03 Thread Tom Lane
Bruce Momjian writes: > I have had trouble getting a table schema that is causing problems, but > received a report via EDB support recently that had a simple schema > (anonymized): > ... > needs_toast_table() computes the length of this table as 2024 bytes in > 9.0, and 2064 bytes on 9.1, with th

[HACKERS] Pg_upgrade and toast tables bug discovered

2014-07-03 Thread Bruce Momjian
There have been periodic reports of pg_upgrade errors related to toast tables. The most recent one was from May of this year: http://www.postgresql.org/message-id/flat/20140520202223.gb3...@momjian.us#20140520202223.gb3...@momjian.us There error was: Copying user relation files