Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-07 Thread hubert depesz lubaczewski
On Tue, Sep 06, 2011 at 09:21:02PM -0400, Bruce Momjian wrote: Tom Lane wrote: hubert depesz lubaczewski dep...@depesz.com writes: Worked a bit to get the ltree problem down to smallest possible, repeatable, situation. I looked at this again and verified that indeed, commit

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-07 Thread Bruce Momjian
Bruce Momjian wrote: Tom Lane wrote: hubert depesz lubaczewski dep...@depesz.com writes: Worked a bit to get the ltree problem down to smallest possible, repeatable, situation. I looked at this again and verified that indeed, commit 8eee65c996048848c20f6637c1d12b319a4ce244

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-06 Thread Peter Eisentraut
On mån, 2011-09-05 at 16:53 -0400, Bruce Momjian wrote: hubert depesz lubaczewski wrote: Good. Is it possible to compile with debug symbols, -g? Odd you are crashing in libc. this had debug: ./configure \ --prefix=/opt/pgsql-9.0.5a-int \ --enable-debug \

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-06 Thread hubert depesz lubaczewski
On Mon, Sep 05, 2011 at 05:26:00PM -0400, Tom Lane wrote: Bruce Momjian br...@momjian.us writes: Odd it is dying in the memory freeing at executor close --- not in the ltree code. Doesn't seem odd. The glibc complaint previously shown already indicates this is a memory stomp problem.

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-06 Thread hubert depesz lubaczewski
Hi, Worked a bit to get the ltree problem down to smallest possible, repeatable, situation. Initial setup: 1. PostgreSQL 8.3.11, configured with: ./configure\ --prefix=/opt/pgsql-8.3.11-int \ --disable-rpath\ --without-perl

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-06 Thread Tom Lane
hubert depesz lubaczewski dep...@depesz.com writes: Worked a bit to get the ltree problem down to smallest possible, repeatable, situation. I looked at this again and verified that indeed, commit 8eee65c996048848c20f6637c1d12b319a4ce244 introduced an incompatible change into the on-disk format

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-06 Thread Bruce Momjian
Tom Lane wrote: hubert depesz lubaczewski dep...@depesz.com writes: Worked a bit to get the ltree problem down to smallest possible, repeatable, situation. I looked at this again and verified that indeed, commit 8eee65c996048848c20f6637c1d12b319a4ce244 introduced an incompatible change

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread hubert depesz lubaczewski
On Thu, Sep 01, 2011 at 08:05:51AM +0200, hubert depesz lubaczewski wrote: On Wed, Aug 31, 2011 at 09:54:20PM -0400, Bruce Momjian wrote: Working with depesz, I have found the cause. The code I added to fix pg_upgrade in 9.0.4 and earlier releases didn't handle old 8.3 servers properly. I

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Bruce Momjian
hubert depesz lubaczewski wrote: On Thu, Sep 01, 2011 at 08:05:51AM +0200, hubert depesz lubaczewski wrote: On Wed, Aug 31, 2011 at 09:54:20PM -0400, Bruce Momjian wrote: Working with depesz, I have found the cause. The code I added to fix pg_upgrade in 9.0.4 and earlier releases didn't

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread hubert depesz lubaczewski
On Mon, Sep 05, 2011 at 05:48:50PM +0200, hubert depesz lubaczewski wrote: On Thu, Sep 01, 2011 at 08:05:51AM +0200, hubert depesz lubaczewski wrote: On Wed, Aug 31, 2011 at 09:54:20PM -0400, Bruce Momjian wrote: Working with depesz, I have found the cause. The code I added to fix

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Bruce Momjian
Bruce Momjian wrote: hubert depesz lubaczewski wrote: On Wed, Aug 31, 2011 at 01:23:05PM -0400, Bruce Momjian wrote: Can you get me the 9.0.X pg_class.relfrozenxid for the toast and heap tables involved? Sure: =# select oid::regclass, relfrozenxid from pg_class where relname in

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Bruce Momjian
hubert depesz lubaczewski wrote: I'm not sure if it's upgrade thing, or is it because of error in ltree/compilation, but it looks bad. Is there any more info I could show/gather to help debug the issue? I am confused by the error --- is it not loading, or can you get a backtrace of the

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread hubert depesz lubaczewski
On Mon, Sep 05, 2011 at 02:18:18PM -0400, Bruce Momjian wrote: hubert depesz lubaczewski wrote: I'm not sure if it's upgrade thing, or is it because of error in ltree/compilation, but it looks bad. Is there any more info I could show/gather to help debug the issue? I am confused by

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Bruce Momjian
hubert depesz lubaczewski wrote: On Mon, Sep 05, 2011 at 02:18:18PM -0400, Bruce Momjian wrote: hubert depesz lubaczewski wrote: I'm not sure if it's upgrade thing, or is it because of error in ltree/compilation, but it looks bad. Is there any more info I could show/gather to help

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Tom Lane
hubert depesz lubaczewski dep...@depesz.com writes: On Mon, Sep 05, 2011 at 02:18:18PM -0400, Bruce Momjian wrote: If I had to take a guess, it would be that there is some ltree incompatibility from PG 8.3 that we didn't know about. it's possible. [ checks the git history... ] This 8.4

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread hubert depesz lubaczewski
On Mon, Sep 05, 2011 at 02:51:12PM -0400, Bruce Momjian wrote: hubert depesz lubaczewski wrote: On Mon, Sep 05, 2011 at 02:18:18PM -0400, Bruce Momjian wrote: hubert depesz lubaczewski wrote: I'm not sure if it's upgrade thing, or is it because of error in ltree/compilation, but it

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Bruce Momjian
hubert depesz lubaczewski wrote: On Mon, Sep 05, 2011 at 02:51:12PM -0400, Bruce Momjian wrote: hubert depesz lubaczewski wrote: On Mon, Sep 05, 2011 at 02:18:18PM -0400, Bruce Momjian wrote: hubert depesz lubaczewski wrote: I'm not sure if it's upgrade thing, or is it because of

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread hubert depesz lubaczewski
On Mon, Sep 05, 2011 at 04:43:47PM -0400, Bruce Momjian wrote: hubert depesz lubaczewski wrote: On Mon, Sep 05, 2011 at 02:51:12PM -0400, Bruce Momjian wrote: hubert depesz lubaczewski wrote: On Mon, Sep 05, 2011 at 02:18:18PM -0400, Bruce Momjian wrote: hubert depesz lubaczewski

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Bruce Momjian
hubert depesz lubaczewski wrote: Good. Is it possible to compile with debug symbols, -g? Odd you are crashing in libc. this had debug: ./configure \ --prefix=/opt/pgsql-9.0.5a-int \ --enable-debug \ --disable-rpath \ --without-perl \

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Bruce Momjian
hubert depesz lubaczewski wrote: On Mon, Sep 05, 2011 at 02:51:12PM -0400, Bruce Momjian wrote: hubert depesz lubaczewski wrote: On Mon, Sep 05, 2011 at 02:18:18PM -0400, Bruce Momjian wrote: hubert depesz lubaczewski wrote: I'm not sure if it's upgrade thing, or is it because of

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Tom Lane
Bruce Momjian br...@momjian.us writes: Odd it is dying in the memory freeing at executor close --- not in the ltree code. Doesn't seem odd. The glibc complaint previously shown already indicates this is a memory stomp problem. --enable-cassert might (or might not) provide additional help.

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread daveg
Sorry I missed your reply, catching up now. On Wed, Aug 31, 2011 at 09:56:59PM -0400, Bruce Momjian wrote: daveg wrote: On Mon, Aug 29, 2011 at 07:49:24PM +0200, hubert depesz lubaczewski wrote: On Mon, Aug 29, 2011 at 06:54:41PM +0200, hubert depesz lubaczewski wrote: vacuumdb:

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Bruce Momjian
daveg wrote: Can you tell me what table is showing this error? Does it happen during vacuum? Can you run a vacuum verbose to see what it is throwing the error on? Thanks. This was upgrading from 8.4.8 to 9.0.4. I don't have the running cluster anymore, but I do have tar.gz archives of

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread daveg
On Mon, Sep 05, 2011 at 08:19:21PM -0400, Bruce Momjian wrote: daveg wrote: Can you tell me what table is showing this error? Does it happen during vacuum? Can you run a vacuum verbose to see what it is throwing the error on? Thanks. This was upgrading from 8.4.8 to 9.0.4. I

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-05 Thread Bruce Momjian
daveg wrote: As far as I can tell pg_upgrade never copied any pg_clog files from the old cluster to the new cluster. I wish I had detected that before running the remove_old_cluster.sh script. Wow, no clogs? That would make the system very confused. You can pull the clogs out of

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-09-01 Thread hubert depesz lubaczewski
On Wed, Aug 31, 2011 at 09:54:20PM -0400, Bruce Momjian wrote: Working with depesz, I have found the cause. The code I added to fix pg_upgrade in 9.0.4 and earlier releases didn't handle old 8.3 servers properly. I mistakenly processed toast table with the same pg_dump query as used for

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Bruce Momjian
hubert depesz lubaczewski wrote: On Fri, Aug 26, 2011 at 12:18:55AM -0400, Bruce Momjian wrote: OK, this was very helpful. I found out that there is a bug in current 9.0.X, 9.1.X, and HEAD that I introduced recently when I excluded temp tables. (The bug is not in any released version

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Bruce Momjian
Alvaro Herrera wrote: Excerpts from hubert depesz lubaczewski's message of lun ago 29 14:49:24 -0300 2011: On Mon, Aug 29, 2011 at 06:54:41PM +0200, hubert depesz lubaczewski wrote: On Fri, Aug 26, 2011 at 05:28:35PM +0200, hubert depesz lubaczewski wrote: On Fri, Aug 26, 2011 at

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Robert Haas
On Wed, Aug 31, 2011 at 12:16 PM, Bruce Momjian br...@momjian.us wrote: hubert depesz lubaczewski wrote: On Fri, Aug 26, 2011 at 12:18:55AM -0400, Bruce Momjian wrote: OK, this was very helpful.  I found out that there is a bug in current 9.0.X, 9.1.X, and HEAD that I introduced recently

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread hubert depesz lubaczewski
On Wed, Aug 31, 2011 at 12:16:03PM -0400, Bruce Momjian wrote: hubert depesz lubaczewski wrote: On Fri, Aug 26, 2011 at 12:18:55AM -0400, Bruce Momjian wrote: OK, this was very helpful. I found out that there is a bug in current 9.0.X, 9.1.X, and HEAD that I introduced recently when

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Bruce Momjian
hubert depesz lubaczewski wrote: INFO: vacuuming pg_toast.pg_toast_106668498 vacuumdb: vacuuming of database etsy_v2 failed: ERROR: could not access status of transaction 3429738606 DETAIL: Could not open file pg_clog/0CC6: No such file or directory. Interestingly. In old dir there

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Bruce Momjian
hubert depesz lubaczewski wrote: On Wed, Aug 31, 2011 at 12:16:03PM -0400, Bruce Momjian wrote: hubert depesz lubaczewski wrote: On Fri, Aug 26, 2011 at 12:18:55AM -0400, Bruce Momjian wrote: OK, this was very helpful. I found out that there is a bug in current 9.0.X, 9.1.X,

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Alvaro Herrera
Excerpts from Bruce Momjian's message of mié ago 31 13:23:07 -0300 2011: Alvaro Herrera wrote: Excerpts from hubert depesz lubaczewski's message of lun ago 29 14:49:24 -0300 2011: On Mon, Aug 29, 2011 at 06:54:41PM +0200, hubert depesz lubaczewski wrote: On Fri, Aug 26, 2011 at

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Bruce Momjian
Alvaro Herrera wrote: I don't understand the pg_upgrade code here. It is setting the datfrozenxid and relfrozenxid values to the latest checkpoint's NextXID, /* set pg_class.relfrozenxid */ PQclear(executeQueryOrDie(conn, UPDATE

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread hubert depesz lubaczewski
On Wed, Aug 31, 2011 at 01:23:05PM -0400, Bruce Momjian wrote: Can you get me the 9.0.X pg_class.relfrozenxid for the toast and heap tables involved? Sure: =# select oid::regclass, relfrozenxid from pg_class where relname in ('transactions', 'pg_toast_106668498'); oid

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Bruce Momjian
FYI, I am working with depesz on IM right now and will report back when we have a cause of the bug. FYI, I was without electric power for 53 hours, which is why I am late in replying to this report. --- daveg wrote: On

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Bruce Momjian
hubert depesz lubaczewski wrote: On Wed, Aug 31, 2011 at 01:23:05PM -0400, Bruce Momjian wrote: Can you get me the 9.0.X pg_class.relfrozenxid for the toast and heap tables involved? Sure: =# select oid::regclass, relfrozenxid from pg_class where relname in ('transactions',

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-31 Thread Bruce Momjian
daveg wrote: On Mon, Aug 29, 2011 at 07:49:24PM +0200, hubert depesz lubaczewski wrote: On Mon, Aug 29, 2011 at 06:54:41PM +0200, hubert depesz lubaczewski wrote: vacuumdb: vacuuming of database etsy_v2 failed: ERROR: could not access status of transaction 3429738606 DETAIL: Could not

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-29 Thread hubert depesz lubaczewski
On Fri, Aug 26, 2011 at 05:28:35PM +0200, hubert depesz lubaczewski wrote: On Fri, Aug 26, 2011 at 12:18:55AM -0400, Bruce Momjian wrote: OK, this was very helpful. I found out that there is a bug in current 9.0.X, 9.1.X, and HEAD that I introduced recently when I excluded temp tables.

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-29 Thread hubert depesz lubaczewski
On Mon, Aug 29, 2011 at 06:54:41PM +0200, hubert depesz lubaczewski wrote: On Fri, Aug 26, 2011 at 05:28:35PM +0200, hubert depesz lubaczewski wrote: On Fri, Aug 26, 2011 at 12:18:55AM -0400, Bruce Momjian wrote: OK, this was very helpful. I found out that there is a bug in current

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-29 Thread Alvaro Herrera
Excerpts from hubert depesz lubaczewski's message of lun ago 29 14:49:24 -0300 2011: On Mon, Aug 29, 2011 at 06:54:41PM +0200, hubert depesz lubaczewski wrote: On Fri, Aug 26, 2011 at 05:28:35PM +0200, hubert depesz lubaczewski wrote: On Fri, Aug 26, 2011 at 12:18:55AM -0400, Bruce Momjian

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-29 Thread daveg
On Mon, Aug 29, 2011 at 07:49:24PM +0200, hubert depesz lubaczewski wrote: On Mon, Aug 29, 2011 at 06:54:41PM +0200, hubert depesz lubaczewski wrote: vacuumdb: vacuuming of database etsy_v2 failed: ERROR: could not access status of transaction 3429738606 DETAIL: Could not open file

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-26 Thread hubert depesz lubaczewski
On Fri, Aug 26, 2011 at 12:18:55AM -0400, Bruce Momjian wrote: OK, this was very helpful. I found out that there is a bug in current 9.0.X, 9.1.X, and HEAD that I introduced recently when I excluded temp tables. (The bug is not in any released version of pg_upgrade.) The attached, applied

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-25 Thread Bruce Momjian
hubert depesz lubaczewski wrote: hi I have 8.3.11 database, ~ 600GB in size. I want to upgrade it to 9.0. First, I tried with 9.0.4, and when I hit problem (the same) I tried git, head of 9.0 branch. Good. pg_upgrade_dump_db.sql- pg_upgrade_dump_db.sql--- For binary upgrade, must

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-25 Thread hubert depesz lubaczewski
On Thu, Aug 25, 2011 at 04:33:07PM -0400, Bruce Momjian wrote: The problem appears to be that the Postgres catalogs think there is a toast table for 'actions', while the file system doesn't seem to have such a file. I can you look in pg_class and verify that? SELECT reltoastrelid FROM

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-25 Thread Bruce Momjian
hubert depesz lubaczewski wrote: On Thu, Aug 25, 2011 at 04:33:07PM -0400, Bruce Momjian wrote: The problem appears to be that the Postgres catalogs think there is a toast table for 'actions', while the file system doesn't seem to have such a file. I can you look in pg_class and verify

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-25 Thread hubert depesz lubaczewski
On Thu, Aug 25, 2011 at 04:43:02PM -0400, Bruce Momjian wrote: Please check the old cluster. Sure: =# SELECT reltoastrelid FROM pg_class WHERE relname = 'actions';

Re: [HACKERS] [GENERAL] pg_upgrade problem

2011-08-25 Thread Bruce Momjian
OK, this was very helpful. I found out that there is a bug in current 9.0.X, 9.1.X, and HEAD that I introduced recently when I excluded temp tables. (The bug is not in any released version of pg_upgrade.) The attached, applied patches should fix it for you. I assume you are running 9.0.X, and