Re: [HACKERS] pg_rawdump

2010-10-21 Thread Stephen R. van den Berg
Robert Haas wrote: On Wed, Oct 20, 2010 at 5:30 PM, Stephen R. van den Berg s...@cuci.nl wrote: Ideal would be: put the table-oid inside the header of each page (either in the official header, or in the special area). This way even lost blocks can be correlated to the same table. I'd still

Re: [HACKERS] pg_rawdump

2010-10-21 Thread Tom Lane
Stephen R. van den Berg s...@cuci.nl writes: Robert Haas wrote: and break on-disk compatibility just to make it easier to If it's inserted in the special area, it will not break any compatibility. I'll tell you what I really don't like about this proposal: we discuss some scheme or other for

Re: [HACKERS] pg_rawdump

2010-10-21 Thread Stephen R. van den Berg
Tom Lane wrote: Stephen R. van den Berg s...@cuci.nl writes: If it's inserted in the special area, it will not break any compatibility. I'll tell you what I really don't like about this proposal: we discuss some scheme or other for taking over the special space in heap pages at least once a

Re: [HACKERS] pg_rawdump

2010-10-21 Thread Greg Stark
On Thu, Oct 21, 2010 at 11:30 AM, Stephen R. van den Berg s...@cuci.nl wrote: For the recovery information I'd like to reserve: identifier: 00: table OID            01: table layout So here's a proposal for something that could maybe be implemented. I think I'm leaning against this currently

Re: [HACKERS] pg_rawdump

2010-10-21 Thread Tom Lane
Stephen R. van den Berg s...@cuci.nl writes: Tom Lane wrote: There are way too many scenarios where you'll have no hope of doing any such manual recovery anyway. True. It's all a matter of statistics. Judging by the number of reports I find by googling net-history, I'd have to conclude

Re: [HACKERS] pg_rawdump

2010-10-21 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Most of the data-loss reports I've seen appeared to come from people who wouldn't be capable of doing such recovery work even if better tools were available. No doubt; but the recovery work often winds up in the hands of people with more skills than those

Re: [HACKERS] pg_rawdump

2010-10-21 Thread Robert Haas
On Thu, Oct 21, 2010 at 5:21 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Tom Lane t...@sss.pgh.pa.us wrote: Most of the data-loss reports I've seen appeared to come from people who wouldn't be capable of doing such recovery work even if better tools were available. No doubt; but

Re: [HACKERS] pg_rawdump

2010-10-21 Thread Stephen R. van den Berg
Kevin Grittner wrote: Tom Lane t...@sss.pgh.pa.us wrote: Most of the data-loss reports I've seen appeared to come from people who wouldn't be capable of doing such recovery work even if better tools were available. No doubt; but the recovery work often winds up in the hands of people with more

Re: [HACKERS] pg_rawdump

2010-10-21 Thread Bruce Momjian
Greg Stark wrote: On Tue, Oct 19, 2010 at 1:12 PM, Stephen R. van den Berg s...@cuci.nl wrote: In order to simplify recovery at this point (enormously), it would have been very helpful (at almost negligible cost), to have the name of the table, the name of the columns, and the types of the

Re: [HACKERS] pg_rawdump

2010-10-21 Thread Stephen R. van den Berg
Greg Stark wrote: On Thu, Oct 21, 2010 at 11:30 AM, Stephen R. van den Berg s...@cuci.nl wrote: For the recovery information I'd like to reserve: identifier: 00: table OID ? ? ? ? ? ?01: table layout So here's a proposal for something that could maybe be implemented. I think I'm leaning

Re: [HACKERS] pg_rawdump

2010-10-21 Thread Stephen R. van den Berg
Bruce Momjian wrote: Greg Stark wrote: On Tue, Oct 19, 2010 at 1:12 PM, Stephen R. van den Berg s...@cuci.nl wrote: In order to simplify recovery at this point (enormously), it would have been very helpful (at almost negligible cost), to have the name of the table, the name of the

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Stephen R. van den Berg
Tom Lane wrote: Stephen R. van den Berg s...@cuci.nl writes: In order to simplify recovery at this point (enormously), it would have been very helpful (at almost negligible cost), to have the name of the table, the name of the columns, and the types of the columns available. Why don't we

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Stephen R. van den Berg
Roberto Mello wrote: On Tue, Oct 19, 2010 at 6:13 PM, Stephen R. van den Berg s...@cuci.nl wrote: Greg Stark wrote: premise this on the idea that you've lost everything in the catalog but not the data in other tables. Which seems like a narrow use case. It happens, more often than you'd think.

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Tom Lane
Stephen R. van den Berg s...@cuci.nl writes: It's just that matching table and file, and subsequently figuring out some missing columns which may have been added/removed later, can be rather timeconsuming and could be made a lot easier (not necessarily perfect) if that information would have

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Stephen R. van den Berg
Tom Lane wrote: Stephen R. van den Berg s...@cuci.nl writes: It's just that matching table and file, and subsequently figuring out some missing columns which may have been added/removed later, can be rather timeconsuming and could be made a lot easier (not necessarily perfect) if that

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Aidan Van Dyk
On Wed, Oct 20, 2010 at 10:28 AM, Tom Lane t...@sss.pgh.pa.us wrote: Stephen R. van den Berg s...@cuci.nl writes: It's just that matching table and file, and subsequently figuring out some missing columns which may have been added/removed later, can be rather timeconsuming and could be made a

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Tom Lane
Aidan Van Dyk ai...@highrise.ca writes: On Wed, Oct 20, 2010 at 10:28 AM, Tom Lane t...@sss.pgh.pa.us wrote: I can see the potential usefulness of a self-documenting table storage format, but this proposal isn't that; it's just an unreliable kluge. If we're looking to have any sort of out of

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Aidan Van Dyk
On Wed, Oct 20, 2010 at 1:09 PM, Tom Lane t...@sss.pgh.pa.us wrote: [ thinks for a bit... ]  Perhaps we could stick some sort of unique ID into tables, which could be correlated to the same unique ID appearing in a metadata fork. It would be awfully nice if the table name was in the first few

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Stephen R. van den Berg
Tom Lane wrote: Aidan Van Dyk ai...@highrise.ca writes: If we're looking to have any sort of out of catalog documentation of table storage format, what about just having a new relation fork that just appends each and every change made to the table formats, including ones rolled back, etc. I

Re: [HACKERS] pg_rawdump

2010-10-20 Thread Robert Haas
On Wed, Oct 20, 2010 at 5:30 PM, Stephen R. van den Berg s...@cuci.nl wrote: Ideal would be: put the table-oid inside the header of each page (either in the official header, or in the special area). This way even lost blocks can be correlated to the same table. I'd still vote for the latest

[HACKERS] pg_rawdump

2010-10-19 Thread Stephen R. van den Berg
I spent yesterday writing a new tool pg_rawdump (which will be released as open source in due course), which takes the table files in an arbitrary pgsql database, and is able to transform those back into tables (including toast values). In the course of doing this (a customer needed it because he

Re: [HACKERS] pg_rawdump

2010-10-19 Thread Greg Stark
On Tue, Oct 19, 2010 at 1:12 PM, Stephen R. van den Berg s...@cuci.nl wrote: In order to simplify recovery at this point (enormously), it would have been very helpful (at almost negligible cost), to have the name of the table, the name of the columns, and the types of the columns available.

Re: [HACKERS] pg_rawdump

2010-10-19 Thread Stephen R. van den Berg
Greg Stark wrote: On Tue, Oct 19, 2010 at 1:12 PM, Stephen R. van den Berg s...@cuci.nl wrote: In order to simplify recovery at this point (enormously), it would have been very helpful (at almost negligible cost), to have the name of the table, the name of the columns, and the types of the

Re: [HACKERS] pg_rawdump

2010-10-19 Thread Tom Lane
Stephen R. van den Berg s...@cuci.nl writes: In order to simplify recovery at this point (enormously), it would have been very helpful (at almost negligible cost), to have the name of the table, the name of the columns, and the types of the columns available. Why don't we insert that data

Re: [HACKERS] pg_rawdump

2010-10-19 Thread Roberto Mello
On Tue, Oct 19, 2010 at 6:13 PM, Stephen R. van den Berg s...@cuci.nl wrote: Greg Stark wrote: But I'm not sure how useful. I mean, you can't really decipher everything properly without the data in the catalog -- and you have to premise this on the idea that you've lost everything in the