Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-11-05 Thread Alvaro Herrera
I'm not sure what to do with this patch. There was some resistance to the idea originally; then after some discussion, there was some apparent agreement that it might be useful on occasion. Later, a patch was posted, but there was almost no review of it; except to say that it should probably be

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-11-05 Thread Amit Kapila
On Monday, November 05, 2012 7:33 PM Alvaro Herrera wrote: I'm not sure what to do with this patch. There was some resistance to the idea originally; then after some discussion, there was some apparent agreement that it might be useful on occasion. Later, a patch was posted, but there was

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-07-04 Thread Amit Kapila
From: Robert Haas [mailto:robertmh...@gmail.com] Sent: Friday, June 22, 2012 8:59 PM On Fri, Jun 22, 2012 at 5:25 AM, Amit Kapila amit.kap...@huawei.com wrote: Based on the discussion and suggestions in this mail chain, following features can be implemented: 1. To compute the value of max LSN

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-23 Thread Amit Kapila
-Original Message- From: Robert Haas [mailto:robertmh...@gmail.com] Sent: Friday, June 22, 2012 8:59 PM To: Amit Kapila Cc: Tom Lane; Alvaro Herrera; Cédric Villemain; Pg Hackers Subject: Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata On Fri, Jun 22, 2012 at 5

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-22 Thread Amit Kapila
Based on the discussion and suggestions in this mail chain, following features can be implemented: 1. To compute the value of max LSN in data pages based on user input whether he wants it for an individual file, a particular directory or whole database. 2a. To search the available WAL files

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-22 Thread Robert Haas
On Fri, Jun 22, 2012 at 5:25 AM, Amit Kapila amit.kap...@huawei.com wrote: Based on the discussion and suggestions in this mail chain, following features can be implemented: 1. To compute the value of max LSN in data pages based on user input whether he wants it for an individual file,   a

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-21 Thread Tom Lane
Amit Kapila amit.kap...@huawei.com writes: The reason I'm concerned about selecting a next-LSN that's certainly beyond every LSN in the database is that not doing so could result in introducing further corruption, which would be entirely avoidable with more care in choosing the next-LSN.

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-21 Thread Amit Kapila
The reason I'm concerned about selecting a next-LSN that's certainly beyond every LSN in the database is that not doing so could result in introducing further corruption, which would be entirely avoidable with more care in choosing the next-LSN. The further corruption can only be

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-20 Thread Amit Kapila
I'm almost inclined to suggest that we not get next-LSN from WAL, but by scanning all the pages in the main data store and computing the max observed LSN. This is clearly not very attractive from a performance standpoint, but it would avoid the obvious failure mode where you lost some

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-20 Thread Aidan Van Dyk
On Wed, Jun 20, 2012 at 9:21 AM, Amit Kapila amit.kap...@huawei.com wrote: Example Scenario - Now assume we have Data files and WAL files intact and only control file is lost. Just so I understand correctly, the aim of this is to fix the situation where out of the thousands of files and

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-20 Thread Tom Lane
Amit Kapila amit.kap...@huawei.com writes: I'm almost inclined to suggest that we not get next-LSN from WAL, but by scanning all the pages in the main data store and computing the max observed LSN. This is clearly not very attractive from a performance standpoint, but it would avoid the

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-20 Thread Fujii Masao
On Wed, Jun 20, 2012 at 12:40 PM, Amit Kapila amit.kap...@huawei.com wrote: I believe if WAL files are proper as mentioned in Alvaro's mail, the purposed logic should generate correct values. Do you see any problem in logic purposed in my original mail. Can I resume my work on this feature?

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-20 Thread Amit Kapila
AFAIK PITR can be used in a scenario where there is a base back-up and we have archived the WAL files after that, now it can use WAL files to apply on base-backup. Yes. If you want to recover the database from the media crash like the corruption of pg_control file, you basically should take

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-20 Thread Amit Kapila
. -Original Message- From: Aidan Van Dyk [mailto:ai...@highrise.ca] Sent: Wednesday, June 20, 2012 7:13 PM To: Amit Kapila Cc: Pg Hackers Subject: Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata On Wed, Jun 20, 2012 at 9:21 AM, Amit Kapila amit.kap...@huawei.com wrote

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-20 Thread Amit Kapila
I think you're missing the point. There is no possible way to guarantee database consistency after applying pg_resetxlog, unless the database had been cleanly shut down beforehand. The reset will lose the xlog information that was needed to restore consistency. So arguing from

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-19 Thread Amit Kapila
I'm almost inclined to suggest that we not get next-LSN from WAL, but by scanning all the pages in the main data store and computing the max observed LSN. This is clearly not very attractive from a performance standpoint, but it would avoid the obvious failure mode where you lost some recent

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-19 Thread Fujii Masao
On Tue, Jun 19, 2012 at 2:44 AM, Amit Kapila amit.kap...@huawei.com wrote: AFAIR you can create pg_control from scratch already with pg_resetxlog. The hard part is coming up with values for the counters, such as the next WAL location.  Some of them such as next OID are pretty harmless if you

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 1:43 AM, Tom Lane t...@sss.pgh.pa.us wrote: Amit Kapila amit.kap...@huawei.com writes: AFAIR you can create pg_control from scratch already with pg_resetxlog. The hard part is coming up with values for the counters, such as the next WAL location.  Some of them such as

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-19 Thread Amit Kapila
it will be difficult to use PITR in this scenario. -Original Message- From: Fujii Masao [mailto:masao.fu...@gmail.com] Sent: Tuesday, June 19, 2012 7:44 PM To: Amit Kapila Cc: Tom Lane; Alvaro Herrera; Cédric Villemain; Pg Hackers; Robert Haas Subject: Re: [HACKERS] Allow WAL information

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-19 Thread Amit Kapila
I've got a problem with the assumption that, when pg_control is trash, megabytes or gigabytes of WAL can still be relied on completely. I'm almost inclined to suggest that we not get next-LSN from WAL, but by scanning all the pages in the main data store and computing the max observed

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-18 Thread Alvaro Herrera
Excerpts from Tom Lane's message of sáb jun 16 02:41:00 -0400 2012: Amit kapila amit.kap...@huawei.com writes: The suggested patch improves the logic to recover corrupt control file. So that is the reason I felt it will be relevant to do this patch. Well, we invented pg_resetxlog with

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-18 Thread Tom Lane
Alvaro Herrera alvhe...@commandprompt.com writes: Excerpts from Tom Lane's message of sáb jun 16 02:41:00 -0400 2012: Well, we invented pg_resetxlog with the thought that it might be useful for such situations, but I'm not sure offhand that we've ever seen a field report of corrupted

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-18 Thread Amit Kapila
AFAIR you can create pg_control from scratch already with pg_resetxlog. The hard part is coming up with values for the counters, such as the next WAL location. Some of them such as next OID are pretty harmless if you don't guess right, but I'm worried that wrong next WAL could make things

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-18 Thread Tom Lane
Amit Kapila amit.kap...@huawei.com writes: AFAIR you can create pg_control from scratch already with pg_resetxlog. The hard part is coming up with values for the counters, such as the next WAL location. Some of them such as next OID are pretty harmless if you don't guess right, but I'm

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-16 Thread Amit kapila
@postgresql.org Subject: Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata On Thu, Jun 14, 2012 at 11:39 AM, Amit Kapila amit.kap...@huawei.com wrote: I am planning to work on the below Todo list item for this CommitFest Allow WAL information to recover corrupted pg_controldata

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-16 Thread Tom Lane
Amit kapila amit.kap...@huawei.com writes: AFAIR pg_controldata fit on a disk sector so it can not be half written. It can be corrupt due to some other reasons as well like torn disk sector. Torn disk sector? Please, this is nonsense. Disks cannot write half a sector and then stop. A

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-16 Thread Amit kapila
. From: Tom Lane [t...@sss.pgh.pa.us] Sent: Saturday, June 16, 2012 12:11 PM To: Amit kapila Cc: Cédric Villemain; pgsql-hackers@postgresql.org; 'Robert Haas' Subject: Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata Amit kapila amit.kap...@huawei.com

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-15 Thread Cédric Villemain
of the deadline. However I still seek your opinion whether it makes sense to work on this feature. -Original Message- From: Robert Haas [mailto:robertmh...@gmail.com] Sent: Friday, June 15, 2012 12:40 AM To: Amit Kapila Cc: pgsql-hackers@postgresql.org Subject: Re: [HACKERS] Allow WAL

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-14 Thread Robert Haas
On Thu, Jun 14, 2012 at 11:39 AM, Amit Kapila amit.kap...@huawei.com wrote: I am planning to work on the below Todo list item for this CommitFest Allow WAL information to recover corrupted pg_controldata http://archives.postgresql.org/pgsql-patches/2006-06/msg00025.php The deadline for patches

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-14 Thread Amit Kapila
[mailto:robertmh...@gmail.com] Sent: Friday, June 15, 2012 12:40 AM To: Amit Kapila Cc: pgsql-hackers@postgresql.org Subject: Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata On Thu, Jun 14, 2012 at 11:39 AM, Amit Kapila amit.kap...@huawei.com wrote: I am planning to work on the below