Re: [GENERAL] Re: corruption issue after server crash - ERROR: unexpected chunk number 0

2013-11-26 Thread Mike Broers
The restore of a post-crash production backup worked as hoped and the 2nd replication slave is back into its happy hot standby state. So if this problem replicated to our standby servers does that indicate that the potential problematic fsync occurred during a pg_xlog write? Would breaking

Re: [GENERAL] Re: corruption issue after server crash - ERROR: unexpected chunk number 0

2013-11-26 Thread Shaun Thomas
So if this problem replicated to our standby servers does that indicate that the potential problematic fsync occurred during a pg_xlog write? Pretty much. You have a couple issues here, and no easy way to approach them. Primarily, you got data corruption during a sync operation. This means

Re: [GENERAL] Re: corruption issue after server crash - ERROR: unexpected chunk number 0

2013-11-25 Thread Shaun Thomas
Update - I have two hot replication slaves of this db, both have the problem. I took one out of recovery and ran REINDEX table session_session and it fixed the errors about this row. Now Im going to run vacuum and see if there are other tables that complain, but Im guessing if so I will need

Re: [GENERAL] Re: corruption issue after server crash - ERROR: unexpected chunk number 0

2013-11-25 Thread Mike Broers
Thanks Shaun, Im planning to schedule a time to do the vacuum freeze suggested previously. So far the extent of the problem seems limited to the one session table and the one session row that was being used by a heavy bot scan at the time of the crash. Currently Im testing a recovery of a

[GENERAL] Re: corruption issue after server crash - ERROR: unexpected chunk number 0

2013-11-21 Thread Mike Broers
Update - I have two hot replication slaves of this db, both have the problem. I took one out of recovery and ran REINDEX table session_session and it fixed the errors about this row. Now Im going to run vacuum and see if there are other tables that complain, but Im guessing if so I will need to