[ADMIN] Help! PostgreSQL stuck at starting up after crash

2012-01-19 Thread Samuel Hwang
version Postgresql 9.1.1 on centos5 x64 We experience slow performance and found the server is running 3 vacuum process on the same db which use up 99% of CPU. Then we kill -9 one of those process which cause postgresql to crash and it tried to restart after the crash However when the starting pro

Re: [ADMIN] Interpreting pg_stat_replication values

2012-01-19 Thread Alexander Fortin
On 1/18/12 9:38 AM, Sergey Konoplev wrote: With 9.1 you can use SELECT now() - pg_last_xact_replay_timestamp() AS time_lag; Hi Sergey, thank you very much for your suggestion. In the meanwhile we went for a check on pg_stat_replication, but we'll add one as you suggest soon on the replicas.

Re: [ADMIN] Help! PostgreSQL stuck at starting up after crash

2012-01-19 Thread Samuel Hwang
Thanks, Rick. I think we learned the lesson not to do kill -9. One weird thing was that the three stuck automatically vacuum processes had been running for probably more than a day and they were all working on the same pretty static database. I doubt there were something wrong with our db. Howeve

Re: [ADMIN] Help! PostgreSQL stuck at starting up after crash

2012-01-19 Thread David Hornsby
Sounds like you have a corrupt wal files that you will have to reset the wal logs with pgresetxlog. http://www.postgresql.org/docs/8.2/static/app-pgresetxlog.html This will result in missing transactions so before you do this shutdown postgres and make a copy of the database files first. That way

Re: [ADMIN] Help! PostgreSQL stuck at starting up after crash

2012-01-19 Thread Samuel Hwang
I don't know how to make sure if WAL logs corrupted. At the end of the recovery in postgresql log I saw 2012-01-18 18:30:58.570 MST 3666 - LOG: consistent recovery state reached at 56C/CD0AFE00 2012-01-18 18:30:58.587 MST 3666 - LOG: recovery stopping before abort of transaction 54180204

Re: [ADMIN] Help! PostgreSQL stuck at starting up after crash

2012-01-19 Thread Kevin Grittner
Samuel Hwang wrote: > I don't know how to make sure if WAL logs corrupted. > At the end of the recovery in postgresql log I saw > > 2012-01-18 18:30:58.570 MST 3666 - LOG: consistent recovery > state reached at 56C/CD0AFE00 > 2012-01-18 18:30:58.587 MST 3666 - LOG: recovery stopping >

Re: [ADMIN] Can't get pg_upgrade work for 8.4

2012-01-19 Thread Bruce Momjian
On Wed, Dec 28, 2011 at 03:38:26PM +0100, Alexander Fortin wrote: > Seems that the problem resides in the pg_ctl packaged with Ubuntu > 10.04, the only way we found to fix this was to recompile Postgresql > 8.4.9 from sources using the following patch, than instruct > pg_upgrade (9.1.2) to use the

Re: [ADMIN] Help! PostgreSQL stuck at starting up after crash

2012-01-19 Thread Samuel Hwang
> just nothing happened after that and postgresql is stuck at > starting up and not getting out of archive recovery mode.\ What do you base that on? (copy/paste) ==> I have waited for at least 2 hours, but the database is still in starting up state. In the mean time, there is not cpu/disk activi

Re: [ADMIN] Help! PostgreSQL stuck at starting up after crash

2012-01-19 Thread Samuel Hwang
pg_resetxlog does the trick and db can be started and readable. I am dumping the data out and import to a newly created database cluster. We pretty much lost the data for the last two days, but since our postgresql were running well, it is fewer than it looks. Thanks a lot for the help. On Thu,

Re: [ADMIN] Help! PostgreSQL stuck at starting up after crash

2012-01-19 Thread Samuel Hwang
correct typo. We pretty much lost the data for the last two days, but since our postgresql wereN'T running well, it is fewer than it looks. On Thu, Jan 19, 2012 at 3:45 PM, Samuel Hwang wrote: > pg_resetxlog does the trick and db can be started and readable. > > I am dumping the data out and im