On Mon, Feb 25, 2008 at 9:53 AM, Aaron Bono <[EMAIL PROTECTED]> wrote:
> On Mon, Feb 25, 2008 at 9:42 AM, Simon Riggs <[EMAIL PROTECTED]> > wrote: > > > On Mon, 2008-02-25 at 08:11 -0600, Aaron Bono wrote: > > > Now I am getting: <2008-02-25 10:51:08 CST username 127.0.0.1>LOG: unexpected EOF on client connection <2008-02-25 10:51:08 CST username 127.0.0.1>LOG: disconnection: session time: 3:17:39.14 user=username database=my_db host=127.0.0.1 port=12345 ... <2008-02-25 10:51:27 CST [unknown] >LOG: connection received: host= 127.0.0.1 port=43222 <2008-02-25 10:51:27 CST username 127.0.0.1>LOG: connection authorized: user=username database=my_db <2008-02-25 10:52:27 CST >LOG: autovacuum: processing database "postgres" <2008-02-25 10:53:18 CST [unknown] >LOG: connection received: host= 127.0.0.1 port=43284 <2008-02-25 10:53:18 CST username 127.0.0.1>LOG: connection authorized: user=username database=my_db <2008-02-25 10:53:22 CST >LOG: server process (PID 17043) was terminated by signal 11 <2008-02-25 10:53:22 CST >LOG: terminating any other active server processes <2008-02-25 10:53:22 CST username 127.0.0.1>WARNING: terminating connection because of crash of another server process <2008-02-25 10:53:22 CST username 127.0.0.1>DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. <2008-02-25 10:53:22 CST username 127.0.0.1>HINT: In a moment you should be able to reconnect to the database and repeat your command. <2008-02-25 10:53:22 CST username 127.0.0.1>CONTEXT: PL/pgSQL function "get_branch_for_zip" line 61 at fetch <2008-02-25 10:53:22 CST username 127.0.0.1>WARNING: terminating connection because of crash of another server process <2008-02-25 10:53:22 CST username 127.0.0.1>DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. <2008-02-25 10:53:22 CST username 127.0.0.1>HINT: In a moment you should be able to reconnect to the database and repeat your command. <2008-02-25 10:53:22 CST username 127.0.0.1>WARNING: terminating connection because of crash of another server process <2008-02-25 10:53:22 CST username 127.0.0.1>DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. <2008-02-25 10:53:22 CST username 127.0.0.1>HINT: In a moment you should be able to reconnect to the database and repeat your command. <2008-02-25 10:53:22 CST username 127.0.0.1>WARNING: terminating connection because of crash of another server process <2008-02-25 10:53:22 CST username 127.0.0.1>DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. <2008-02-25 10:53:22 CST username 127.0.0.1>HINT: In a moment you should be able to reconnect to the database and repeat your command. <2008-02-25 10:53:22 CST username 127.0.0.1>WARNING: terminating connection because of crash of another server process <2008-02-25 10:53:22 CST username 127.0.0.1>DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. <2008-02-25 10:53:22 CST username 127.0.0.1>HINT: In a moment you should be able to reconnect to the database and repeat your command. <2008-02-25 10:53:22 CST >LOG: all server processes terminated; reinitializing <2008-02-25 10:53:22 CST >LOG: database system was interrupted at 2008-02-25 10:50:59 CST <2008-02-25 10:53:22 CST >LOG: checkpoint record is at 14/9F8126D8 <2008-02-25 10:53:22 CST >LOG: redo record is at 14/9F8126D8; undo record is at 0/0; shutdown FALSE <2008-02-25 10:53:22 CST >LOG: next transaction ID: 911531009; next OID: 90141237 <2008-02-25 10:53:22 CST >LOG: next MultiXactId: 27095; next MultiXactOffset: 66130 <2008-02-25 10:53:22 CST >LOG: database system was not properly shut down; automatic recovery in progress <2008-02-25 10:53:22 CST >LOG: redo starts at 14/9F81271C <2008-02-25 10:53:22 CST >LOG: record with zero length at 14/9F826D68 <2008-02-25 10:53:22 CST >LOG: redo done at 14/9F826D40 <2008-02-25 10:53:22 CST >LOG: could not truncate directory "pg_multixact/members": apparent wraparound <2008-02-25 10:53:22 CST >LOG: database system is ready <2008-02-25 10:53:22 CST >LOG: transaction ID wrap limit is 1576224515, limited by database "another_db" <2008-02-25 10:53:23 CST [unknown] >LOG: connection received: host= 127.0.0.1 port=43289 <2008-02-25 10:53:23 CST username 127.0.0.1>LOG: connection authorized: user=username database=my_db > > > > Our database was going along just fine this morning when, out of the > > > blue, we started getting the following errors: > > > > > > FATAL: the database system is in recovery mode > > > > > > We are using PostgreSQL 8.1.3 on CentOS. I was the only person on the > > > machine and had recently started a small batch that had just finished > > > querying and updating the database. This batch runs constantly and > > > has not been changed in months. > > > > Your database has crashed and is recovering. You should look at the > > server log to find out when and why this happened, then either correct > > the problem or report the bug. <http://www.2ndQuadrant.com> > > > > > Here is what I am seeing: > > <2008-02-25 07:15:24 CST username 127.0.0.1>ERROR: unrecognized node > type: 4260493 > <2008-02-25 07:15:25 CST [unknown] >LOG: connection received: host= > 127.0.0.1 port=51295 > <2008-02-25 07:15:25 CST username 127.0.0.1>LOG: connection authorized: > user=username database=my_db > <2008-02-25 07:15:25 CST username 127.0.0.1>LOG: disconnection: session > time: 0:00:00.04 user=username database=my_db host=127.0.0.1 port=51295 > <2008-02-25 07:15:25 CST [unknown] >LOG: connection received: host= > 127.0.0.1 port=51296 > <2008-02-25 07:15:25 CST username 127.0.0.1>LOG: connection authorized: > user=username database=clockwork_db_prod > <2008-02-25 07:15:38 CST username 127.0.0.1>ERROR: unrecognized node > type: 4260493 > <2008-02-25 07:15:49 CST >LOG: server process (PID 10876) was terminated > by signal 11 > <2008-02-25 07:15:49 CST >LOG: terminating any other active server > processes > <2008-02-25 07:15:49 CST username 127.0.0.1>WARNING: terminating > connection because of crash of another server process > <2008-02-25 07:15:49 CST username 127.0.0.1>DETAIL: The postmaster has > commanded this server process to roll back the current transaction and exit, > because another server process exited abnormally and possibly corrupted > shared memory. > <2008-02-25 07:15:49 CST username 127.0.0.1>HINT: In a moment you should > be able to reconnect to the database and repeat your command. > <2008-02-25 07:15:49 CST username 127.0.0.1>WARNING: terminating > connection because of crash of another server process > > and it continues like this as it terminated all connections followed by: > > <2008-02-25 07:15:49 CST username 127.0.0.1>CONTEXT: PL/pgSQL function > "get_branch_for_zip" line 61 at fetch > <2008-02-25 07:15:50 CST >LOG: all server processes terminated; > reinitializing > <2008-02-25 07:15:50 CST >LOG: database system was interrupted at > 2008-02-25 07:11:10 CST > <2008-02-25 07:15:50 CST >LOG: checkpoint record is at 14/9F015F54 > <2008-02-25 07:15:50 CST >LOG: redo record is at 14/9F015F54; undo > record is at 0/0; shutdown FALSE > <2008-02-25 07:15:50 CST >LOG: next transaction ID: 911288200; next OID: > 90133045 > <2008-02-25 07:15:50 CST >LOG: next MultiXactId: 27095; next > MultiXactOffset: 66130 > <2008-02-25 07:15:50 CST >LOG: database system was not properly shut > down; automatic recovery in progress > <2008-02-25 07:15:50 CST >LOG: redo starts at 14/9F015F98 > <2008-02-25 07:15:50 CST >LOG: record with zero length at 14/9F08C23C > <2008-02-25 07:15:50 CST >LOG: redo done at 14/9F08C214 > <2008-02-25 07:15:50 CST >LOG: could not truncate directory > "pg_multixact/members": apparent wraparound > <2008-02-25 07:15:50 CST >LOG: database system is ready > <2008-02-25 07:15:50 CST >LOG: transaction ID wrap limit is 1576224515, > limited by database "another_db" > <2008-02-25 07:15:56 CST [unknown] >LOG: connection received: host= > 127.0.0.1 port=51987 > <2008-02-25 07:15:56 CST username 127.0.0.1>LOG: connection authorized: > user=onehour database=my_db > <2008-02-25 07:15:56 CST >LOG: server process (PID 14433) was terminated > by signal 11 > <2008-02-25 07:15:56 CST >LOG: terminating any other active server > processes > <2008-02-25 07:15:56 CST >LOG: all server processes terminated; > reinitializing > <2008-02-25 07:15:56 CST >LOG: database system was interrupted at > 2008-02-25 07:15:50 CST > <2008-02-25 07:15:56 CST >LOG: checkpoint record is at 14/9F08C23C > <2008-02-25 07:15:56 CST >LOG: redo record is at 14/9F08C23C; undo > record is at 0/0; shutdown TRUE > <2008-02-25 07:15:56 CST >LOG: next transaction ID: 911293548; next OID: > 90133045 > <2008-02-25 07:15:56 CST >LOG: next MultiXactId: 27095; next > MultiXactOffset: 66130 > <2008-02-25 07:15:56 CST >LOG: database system was not properly shut > down; automatic recovery in progress > <2008-02-25 07:15:56 CST >LOG: record with zero length at 14/9F08C280 > <2008-02-25 07:15:56 CST >LOG: redo is not required > <2008-02-25 07:15:56 CST >LOG: could not truncate directory > "pg_multixact/members": apparent wraparound > <2008-02-25 07:15:56 CST >LOG: database system is ready > <2008-02-25 07:15:56 CST >LOG: transaction ID wrap limit is 1576224515, > limited by database "another_db" > > and it then continues with this. > > > -- ================================================================== Aaron Bono Aranya Software Technologies, Inc. http://www.aranya.com http://codeelixir.com ==================================================================
