[HACKERS] Help: 8.0.3 Vacuum of an empty table never completes ...

2005-11-28 Thread James Robinson
G'day folks. We have a production database running 8.0.3 which gets fully pg_dump'd and vacuum analyze'd hourly by cron. Something strange happened to us on the 5AM Friday Nov. 25'th cron run -- the: /usr/local/pgsql/bin/vacuumdb -U postgres --all --analyze --verbose $DATE/vacuum.log

Re: [HACKERS] Help: 8.0.3 Vacuum of an empty table never completes ...

2005-11-28 Thread Tom Lane
James Robinson [EMAIL PROTECTED] writes: Comparing the logs further with when it did complete, it seems that one table in particular (at least) seems afflicted: social=# vacuum verbose analyze agency.swlog_client; hangs up forever -- have to control-c the client. Likewise for w/o

Re: [HACKERS] Help: 8.0.3 Vacuum of an empty table never completes ...

2005-11-28 Thread James Robinson
As fate would have it, the vacuumdb frontend and backend which were initially afflicted are still in existence: sscadmin 19236 19235 0 Nov25 ?00:00:00 /usr/local/pgsql/bin/ vacuumdb -U postgres --all --analyze --verbose postgres 19244 3596 0 Nov25 ?00:00:02 postgres:

Re: [HACKERS] Help: 8.0.3 Vacuum of an empty table never completes ...

2005-11-28 Thread Tom Lane
James Robinson [EMAIL PROTECTED] writes: As fate would have it, the vacuumdb frontend and backend which were initially afflicted are still in existence: OK, so pid 19244 isn't blocked on any lmgr lock; else we'd see an entry with granted = f for it in pg_locks. It could be blocked on a lower

Re: [HACKERS] Help: 8.0.3 Vacuum of an empty table never completes ...

2005-11-28 Thread James Robinson
Here ya go -- BTW -- your guys support is the _best_. But you know that already: [EMAIL PROTECTED]:/home/sscadmin gdb /usr/local/pgsql/bin/postgres 19244 GNU gdb 6.2.1 Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are

Re: [HACKERS] Help: 8.0.3 Vacuum of an empty table never completes ...

2005-11-28 Thread Tom Lane
James Robinson [EMAIL PROTECTED] writes: (gdb) bt #0 0xe410 in ?? () #1 0xbfffd508 in ?? () #2 0x082aef97 in PqSendBuffer () #3 0xbfffd4f0 in ?? () #4 0xb7ec03e1 in send () from /lib/tls/libc.so.6 #5 0x08137d27 in secure_write () #6 0x0813c2a7 in internal_flush () #7

Re: [HACKERS] Help: 8.0.3 Vacuum of an empty table never completes ...

2005-11-28 Thread James Robinson
On Nov 28, 2005, at 11:38 AM, Tom Lane wrote: Can you get a similar backtrace from the vacuumdb process? (Obviously, give gdb the vacuumdb executable not the postgres one.) OK: (gdb) bt #0 0xe410 in ?? () #1 0xbfffe4f8 in ?? () #2 0x0030 in ?? () #3 0x08057b68 in ?? () #4

Re: [HACKERS] Help: 8.0.3 Vacuum of an empty table never completes ...

2005-11-28 Thread Tom Lane
James Robinson [EMAIL PROTECTED] writes: On Nov 28, 2005, at 11:38 AM, Tom Lane wrote: Can you get a similar backtrace from the vacuumdb process? OK: (gdb) bt #0 0xe410 in ?? () #1 0xbfffe4f8 in ?? () #2 0x0030 in ?? () #3 0x08057b68 in ?? () #4 0xb7e98533 in

Re: [HACKERS] Help: 8.0.3 Vacuum of an empty table never completes ...

2005-11-28 Thread James Robinson
On Nov 28, 2005, at 12:00 PM, Tom Lane wrote: Your next move is to look at the state of sshd and whatever is running at the client end of the ssh tunnel. backtrace of the sshd doesn't look good: (gdb) bt #0 0xe410 in ?? () #1 0xbfffdb48 in ?? () #2 0x080a1e28 in ?? () #3 0x080a1e78

Re: [HACKERS] Help: 8.0.3 Vacuum of an empty table never completes ...

2005-11-28 Thread Tom Lane
James Robinson [EMAIL PROTECTED] writes: backtrace of the sshd doesn't look good: Stripped executable :-( ... you won't get much info there. What of the client at the far end of the ssh connection? You should probably assume that the blockage is there, rather than in a commonly used bit of

Re: [HACKERS] Help: 8.0.3 Vacuum of an empty table never completes ...

2005-11-28 Thread Tom Lane
James Robinson [EMAIL PROTECTED] writes: Given the other culprits in play are bash running a straightforward shellscript line with redirected output to a simple file on a non- full filesystem, I'm leaning more towards the odds that something related to the sshd + tcp/ip + ssh client

Re: [HACKERS] Help: 8.0.3 Vacuum of an empty table never completes ...

2005-11-28 Thread James Robinson
On Nov 28, 2005, at 1:46 PM, Tom Lane wrote: James Robinson [EMAIL PROTECTED] writes: backtrace of the sshd doesn't look good: Stripped executable :-( ... you won't get much info there. What of the client at the far end of the ssh connection? You should probably assume that the blockage

Re: [HACKERS] Help: 8.0.3 Vacuum of an empty table never completes ...

2005-11-28 Thread James Robinson
On Nov 28, 2005, at 4:13 PM, Tom Lane wrote: Yeah, could be. Anyway it doesn't seem like we can learn much more today. You might as well just zing the vacuumdb process and let things get back to normal. If it happens again, we'd have reason to dig deeper. Final report [ and apologies to