Hello list,
I've got a web server that is serving php (drupal) sites from various AFS
volumes, and was running fine with an uptime of 160 days until a few days
ago. It has locked up twice in the past two days. After a hard reboot, it
does come back up okay and ran for slightly longer than 24 hours without
incident before hanging again. Although I don't know for sure that AFS is
the problem, I am seeing some indications (references to max execution time
exceeded on php files in /afs/...).
CentOS 5.6
Kernel 2.6.18-238.12.1.el5 i386
OpenAFS RPMs 1.4.14-el5.1.1 installed from
dl.openafs.org/dl/1.4.14/rhel5/i386/
I realize this system is fairly out of date, so I will try to schedule
downtime for an update soon (to CentOS 5.7 with OpenAFS 1.6.0). But until
then, I was hoping to try to interpret what may be happening now.
The symptoms are that the server becomes nearly completely unresponsive (no
reply over port 80 or 22, and the console is completely unresponsive).
However, "cmdebug web1 7001" did return:
Lock afs_xdcache status: (writer_waiting, write_locked(pid:3700 at:617), 1
waiters)
** Cache entry @ 0xea61a4c0 for 2.536870959.1.1 [ss2k.uci.edu]
locks: (none_waiting, 1 read_locks(pid:22600))
2048 bytes DV 57 refcnt 2
callback 00000000 expires 1320206924
0 opens 0 writers
volume root
states (0x4), read-only
Unfortunately, I do not have a process list, so I don't know what pid 3700
is, but most likely it is httpd I assume. Now, trying to determine the file
that this cache entry refers to, I found this page from IBM about using
"calcinode" to translate the volume id and vnode number into an inode
number.
http://www-01.ibm.com/support/docview.wss?uid=swg21112954
$ calcinode 536870959.1
volume ID = 536870959
vnode number = 1
inode number = 3080193
But unfortunately "ls -lRi | grep 3080193" didn't return anything.
Anyone else know how to interpret the output of cmdebug? Or anything else I
should be looking for?
Thanks so much!
--
Jonathan.Nilsson at uci dot edu
Social Sciences Computing Services
SSPB 1265 | 949.824.1536
