We have been experiencing a burst of kernel panics recently that have been giving us some trouble. I have worked with some of you off-list, but I wanted to put up a mailing list post for anyone else who might be interested.
There is one particular workload that seems to be good at triggering the problem. During the job the pvfs2-client-core will spin one CPU up to 100% for some variable length of time, and the machine will eventually panic. The panic occurs in hash_compare called by the client-core process, and the call trace looks something like this: pvfs2_devreq_writev [pvfs2] sock_readv_writev [kernel] sock_readv [kernel] do_readv_writev [kernel] sys_writev [kernel] Before the panic, these messages appear in the logs: client2 PVFS2: [E] Got an unrecognized/unimplemented vfs operation of type ff000000. client2 PVFS2: [E] Operation failed: Invalid argument It's been hard to nail down if these messages were logged before or after the processor spikes. Since it does not immediately panic, I tried attaching with gdb but was never able to attach to the process; it always hangs. I also tried enabling some logging via /proc/sys/pvfs2/debug, but I never got any messages there or from dmesg. Phil mentioned that the hash_compare function is triggered indirectly when manipulating the htable_ops_in_progress and that there might be a race where two cores are trying to add or remove something in the table simultaneously. He also mentioned that devpvfs2-req.c does most of that manipulation but is artificially serialized by way of pvfs2-client-core only doing one read or write on the device file at a time. Phil aslo pointed out that there is a kernel function in waitqueue.c, clean_up_interrupted_operation, that could be called and conflict with the devpvfs2-req.c functions because there is no locking on the hash table itself. It gets called in some error handling situations, so it would not occur in the normal path. Michael Moore created a patch for quickhash.h that replaces two qhash_for_each calls with their safe equivalents. My initial test with the patch did not resolve the issue. Per Phil's request I added a log message to clean_up_interrupted_operation in waitqueue.c to let us know when the path was triggered. I don't think the machine that had a panic logged one of these messages, but after checking on these machines recently, I see that several of these messages have been logged. The servers are 32-bit machines running running 2.6 kernels and PVFS 2.6 with a barrage of patches. The clients are running 2.4 kernels with the same PVFS 2.6 code. Once I get my hands on it, I will also run this job against a PVFS 2.8 file system to see if it has any affect on the kernel panics. Bart.
_______________________________________________ Pvfs2-developers mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
