We recently saw some strange behvior in pvfs2-client-core when a server goes away (via segfault) and the client is unable to re-mount the filesystem. The pvfs2-client-core process takes up 100% of a core just spinning on process_vfs_request -> PVFS_sys_testsome and subsequent calls. Full backtrace follows.
In looking at the code in pvfs2-client-core it seems to assume that the re-mount will always succeed (around line 3579). However, I don't know that it's the root cause of the issue. I'll continue looking but wondered if anyone had ideas on this. This appears to be re-creatable by: 1) cleanly mounting and using the filesystem for some I/O 2) either killing the servers or adding iptables rules to the client to reject traffic to the server, 3) Attemping I/O from the client The operation correctly dies with connection refused but the client begins to spin taking up CPU. (gdb) bt #0 0x00511402 in __kernel_vsyscall () #1 0x001d8023 in poll () from /lib/libc.so.6 #2 0x0082ebed in PINT_dev_test_unexpected (incount=5, outcount=0xbf84e4f8, info_array=0x8bbbc0, max_idle_time=10) at src/io/dev/pint-dev.c:398 #3 0x00848f50 in PINT_thread_mgr_dev_push (max_idle_time=10) at src/io/job/thread-mgr.c:332 #4 0x00844caf in do_one_work_cycle_all (idle_time_ms=10) at src/io/job/job.c:5238 #5 0x008454a1 in job_testcontext (out_id_array_p=0xbf8515f0, inout_count_p=0xbf8521f4, returned_user_ptr_array=0xbf851df4, out_status_array_p=0xbf84e5f0, timeout_ms=10, context_id=0) at src/io/job/job.c:4273 #6 0x00857dba in PINT_client_state_machine_testsome (op_id_array=0xbf8522a8, op_count=0xbf8528c4, user_ptr_array=0xbf8527a8, error_code_array=0xbf8526a8, timeout_ms=10) at src/client/sysint/client-state-machine.c:756 #7 0x00857fb9 in PVFS_sys_testsome (op_id_array=0xbf8522a8, op_count=0xbf8528c4, user_ptr_array=0xbf8527a8, error_code_array=0xbf8526a8, timeout_ms=10) at src/client/sysint/client-state-machine.c:971 #8 0x08050cd6 in process_vfs_requests () at src/apps/kernel/linux/pvfs2-client-core.c:3119 #9 0x08052658 in main (argc=10, argv=0xbf852a74) at src/apps/kernel/linux/pvfs2-client-core.c:3579 Let me now if more information is needed, thanks for the input! Michael _______________________________________________ Pvfs2-developers mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
