Hi Randall, Any messages in the client log? These operations are waiting for the client daemon, which isn't responding. Do you see a pvfs2-client-core process on that node?
-sam On Dec 18, 2009, at 12:30 PM, Randall Martin wrote: > I have been seeing a lot of process hangs/loops on our cluster. The > processes end up in the zombie state and are not killable, yet they are > consuming all of the cpu cycles on the executing core. These processes all > seem to be hung up in the kernel with pvfs calls. Is there a way to > gracefully get out of this situation? Thus far the only solution has been to > reboot. I wish there was a way to signal the kernel module to stop whatever > it is it’s doing in this loop. > > > >> Dec 10 10:01:06 node0803 kernel: SysRq : Show CPUs > >> Dec 10 10:01:06 node0803 kernel: CPU3: > >> Dec 10 10:01:06 node0803 kernel: ffff81010efbff48 0000000000000000 > >> ffff8103746d5b40 ffffffff801aee62 > >> Dec 10 10:01:06 node0803 kernel: 0000000000000000 0000000000000000 > >> 0000000000000000 ffffffff801aee91 > >> Dec 10 10:01:06 node0803 kernel: ffffffff8005e2fc ffffffff80022da9 > >> ffff8103746d5c08 ffff81010ef90000 > >> Dec 10 10:01:06 node0803 kernel: Call Trace: > >> Dec 10 10:01:06 node0803 kernel: <IRQ> [<ffffffff801aee62>] > >> showacpu+0x0/0x3b > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff801aee91>] showacpu+0x2f/0x3b > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8005e2fc>] > >> call_softirq+0x1c/0x28 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff80022da9>] > >> smp_call_function_interrupt+0x57/0x75 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8005dc22>] > >> call_function_interrupt+0x66/0x6c > >> Dec 10 10:01:06 node0803 kernel: <EOI> [<ffffffff80064b20>] > >> _spin_unlock_irqrestore+0x8/0x9 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8001ca65>] > >> __mod_timer+0xb0/0xbe > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff80063876>] > >> schedule_timeout+0x85/0xad > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff80096fee>] > >> process_timeout+0x0/0x5 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885b5b54>] > >> :pvfs2:wait_for_cancellation_downcall+0xad/0x113 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8008c3bd>] > >> default_wake_function+0x0/0xe > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885b600f>] > >> :pvfs2:service_operation+0x1e0/0x2a6 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885a92ed>] > >> :pvfs2:pvfs2_cancel_op_in_progress+0x6b/0x97 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885ad58f>] > >> :pvfs2:wait_for_direct_io+0x1c7/0x3de > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885aebf8>] > >> :pvfs2:do_readv_writev+0x5c5/0x6c3 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885affcf>] > >> :pvfs2:pvfs2_file_read+0xc9/0xd8 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8000b695>] vfs_read+0xcb/0x171 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff80011b72>] sys_read+0x45/0x6e > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8005d116>] > >> system_call+0x7e/0x83 > >> Dec 10 10:01:06 node0803 kernel: > >> Dec 10 10:01:06 node0803 kernel: CPU5: > >> Dec 10 10:01:06 node0803 kernel: ffff81043fc47f48 0000000000000000 > >> ffff8103bb96dbd0 ffffffff801aee62 > >> Dec 10 10:01:06 node0803 kernel: 0000000000000000 0000000000000000 > >> 0000000000000000 ffffffff801aee91 > >> Dec 10 10:01:06 node0803 kernel: ffffffff8005e2fc ffffffff80022da9 > >> ffff8101a1ee2ab8 0000000000000004 > >> Dec 10 10:01:06 node0803 kernel: Call Trace: > >> Dec 10 10:01:06 node0803 kernel: <IRQ> [<ffffffff801aee62>] > >> showacpu+0x0/0x3b > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff801aee91>] showacpu+0x2f/0x3b > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8005e2fc>] > >> call_softirq+0x1c/0x28 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff80022da9>] > >> smp_call_function_interrupt+0x57/0x75 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8005dc22>] > >> call_function_interrupt+0x66/0x6c > >> Dec 10 10:01:06 node0803 kernel: <EOI> [<ffffffff883fb0b7>] > >> :ipmi_si:port_inb+0x0/0xf > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff80064a58>] _spin_lock+0x3/0xa > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885b5b16>] > >> :pvfs2:wait_for_cancellation_downcall+0x6f/0x113 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8008c3bd>] > >> default_wake_function+0x0/0xe > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885b600f>] > >> :pvfs2:service_operation+0x1e0/0x2a6 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885a92ed>] > >> :pvfs2:pvfs2_cancel_op_in_progress+0x6b/0x97 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885ad58f>] > >> :pvfs2:wait_for_direct_io+0x1c7/0x3de > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885aebf8>] > >> :pvfs2:do_readv_writev+0x5c5/0x6c3 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885affcf>] > >> :pvfs2:pvfs2_file_read+0xc9/0xd8 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8000b695>] vfs_read+0xcb/0x171 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff80011b72>] sys_read+0x45/0x6e > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8005d116>] > >> system_call+0x7e/0x83 > >> Dec 10 10:01:06 node0803 kernel: > >> Dec 10 10:01:06 node0803 kernel: CPU1: > >> Dec 10 10:01:06 node0803 kernel: ffff8101e71d5e58 0000000000000000 > >> 0000000000000000 0000000000000000 > >> Dec 10 10:01:06 node0803 kernel: ffffffff801aee62 0000000000000000 > >> 0000000000000006 ffffffff801aee91 > >> Dec 10 10:01:06 node0803 kernel: 0000000000000000 ffffffff80093aea > >> ffffffff8032ddc0 0000000000000077 > >> Dec 10 10:01:06 node0803 kernel: Call Trace: > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff801aee62>] showacpu+0x0/0x3b > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff801aee91>] showacpu+0x2f/0x3b > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff80093aea>] > >> on_each_cpu+0x19/0x22 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff801aed97>] > >> __handle_sysrq+0x90/0x121 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff80107395>] > >> write_sysrq_trigger+0x2a/0x32 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8001691b>] > >> vfs_write+0xce/0x174 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff800171d3>] sys_write+0x45/0x6e > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8005d116>] > >> system_call+0x7e/0x83 > >> Dec 10 10:01:06 node0803 kernel: > >> Dec 10 10:01:06 node0803 kernel: CPU7: > >> Dec 10 10:01:06 node0803 kernel: ffff81043fccff48 0000000000000000 > >> ffff81020dd19b80 ffffffff801aee62 > >> Dec 10 10:01:06 node0803 kernel: 0000000000000000 0000000000000000 > >> 0000000000000000 ffffffff801aee91 > >> Dec 10 10:01:06 node0803 kernel: ffffffff8005e2fc ffffffff80022da9 > >> 0000000199986190 0000000199986190 > >> Dec 10 10:01:06 node0803 kernel: Call Trace: > >> Dec 10 10:01:06 node0803 kernel: <IRQ> [<ffffffff801aee62>] > >> showacpu+0x0/0x3b > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff801aee91>] showacpu+0x2f/0x3b > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8005e2fc>] > >> call_softirq+0x1c/0x28 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff80022da9>] > >> smp_call_function_interrupt+0x57/0x75 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8005dc22>] > >> call_function_interrupt+0x66/0x6c > >> Dec 10 10:01:06 node0803 kernel: <EOI> [<ffffffff8005bae0>] > >> del_timer_sync+0x0/0x16 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff80063883>] > >> schedule_timeout+0x92/0xad > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff80096fee>] > >> process_timeout+0x0/0x5 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885b5b54>] > >> :pvfs2:wait_for_cancellation_downcall+0xad/0x113 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8008c3bd>] > >> default_wake_function+0x0/0xe > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885b600f>] > >> :pvfs2:service_operation+0x1e0/0x2a6 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885a92ed>] > >> :pvfs2:pvfs2_cancel_op_in_progress+0x6b/0x97 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885ad58f>] > >> :pvfs2:wait_for_direct_io+0x1c7/0x3de > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885aebf8>] > >> :pvfs2:do_readv_writev+0x5c5/0x6c3 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885affcf>] > >> :pvfs2:pvfs2_file_read+0xc9/0xd8 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8000b695>] vfs_read+0xcb/0x171 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff80011b72>] sys_read+0x45/0x6e > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8005d116>] > >> system_call+0x7e/0x83 > >> Dec 10 10:01:06 node0803 kernel: > >> Dec 10 10:01:06 node0803 kernel: CPU2: > >> Dec 10 10:01:06 node0803 kernel: ffff81010ef6bf48 0000000000000000 > >> ffff81037fad3b40 ffffffff801aee62 > >> Dec 10 10:01:06 node0803 kernel: 0000000000000000 0000000000000000 > >> 0000000000000000 ffffffff801aee91 > >> Dec 10 10:01:06 node0803 kernel: ffffffff8005e2fc ffffffff80022da9 > >> ffff81037fad3c08 0000000199986190 > >> Dec 10 10:01:06 node0803 kernel: Call Trace: > >> Dec 10 10:01:06 node0803 kernel: <IRQ> [<ffffffff801aee62>] > >> showacpu+0x0/0x3b > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff801aee91>] showacpu+0x2f/0x3b > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8005e2fc>] > >> call_softirq+0x1c/0x28 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff80022da9>] > >> smp_call_function_interrupt+0x57/0x75 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8005dc22>] > >> call_function_interrupt+0x66/0x6c > >> Dec 10 10:01:06 node0803 kernel: <EOI> [<ffffffff80064b20>] > >> _spin_unlock_irqrestore+0x8/0x9 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8004b256>] > >> try_to_del_timer_sync+0x51/0x5a > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8005baec>] > >> del_timer_sync+0xc/0x16 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff80063883>] > >> schedule_timeout+0x92/0xad > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff80096fee>] > >> process_timeout+0x0/0x5 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885b5b54>] > >> :pvfs2:wait_for_cancellation_downcall+0xad/0x113 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8008c3bd>] > >> default_wake_function+0x0/0xe > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885b600f>] > >> :pvfs2:service_operation+0x1e0/0x2a6 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885a92ed>] > >> :pvfs2:pvfs2_cancel_op_in_progress+0x6b/0x97 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885ad58f>] > >> :pvfs2:wait_for_direct_io+0x1c7/0x3de > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885aebf8>] > >> :pvfs2:do_readv_writev+0x5c5/0x6c3 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8003bc80>] > >> remove_wait_queue+0x1c/0x2c > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885b5d7c>] > >> :pvfs2:wait_for_matching_downcall+0x1c2/0x1d5 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff885affcf>] > >> :pvfs2:pvfs2_file_read+0xc9/0xd8 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8000b695>] vfs_read+0xcb/0x171 > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff80011b72>] sys_read+0x45/0x6e > >> Dec 10 10:01:06 node0803 kernel: [<ffffffff8005d116>] > >> system_call+0x7e/0x83 > >> Dec 10 10:01:06 node0803 kernel: > >> Dec 10 10:01:06 node0803 kernel: CPU4: > >> Dec 10 10:01:06 node0803 kernel: ffff81010eff3f48 0000000000000000 > >> ffff8103ee517b40 ffffffff801aee62 > >> Dec 10 10:01:06 node0803 kernel: 0000000000000000 0000000000000000 > >> 0000000000000000 ffffffff801aee91 > >> Dec 10 10:01:08 node0803 kernel: ffffffff8005e2fc ffffffff80022da9 > >> ffff8103ee517c08 0000000199986190 > >> Dec 10 10:01:08 node0803 kernel: Call Trace: > >> Dec 10 10:01:08 node0803 kernel: <IRQ> [<ffffffff801aee62>] > >> showacpu+0x0/0x3b > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff801aee91>] showacpu+0x2f/0x3b > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8005e2fc>] > >> call_softirq+0x1c/0x28 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff80022da9>] > >> smp_call_function_interrupt+0x57/0x75 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8005dc22>] > >> call_function_interrupt+0x66/0x6c > >> Dec 10 10:01:08 node0803 kernel: <EOI> [<ffffffff8005d116>] > >> system_call+0x7e/0x83 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff80064b20>] > >> _spin_unlock_irqrestore+0x8/0x9 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8004b256>] > >> try_to_del_timer_sync+0x51/0x5a > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8005baec>] > >> del_timer_sync+0xc/0x16 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff80063883>] > >> schedule_timeout+0x92/0xad > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff80096fee>] > >> process_timeout+0x0/0x5 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff885b5b54>] > >> :pvfs2:wait_for_cancellation_downcall+0xad/0x113 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8008c3bd>] > >> default_wake_function+0x0/0xe > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff885b600f>] > >> :pvfs2:service_operation+0x1e0/0x2a6 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff885a92ed>] > >> :pvfs2:pvfs2_cancel_op_in_progress+0x6b/0x97 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff885ad58f>] > >> :pvfs2:wait_for_direct_io+0x1c7/0x3de > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff885aebf8>] > >> :pvfs2:do_readv_writev+0x5c5/0x6c3 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8003bc80>] > >> remove_wait_queue+0x1c/0x2c > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff885b5d7c>] > >> :pvfs2:wait_for_matching_downcall+0x1c2/0x1d5 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff885affcf>] > >> :pvfs2:pvfs2_file_read+0xc9/0xd8 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8000b695>] vfs_read+0xcb/0x171 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff80011b72>] sys_read+0x45/0x6e > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8005d116>] > >> system_call+0x7e/0x83 > >> Dec 10 10:01:08 node0803 kernel: > >> Dec 10 10:01:08 node0803 kernel: CPU0: > >> Dec 10 10:01:08 node0803 kernel: ffffffff8043df48 0000000000000000 > >> ffff810253083ab0 ffffffff801aee62 > >> Dec 10 10:01:08 node0803 kernel: 0000000000000000 0000000000000000 > >> ffff810001004420 ffffffff801aee91 > >> Dec 10 10:01:08 node0803 kernel: ffffffff8005e2fc ffffffff80022da9 > >> ffff8103c97f37e0 ffff8103c97f3818 > >> Dec 10 10:01:08 node0803 kernel: Call Trace: > >> Dec 10 10:01:08 node0803 kernel: <IRQ> [<ffffffff801aee62>] > >> showacpu+0x0/0x3b > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff801aee91>] showacpu+0x2f/0x3b > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8005e2fc>] > >> call_softirq+0x1c/0x28 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff80022da9>] > >> smp_call_function_interrupt+0x57/0x75 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8005dc22>] > >> call_function_interrupt+0x66/0x6c > >> Dec 10 10:01:08 node0803 kernel: <EOI> [<ffffffff80063019>] > >> thread_return+0xb3/0xfe > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff80062486>] > >> __sched_text_start+0xf6/0xbd6 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8003da82>] > >> lock_timer_base+0x1b/0x3c > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8001ca65>] > >> __mod_timer+0xb0/0xbe > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8006387b>] > >> schedule_timeout+0x8a/0xad > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff80096fee>] > >> process_timeout+0x0/0x5 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff885b5b54>] > >> :pvfs2:wait_for_cancellation_downcall+0xad/0x113 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8008c3bd>] > >> default_wake_function+0x0/0xe > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff885b600f>] > >> :pvfs2:service_operation+0x1e0/0x2a6 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff885a92ed>] > >> :pvfs2:pvfs2_cancel_op_in_progress+0x6b/0x97 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff885ad58f>] > >> :pvfs2:wait_for_direct_io+0x1c7/0x3de > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff885aebf8>] > >> :pvfs2:do_readv_writev+0x5c5/0x6c3 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff885affcf>] > >> :pvfs2:pvfs2_file_read+0xc9/0xd8 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8000b695>] vfs_read+0xcb/0x171 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff80011b72>] sys_read+0x45/0x6e > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8005d116>] > >> system_call+0x7e/0x83 > >> Dec 10 10:01:08 node0803 kernel: > >> Dec 10 10:01:08 node0803 kernel: CPU6: > >> Dec 10 10:01:08 node0803 kernel: ffff81043fc7bf48 0000000000000000 > >> ffff8103e36c9ab0 ffffffff801aee62 > >> Dec 10 10:01:08 node0803 kernel: 0000000000000000 0000000000000000 > >> ffff810001036b20 ffffffff801aee91 > >> Dec 10 10:01:08 node0803 kernel: ffffffff8005e2fc ffffffff80022da9 > >> ffff8102dea33100 ffff8102dea33138 > >> Dec 10 10:01:08 node0803 kernel: Call Trace: > >> Dec 10 10:01:08 node0803 kernel: <IRQ> [<ffffffff801aee62>] > >> showacpu+0x0/0x3b > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff801aee91>] showacpu+0x2f/0x3b > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8005e2fc>] > >> call_softirq+0x1c/0x28 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff80022da9>] > >> smp_call_function_interrupt+0x57/0x75 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8005dc22>] > >> call_function_interrupt+0x66/0x6c > >> Dec 10 10:01:08 node0803 kernel: <EOI> [<ffffffff80013a31>] > >> tcp_poll+0x0/0x12f > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff80063019>] > >> thread_return+0xb3/0xfe > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff80062486>] > >> __sched_text_start+0xf6/0xbd6 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8003da82>] > >> lock_timer_base+0x1b/0x3c > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8001ca65>] > >> __mod_timer+0xb0/0xbe > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8006387b>] > >> schedule_timeout+0x8a/0xad > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff80096fee>] > >> process_timeout+0x0/0x5 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff885b5b54>] > >> :pvfs2:wait_for_cancellation_downcall+0xad/0x113 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8008c3bd>] > >> default_wake_function+0x0/0xe > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff885b600f>] > >> :pvfs2:service_operation+0x1e0/0x2a6 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff885a92ed>] > >> :pvfs2:pvfs2_cancel_op_in_progress+0x6b/0x97 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff885ad58f>] > >> :pvfs2:wait_for_direct_io+0x1c7/0x3de > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff885aebf8>] > >> :pvfs2:do_readv_writev+0x5c5/0x6c3 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8002ade0>] > >> get_signal_to_deliver+0x19b/0x45a > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff885affcf>] > >> :pvfs2:pvfs2_file_read+0xc9/0xd8 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8000b695>] vfs_read+0xcb/0x171 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff80011b72>] sys_read+0x45/0x6e > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8005d19f>] > >> sysret_signal+0x1c/0x27 > >> Dec 10 10:01:08 node0803 kernel: [<ffffffff8005d116>] > >> system_call+0x7e/0x83 > >> > > -Randy > _______________________________________________ > Pvfs2-developers mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________ Pvfs2-developers mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
