Hi, we just got rpc errors and ptlrpcd stayed at 100% cpu, anyone having an idea what might have caused that or where I should look at?
Thanks, Bernd [ 241.261217] Lustre: Lustre Lite Client File System; [EMAIL PROTECTED] [ 2668.962381] LustreError: 3211:0:(events.c:129:client_bulk_callback()) event type 0, status -5, desc ffff810075c0c000 [ 2668.973017] LustreError: 3212:0:(events.c:53:request_out_callback()) @@@ type 4, status -103 [EMAIL PROTECTED] x187993/t0 o4->[EMAIL PROTECTED]:28 lens 328/288 ref 3 fl Rpc:/0/0 rc 0/0 [ 2668.990284] LustreError: 3209:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 [EMAIL PROTECTED] x187987/t9602764 o4->[EMAIL PROTECTED]:28 lens 328/288 ref 3 fl Bulk:R/4/0 rc 0/0 [ 2669.008051] LustreError: 3211:0:(events.c:129:client_bulk_callback()) event type 0, status -103, desc ffff81006d282000 [ 2669.018824] LustreError: 3212:0:(events.c:53:request_out_callback()) @@@ type 4, status -103 [EMAIL PROTECTED] x187992/t0 o4->[EMAIL PROTECTED]:28 lens 328/288 ref 3 fl Rpc:/0/0 rc 0/0 [ 2669.036120] LustreError: 3208:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 [EMAIL PROTECTED] x187988/t9602766 o4->[EMAIL PROTECTED]:28 lens 328/288 ref 4 fl Bulk:R/4/0 rc 0/0 [ 2669.053955] LustreError: 3211:0:(events.c:129:client_bulk_callback()) event type 0, status -5, desc ffff8100d9fbc000 [ 2669.064591] LustreError: 3210:0:(events.c:129:client_bulk_callback()) event type 0, status -5, desc ffff8100d9f80000 [ 2669.075208] LustreError: 3211:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 [EMAIL PROTECTED] x187989/t9602767 o4->[EMAIL PROTECTED]:28 lens 328/288 ref 4 fl Bulk:R/4/0 rc 0/0 [ 2669.075348] LustreError: 3260:0:(client.c:894:ptlrpc_check_set()) @@@ bulk transfer failed [EMAIL PROTECTED] x187986/t9602763 o4->[EMAIL PROTECTED]:28 lens 328/288 ref 2 fl Bulk:R/4/0 rc 0/0 [ 2669.075355] LustreError: 3260:0:(client.c:895:ptlrpc_check_set()) LBUG [ 2669.075360] Lustre: 3260:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for process 3260 [ 2669.075363] ptlrpcd R running task 0 3260 1 3261 3231 (L-TLB) [ 2669.075369] 0000000000000000 0000000000000000 ffffffff8812ea1e 0000000000000000 [ 2669.075374] ffff810075c0c008 ffff810077cb6488 ffff810077f07dc0 ffffffff881f89c9 [ 2669.075379] ffff810077f07d50 ffffffff8046e2dd ffff8100326be080 ffff81011d498000 [ 2669.075383] Call Trace: [ 2669.075439] [<ffffffff881f89c9>] :ptlrpc:ptlrpc_check_set+0x804/0xa27 [ 2669.075446] [<ffffffff8046e2dd>] _spin_lock_irqsave+0xe/0x15 [ 2669.075488] [<ffffffff8821b1c7>] :ptlrpc:ptlrpcd_check+0x16a/0x273 [ 2669.075493] [<ffffffff8046cb16>] schedule_timeout+0x98/0xb4 [ 2669.075499] [<ffffffff802323e9>] process_timeout+0x0/0xb [ 2669.075536] [<ffffffff8821b5c5>] :ptlrpc:ptlrpcd+0x2f5/0x42b [ 2669.075541] [<ffffffff80234243>] do_notify_parent+0x19a/0x1a7 [ 2669.075547] [<ffffffff8022387f>] default_wake_function+0x0/0xf [ 2669.075585] [<ffffffff881f749a>] :ptlrpc:ptlrpc_expired_set+0x0/0x177 [ 2669.075622] [<ffffffff881f749a>] :ptlrpc:ptlrpc_expired_set+0x0/0x177 [ 2669.075628] [<ffffffff8022387f>] default_wake_function+0x0/0xf [ 2669.075632] [<ffffffff8046e3d0>] _spin_unlock_irq+0x9/0xc [ 2669.075638] [<ffffffff8020a3e8>] child_rip+0xa/0x12 [ 2669.075678] [<ffffffff8821b2d0>] :ptlrpc:ptlrpcd+0x0/0x42b [ 2669.075682] [<ffffffff8020a3de>] child_rip+0x0/0x12 [ 2669.075684] [ 2669.249120] LustreError: 3211:0:(events.c:53:request_out_callback()) Skipped 1 previous similar message [ 2669.262065] LustreError: 3208:0:(events.c:129:client_bulk_callback()) event type 0, status -5, desc ffff8100a46dc000 [ 2669.269022] LustreError: 14:0:(linux-debug.c:91:libcfs_run_upcall()) Error -2 invoking LNET upcall /usr/lib/lustre/lnet_upcall ROUTER_NOTIFY,[EMAIL PROTECTED],down,1175601855; check /proc/sys/lnet/upcall [ 2669.290995] LustreError: 3211:0:(events.c:53:request_out_callback()) @@@ type 4, status -5 [EMAIL PROTECTED] x187990/t0 o4->[EMAIL PROTECTED]:28 lens 328/288 ref 3 fl Rpc:/0/0 rc 0/0 [ 2669.308211] LustreError: dumping log to /tmp/lustre-log.1175601855.3260 [ 2669.314858] LustreError: 3260:0:(linux-debug.c:91:libcfs_run_upcall()) Error -2 invoking LNET upcall /usr/lib/lustre/lnet_upcall LBUG,/data/home/bschubert/tmpa/Lustre/lustre-1.4.9/lustre-1.4.9/lustre/ptlrpc/client.c,ptlrpc_check_set,895; check /proc/sys/lnet/upcall -- Bernd Schubert Q-Leap Networks GmbH _______________________________________________ Lustre-discuss mailing list [email protected] https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
