Hi, my system load shows that quite a number of processes are waiting. ps shows me the same number of processes in state D (uniterruptable sleep). All processes are ll_mdt_NN, where NN is a decimal number.
In the logs I find the entry ( see log below). My questions are: What causes the problem? Can I kill the "hanging" processes? System: Luste 1.8.1 on RHEL5.3 thanks for any hints. --- Oct 5 10:28:03 sosmds2 kernel: Lustre: 0:0:(watchdog.c:181:lcw_cb()) Watchdog triggered for pid 28402: it was inactive for 200.00s Oct 5 10:28:03 sosmds2 kernel: ll_mdt_35 D ffff81000100c980 0 28402 1 28403 28388 (L-TLB) Oct 5 10:28:03 sosmds2 kernel: ffff81041c723810 0000000000000046 0000000000000000 7fffffffffffffff Oct 5 10:28:03 sosmds2 kernel: ffff81041c7237d0 0000000000000001 ffff81022f3e60c0 ffff81022f12e080 Oct 5 10:28:03 sosmds2 kernel: 000177b2feff847c 00000000000014df ffff81022f3e62a8 000000010000028f Oct 5 10:28:03 sosmds2 kernel: Call Trace: Oct 5 10:28:03 sosmds2 kernel: [<ffffffff8008a3ef>] default_wake_function+0x0/0xe Oct 5 10:28:03 sosmds2 kernel: [<ffffffff885b1b26>] :libcfs:lbug_with_loc+0xc6/0xd0 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff885b9c70>] :libcfs:tracefile_init+0x0/0x110 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff88712218>] :ptlrpc:lustre_shrink_reply_v2+0xa8/0x240 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff889ec529>] :mds:mds_getattr_lock+0xc59/0xce0 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff88710ea4>] :ptlrpc:lustre_msg_add_version+0x34/0x110 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff88602923>] :lnet:lnet_ni_send+0x93/0xd0 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff88604d23>] :lnet:lnet_send+0x973/0x9a0 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff889e6fca>] :mds:fixup_handle_for_resent_req+0x5a/0x2c0 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff889f2a76>] :mds:mds_intent_policy+0x636/0xc10 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff886d36f6>] :ptlrpc:ldlm_resource_putref+0x1b6/0x3a0 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff886d0d46>] :ptlrpc:ldlm_lock_enqueue+0x186/0xb30 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff886ecacf>] :ptlrpc:ldlm_export_lock_get+0x6f/0xe0 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff8864fe48>] :obdclass:lustre_hash_add+0x218/0x2e0 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff886f5530>] :ptlrpc:ldlm_server_blocking_ast+0x0/0x83d Oct 5 10:28:03 sosmds2 kernel: [<ffffffff886f3669>] :ptlrpc:ldlm_handle_enqueue+0xc19/0x1210 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff889f0630>] :mds:mds_handle+0x4080/0x4cb0 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff885e0047>] :lvfs:lprocfs_counter_sub+0x57/0x90 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff80148d4f>] __next_cpu+0x19/0x28 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff88715a15>] :ptlrpc:lustre_msg_get_conn_cnt+0x35/0xf0 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff80089d89>] enqueue_task+0x41/0x56 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff8871a72d>] :ptlrpc:ptlrpc_check_req+0x1d/0x110 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff8871ce67>] :ptlrpc:ptlrpc_server_handle_request+0xa97/0x1160 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff8003dc3f>] lock_timer_base+0x1b/0x3c Oct 5 10:28:03 sosmds2 kernel: [<ffffffff80088819>] __wake_up_common+0x3e/0x68 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff88720908>] :ptlrpc:ptlrpc_main+0x1218/0x13e0 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff8008a3ef>] default_wake_function+0x0/0xe Oct 5 10:28:03 sosmds2 kernel: [<ffffffff800b48dd>] audit_syscall_exit+0x327/0x342 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff8871f6f0>] :ptlrpc:ptlrpc_main+0x0/0x13e0 Oct 5 10:28:03 sosmds2 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11 -- Dr. Michael Schwartzkopff MultiNET Services GmbH Addresse: Bretonischer Ring 7; 85630 Grasbrunn; Germany Tel: +49 - 89 - 45 69 11 0 Fax: +49 - 89 - 45 69 11 21 mob: +49 - 174 - 343 28 75 mail: [email protected] web: www.multinet.de Sitz der Gesellschaft: 85630 Grasbrunn Registergericht: Amtsgericht München HRB 114375 Geschäftsführer: Günter Jurgeneit, Hubert Martens --- PGP Fingerprint: F919 3919 FF12 ED5A 2801 DEA6 AA77 57A4 EDD8 979B Skype: misch42 _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
