Lustre Version: lustre-1.8.6-wc1_2.6.18_238.12.1.el5_lustre.1.8.6.x86_64 OSS Configuration: ------------------ Dual E5620 Processors (8 2.4 GHz cores) 24 GB RAM: 8 OSTs ( two per controller) 4 x Adaptec 51245 2 x RAID-6 LUN's per controller 7200 RPM Hitachi Drives (SATA HUA722020ALA330)) 128 KB Stripe Size 512 KB RPCs
We've tested the configuration extensively and know that we can sustain 2.4 GB/sec to the OSSs for large-block sequential I/O for long periods of time with no issues. The problem comes in production under more typical client work loads where we see far too many of the messages below - even when the load on the servers is not all that great (8 - 10). Five minutes for an IOP to complete seems like a looooong time. Seems like we must be either hitting a bug or running out of some resource (locks?). Iostat tends to show fairly typical service, queue, and wait times which further suggests that there is more going on here than just busy disks. We have about 600 clients with the following settings... lctl set_param ldlm.namespaces.*osc*.lru_size=600 lctl set_param ldlm.namespaces.*mdc*.lru_size=600 max_rpcs_in_flight=32 max_pages_per_rpc=128 Note that we have tuned down the number of ost threads from the default to 96 but it has had little impact. If we are to believe the messages, we should probably reduce the thread count further but it feels like something else is wrong. Perhaps someone else has encountered this or can see an obvious problem in our setup. Any ideas or suggestions are welcome. Charlie Taylor UF HPC Center Dec 26 15:05:10 hpcoss8 kernel: Lustre: Service thread pid 26929 was inactive for 320.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Dec 26 15:05:10 hpcoss8 kernel: Pid: 26929, comm: ll_ost_io_36 Dec 26 15:05:10 hpcoss8 kernel: Dec 26 15:05:10 hpcoss8 kernel: Call Trace: Dec 26 15:05:10 hpcoss8 kernel: [<ffffffff80047144>] try_to_wake_up+0x472/0x484 Dec 26 15:05:10 hpcoss8 kernel: [<ffffffff8008c871>] __wake_up_common+0x3e/0x68 Dec 26 15:05:10 hpcoss8 kernel: [<ffffffff8028882e>] __down_trylock+0x39/0x4e Dec 26 15:05:10 hpcoss8 kernel: [<ffffffff8006472d>] __down_failed_trylock+0x35/0x3a Dec 26 15:05:10 hpcoss8 kernel: [<ffffffff800646b9>] __down_failed+0x35/0x3a Dec 26 15:05:10 hpcoss8 kernel: [<ffffffff88b491e6>] .text.lock.ldlm_resource+0x7d/0x87 [ptlrpc] Dec 26 15:05:10 hpcoss8 kernel: [<ffffffff88b6c337>] ldlm_pools_shrink+0x247/0x2f0 [ptlrpc] Dec 26 15:05:10 hpcoss8 kernel: [<ffffffff80064604>] __down_read+0x12/0x92 Dec 26 15:05:10 hpcoss8 kernel: [<ffffffff8002231e>] __up_read+0x19/0x7f Dec 26 15:05:10 hpcoss8 kernel: [<ffffffff8003f6c0>] shrink_slab+0x60/0x153 Dec 26 15:05:10 hpcoss8 kernel: [<ffffffff800cdd0a>] zone_reclaim+0x235/0x2cd Dec 26 15:05:10 hpcoss8 kernel: [<ffffffff800ca13d>] __rmqueue+0x44/0xc7 Dec 26 15:05:10 hpcoss8 kernel: [<ffffffff8000a919>] get_page_from_freelist+0xbf/0x43a Dec 26 15:05:10 hpcoss8 kernel: [<ffffffff8000f41a>] __alloc_pages+0x78/0x308 Dec 26 15:05:10 hpcoss8 kernel: [<ffffffff80025d41>] find_or_create_page+0x32/0x72 Dec 26 15:05:10 hpcoss8 kernel: [<ffffffff88e694e5>] filter_get_page+0x35/0x70 [obdfilter] Dec 26 15:05:10 hpcoss8 kernel: [<ffffffff88e6b72a>] filter_preprw+0x14da/0x1e00 [obdfilter] Dec 26 15:05:10 hpcoss8 kernel: [<ffffffff88a41a54>] kiblnd_init_tx_msg+0x154/0x1d0 [ko2iblnd] Dec 26 15:05:10 hpcoss8 kernel: [<ffffffff88ad4dc0>] class_handle2object+0xe0/0x170 [obdclass] Dec 26 15:05:11 hpcoss8 kernel: [<ffffffff88a49f2d>] kiblnd_send+0x86d/0x8b0 [ko2iblnd] Dec 26 15:05:11 hpcoss8 kernel: [<ffffffff88e1600c>] ost_brw_write+0xf9c/0x2480 [ost] Dec 26 15:05:11 hpcoss8 kernel: [<ffffffff889fe111>] LNetMDBind+0x301/0x450 [lnet] Dec 26 15:05:11 hpcoss8 kernel: [<ffffffff88b88c65>] lustre_msg_set_limit+0x35/0xf0 [ptlrpc] Dec 26 15:05:11 hpcoss8 kernel: [<ffffffff88b7eac8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Dec 26 15:05:11 hpcoss8 kernel: [<ffffffff88b82fe5>] lustre_msg_get_version+0x35/0xf0 [ptlrpc] Dec 26 15:05:11 hpcoss8 kernel: [<ffffffff88b82ef5>] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Dec 26 15:05:11 hpcoss8 kernel: [<ffffffff88b830a8>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Dec 26 15:05:11 hpcoss8 kernel: [<ffffffff88e1a09e>] ost_handle+0x2bae/0x55b0 [ost] Dec 26 15:05:11 hpcoss8 kernel: [<ffffffff80153e70>] __next_cpu+0x19/0x28 Dec 26 15:05:11 hpcoss8 kernel: [<ffffffff8008dc31>] dequeue_task+0x18/0x37 Dec 26 15:05:11 hpcoss8 kernel: [<ffffffff88b926d9>] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Dec 26 15:05:11 hpcoss8 kernel: [<ffffffff88b92e35>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Dec 26 15:05:11 hpcoss8 kernel: [<ffffffff8008e435>] default_wake_function+0x0/0xe Dec 26 15:05:11 hpcoss8 kernel: [<ffffffff88b93dc6>] ptlrpc_main+0xf66/0x1120 [ptlrpc] Dec 26 15:05:11 hpcoss8 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11 Dec 26 15:05:11 hpcoss8 kernel: [<ffffffff88b92e60>] ptlrpc_main+0x0/0x1120 [ptlrpc] Dec 26 15:05:11 hpcoss8 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11 _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
