Dear All,
Our small but hardly used cluster was running for weeks without any problems, but yesterday 3 of 8 nodes froze up in 1 hour. This is from messages log, there was many of this in it before this last one: May 17 22:00:31 node8 kernel: LustreError: dumping log to /tmp/lustre-log.1211054431.7807 May 17 22:00:33 node8 kernel: LustreError: 7807:0:(service.c:668:ptlrpc_server_handle_request()) request 3062525 opc 4 from [EMAIL PROTECTED] processed in 101s trans 2522291 rc 0/0 May 17 22:00:33 node8 kernel: Lustre: 7807:0:(watchdog.c:312:lcw_update_time()) Expired watchdog for pid 7807 disabled after 101.5718s May 17 22:00:43 node8 kernel: Lustre: 7641:0:(ldlm_lib.c:519:target_handle_reconnect()) cubefs-OST0008: 9737b5de-8c67-99be-94ae-12f66fe80edf reconnecting May 17 22:00:43 node8 kernel: Lustre: 7641:0:(ldlm_lib.c:519:target_handle_reconnect()) Skipped 1 previous similar message May 17 22:01:05 node8 kernel: Lustre: 7627:0:(ldlm_lib.c:747:target_handle_connect()) cubefs-OST0008: refuse reconnection from 1a535b99-47d7-7f5f-f3cb-277793 [EMAIL PROTECTED]@tcp to 0xffff810010759000; still busy with 2 active RPCs May 17 22:01:05 node8 kernel: LustreError: 7627:0:(ldlm_lib.c:1442:target_send_reply_msg()) @@@ processing error (-16) [EMAIL PROTECTED] x2179620/t0 o8->[EMAIL PROTECTED]:-1 lens 304/200 ref 0 fl Interpret:/0/0 rc -16/0 May 17 22:01:05 node8 kernel: Lustre: 0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 7738: it was inactive for 100s May 17 22:01:05 node8 kernel: Lustre: 0:0:(watchdog.c:130:lcw_cb()) Skipped 1 previous similar message May 17 22:01:05 node8 kernel: Lustre: 0:0:(linux-debug.c:168:libcfs_debug_dumpstack()) showing stack for process 7738 May 17 22:01:05 node8 kernel: Lustre: 0:0:(linux-debug.c:168:libcfs_debug_dumpstack()) Skipped 1 previous similar message May 17 22:01:05 node8 kernel: ll_ost_io_13 D 000000000000712e 0 7738 1 7739 7737 (L-TLB) May 17 22:01:05 node8 kernel: ffff8100655273f0 0000000000000046 ffff810003d18c58 ffffffff80012c28 May 17 22:01:05 node8 kernel: 000000000000000a ffff810065511820 ffff810063e1f0c0 0007d730dc4a35a5 May 17 22:01:05 node8 kernel: 0000000000000213 ffff810065511a08 ffff810000000001 ffff810063c0a7e0 May 17 22:01:05 node8 kernel: Call Trace: May 17 22:01:05 node8 kernel: [<ffffffff80012c28>] get_request+0x1a6/0x36b May 17 22:01:05 node8 kernel: [<ffffffff883354ac>] :drbd:lc_find+0x30/0x51 May 17 22:01:05 node8 kernel: [<ffffffff88334e23>] :drbd:drbd_al_begin_io+0x160/0x24c May 17 22:01:05 node8 kernel: [<ffffffff80022d47>] mempool_alloc+0x24/0xda May 17 22:01:05 node8 kernel: [<ffffffff8009b3d0>] autoremove_wake_function+0x0/0x2e May 17 22:01:05 node8 kernel: [<ffffffff800d7ad5>] __bio_clone+0x71/0x8a May 17 22:01:05 node8 kernel: [<ffffffff883320d9>] :drbd:drbd_make_request_common+0x593/0x8c5 May 17 22:01:05 node8 kernel: [<ffffffff8001b998>] generic_make_request+0x204/0x21b May 17 22:01:05 node8 kernel: [<ffffffff80022d47>] mempool_alloc+0x24/0xda May 17 22:01:05 node8 kernel: [<ffffffff80032b85>] submit_bio+0xcd/0xd4 May 17 22:01:05 node8 kernel: [<ffffffff883fa686>] :obdclass:lprocfs_oh_tally+0x26/0x50 May 17 22:01:05 node8 kernel: [<ffffffff886ba31c>] :fsfilt_ldiskfs:fsfilt_ldiskfs_send_bio+0xc/0x20 May 17 22:01:05 node8 kernel: [<ffffffff886ee45e>] :obdfilter:filter_do_bio+0x53e/0xac0 May 17 22:01:05 node8 kernel: [<ffffffff8803374e>] :jbd:journal_callback_set+0x2d/0x47 May 17 22:01:05 node8 kernel: [<ffffffff886ba630>] :fsfilt_ldiskfs:fsfilt_ldiskfs_commit_async+0xd0/0x150 May 17 22:01:05 node8 kernel: [<ffffffff886ef694>] :obdfilter:filter_direct_io+0xcb4/0xce0 May 17 22:01:05 node8 kernel: [<ffffffff886f1535>] :obdfilter:filter_commitrw_write+0x1855/0x2570 May 17 22:01:05 node8 kernel: [<ffffffff88476eca>] :ptlrpc:ldlm_resource_foreach+0x11a/0x390 May 17 22:01:05 node8 kernel: [<ffffffff8003ceb4>] lock_timer_base+0x1b/0x3c May 17 22:01:05 node8 kernel: [<ffffffff8869eb46>] :ost:ost_brw_write+0x21b6/0x28c0 May 17 22:01:05 node8 kernel: [<ffffffff884942d0>] :ptlrpc:ptlrpc_send_reply+0x370/0x380 May 17 22:01:05 node8 kernel: [<ffffffff88493db0>] :ptlrpc:ptl_send_buf+0x4a0/0x650 May 17 22:01:05 node8 kernel: [<ffffffff80088436>] default_wake_function+0x0/0xe May 17 22:01:05 node8 kernel: [<ffffffff886a283e>] :ost:ost_handle+0x2a8e/0x58d8 May 17 22:01:05 node8 kernel: [<ffffffff800d1003>] kmem_freepages+0xe6/0x110 May 17 22:01:05 node8 kernel: [<ffffffff88496c58>] :ptlrpc:lustre_unpack_msg_v1+0x118/0x520 May 17 22:01:05 node8 kernel: [<ffffffff88400cc2>] :obdclass:class_handle2object+0xd2/0x160 May 17 22:01:05 node8 kernel: [<ffffffff8849c220>] :ptlrpc:lustre_swab_ptlrpc_body+0x0/0x90 May 17 22:01:05 node8 kernel: [<ffffffff88499de5>] :ptlrpc:lustre_swab_buf+0xc5/0xf0 May 17 22:01:05 node8 kernel: [<ffffffff884a1a2b>] :ptlrpc:ptlrpc_server_handle_request+0xb0b/0x1270 May 17 22:01:05 node8 kernel: [<ffffffff80060f29>] thread_return+0x0/0xeb May 17 22:01:05 node8 kernel: [<ffffffff8006b6c9>] do_gettimeofday+0x50/0x92 May 17 22:01:05 node8 kernel: [<ffffffff8835a066>] :libcfs:lcw_update_time+0x16/0x100 May 17 22:01:05 node8 kernel: [<ffffffff8003ceb4>] lock_timer_base+0x1b/0x3c May 17 22:01:05 node8 kernel: [<ffffffff884a446c>] :ptlrpc:ptlrpc_main+0x7dc/0x950 May 17 22:01:05 node8 kernel: [<ffffffff80088436>] default_wake_function+0x0/0xe May 17 22:01:05 node8 kernel: [<ffffffff8005bfb1>] child_rip+0xa/0x11 May 17 22:01:05 node8 kernel: [<ffffffff884a3c90>] :ptlrpc:ptlrpc_main+0x0/0x950 May 17 22:01:05 node8 kernel: [<ffffffff8005bfa7>] child_rip+0x0/0x11 May 17 22:01:05 node8 kernel: May 17 22:01:05 node8 kernel: LustreError: dumping log to /tmp/lustre-log.1211054465.7738 May 17 22:01:08 node8 kernel: LustreError: 7738:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0008: slow direct_io 102s May 17 22:01:08 node8 kernel: LustreError: 7738:0:(filter_io_26.c:763:filter_commitrw_write()) Skipped 24 previous similar messages May 17 22:01:08 node8 kernel: LustreError: 7738:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0008: slow commitrw commit 102s May 17 22:01:08 node8 kernel: LustreError: 7738:0:(filter_io_26.c:776:filter_commitrw_write()) Skipped 24 previous similar messages May 17 22:01:08 node8 kernel: LustreError: 7738:0:(service.c:668:ptlrpc_server_handle_request()) request 2179029 opc 4 from [EMAIL PROTECTED] processed in 102s trans 2522315 rc 0/0 May 17 22:01:08 node8 kernel: LustreError: 7738:0:(service.c:668:ptlrpc_server_handle_request()) Skipped 2 previous similar messages May 17 22:01:08 node8 kernel: Lustre: 7738:0:(watchdog.c:312:lcw_update_time()) Expired watchdog for pid 7738 disabled after 102.7567s May 17 22:01:08 node8 kernel: Lustre: 7738:0:(watchdog.c:312:lcw_update_time()) Skipped 2 previous similar messages May 17 22:01:30 node8 kernel: Lustre: 7647:0:(ldlm_lib.c:519:target_handle_reconnect()) cubefs-OST0008: 1a535b99-47d7-7f5f-f3cb-2777930c9217 reconnecting May 17 22:01:30 node8 kernel: Lustre: 7647:0:(ldlm_lib.c:519:target_handle_reconnect()) Skipped 2 previous similar messages May 18 10:10:17 node8 syslogd 1.4.1: restart. I also attach the log dump and if it neccessary I can send more logs. What could be the problem? Lustre, linux or hw? The other node was node1, but it has no messages in logs like this, but before I restarted it, I saw the same on it's screen. In tme the first one was node3. It has in message log this: May 17 19:57:55 node3 kernel: Lustre: cubefs-OST0003: haven't heard from client d955fc8d-934e-2fef-9614-cd5b90f10298 (at [EMAIL PROTECTED]) in 227 seconds. I think it's dead, and I am evicting it. May 17 19:57:55 node3 kernel: Lustre: Skipped 1 previous similar message May 17 19:59:33 node3 kernel: LustreError: 7650:0:(client.c:975:ptlrpc_expire_one_request()) @@@ timeout (sent at 1211047153, 20s ago) [EMAIL PROTECTED] x454190/t0 o104->@NET_0x20000c0a80089_UUID:15 lens 232/128 ref 1 fl Rpc:N/0/0 rc 0/-22 May 17 19:59:33 node3 kernel: LustreError: 138-a: cubefs-OST0003: A client on nid [EMAIL PROTECTED] was evicted due to a lock blocking callback to [EMAIL PROTECTED] timed out: rc -107 May 17 20:03:08 node3 kernel: Lustre: 7109:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow direct_io 31s May 17 20:03:08 node3 kernel: Lustre: 7109:0:(filter_io_26.c:763:filter_commitrw_write()) Skipped 78 previous similar messages May 17 20:03:08 node3 kernel: Lustre: 7109:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow commitrw commit 31s May 17 20:03:08 node3 kernel: Lustre: 7109:0:(filter_io_26.c:776:filter_commitrw_write()) Skipped 79 previous similar messages May 17 20:03:11 node3 kernel: Lustre: 7722:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow direct_io 31s May 17 20:03:11 node3 kernel: Lustre: 7722:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow commitrw commit 31s May 17 20:03:17 node3 kernel: Lustre: 7749:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow direct_io 31s May 17 20:03:17 node3 kernel: Lustre: 7749:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow commitrw commit 31s May 17 20:03:29 node3 kernel: Lustre: 23971:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow direct_io 48s May 17 20:03:29 node3 kernel: Lustre: 23971:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow commitrw commit 48s May 17 20:03:29 node3 kernel: LustreError: 7690:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow direct_io 50s May 17 20:03:29 node3 kernel: LustreError: 7690:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow commitrw commit 50s May 17 20:03:38 node3 kernel: LustreError: 7117:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow direct_io 53s May 17 20:03:38 node3 kernel: LustreError: 7117:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow commitrw commit 53s May 17 20:03:49 node3 kernel: LustreError: 12335:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow direct_io 52s May 17 20:03:49 node3 kernel: LustreError: 12335:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow commitrw commit 52s May 17 20:03:55 node3 kernel: LustreError: 7727:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow direct_io 53s May 17 20:03:55 node3 kernel: LustreError: 7727:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow commitrw commit 53s May 17 20:03:57 node3 kernel: Lustre: 7735:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow direct_io 41s May 17 20:03:57 node3 kernel: Lustre: 7735:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow commitrw commit 41s May 17 20:04:00 node3 kernel: LustreError: 29950:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow direct_io 60s May 17 20:04:00 node3 kernel: LustreError: 29950:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow commitrw commit 60s May 17 20:04:06 node3 kernel: LustreError: 7169:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow direct_io 66s May 17 20:04:06 node3 kernel: LustreError: 7169:0:(filter_io_26.c:763:filter_commitrw_write()) Skipped 1 previous similar message May 17 20:04:06 node3 kernel: LustreError: 7169:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow commitrw commit 66s May 17 20:04:06 node3 kernel: LustreError: 7169:0:(filter_io_26.c:776:filter_commitrw_write()) Skipped 1 previous similar message May 17 20:04:15 node3 kernel: LustreError: 7124:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0003: slow direct_io 67s May 17 20:04:15 node3 kernel: LustreError: 7124:0:(filter_io_26.c:763:filter_commitrw_write()) Skipped 2 previous similar messages May 17 20:04:15 node3 kernel: LustreError: 7124:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0003: slow commitrw commit 67s May 17 20:04:15 node3 kernel: LustreError: 7124:0:(filter_io_26.c:776:filter_commitrw_write()) Skipped 2 previous similar messages May 17 20:46:50 node3 syslogd 1.4.1: restart. Servers are Intel mainboards with Adaptec ASR-2820SA . Linux node3 2.6.18-53.1.13.el5_lustre.1.6.4.3custom #2 SMP Tue Mar 11 12:57:23 CET 2008 x86_64 x86_64 x86_64 GNU/Linux And we use drbd 0.7.25 . Please give me some help. Thank you, tamas ps.: I' don't know, it is relevant or not, but after this whole thing I restarted the whole cluster and node2 was cannot be mounted. it said, "File exists" This is the log for it: May 18 11:16:29 node2 kernel: Lustre: OBD class driver, [EMAIL PROTECTED] May 18 11:16:29 node2 kernel: Lustre Version: 1.6.4.3 May 18 11:16:29 node2 kernel: Build Version: 1.6.4.3-19700101010000-PRISTINE-.usr.src.linux-2.6.18-53.1.13.el5_lustre.1.6.4.3-2.6.18-53.1.13.el5_lustre.1.6.4.3-cube1 May 18 11:16:29 node2 kernel: Lustre: Added LNI [EMAIL PROTECTED] [8/256] May 18 11:16:29 node2 kernel: Lustre: Accept secure, port 988 May 18 11:16:29 node2 kernel: Lustre: Lustre Client File System; [EMAIL PROTECTED] May 18 11:16:30 node2 kernel: kjournald starting. Commit interval 5 seconds May 18 11:16:30 node2 kernel: LDISKFS FS on drbd1, internal journal May 18 11:16:30 node2 kernel: LDISKFS-fs: mounted filesystem with ordered data mode. May 18 11:16:30 node2 kernel: kjournald starting. Commit interval 5 seconds May 18 11:16:30 node2 kernel: LDISKFS FS on drbd1, internal journal May 18 11:16:30 node2 kernel: LDISKFS-fs: mounted filesystem with ordered data mode. May 18 11:16:30 node2 kernel: LDISKFS-fs: file extents enabled May 18 11:16:30 node2 kernel: LDISKFS-fs: mballoc enabled May 18 11:16:30 node2 kernel: LustreError: 3952:0:(socklnd_cb.c:2167:ksocknal_recv_hello()) Error -104 reading HELLO from 10.1.1.1 May 18 11:16:30 node2 kernel: LustreError: 11b-b: Connection to [EMAIL PROTECTED] at host 10.1.1.1 on port 988 was reset: is it running a compatible version of Lustre and is [EMAIL PROTECTED] one of its NIDs? May 18 11:16:33 node2 kernel: LustreError: 137-5: UUID 'cubefs-OST0002_UUID' is not available for connect (no target) May 18 11:16:33 node2 kernel: LustreError: 4023:0:(ldlm_lib.c:1442:target_send_reply_msg()) @@@ processing error (-19) [EMAIL PROTECTED] x79617/t0 o8-><?>@<?>:-1 lens 304/0 ref 0 fl Interpret:/0/0 rc -19/0 May 18 11:16:35 node2 kernel: LustreError: 3956:0:(client.c:975:ptlrpc_expire_one_request()) @@@ timeout (sent at 1211102190, 5s ago) [EMAIL PROTECTED] x1/t0 o250->[EMAIL PROTECTED]@tcp_0:26 lens 240/272 ref 1 fl Rpc:/0/0 rc 0/-22 May 18 11:16:35 node2 kernel: Lustre: Changing connection for [EMAIL PROTECTED] to [EMAIL PROTECTED]/[EMAIL PROTECTED] May 18 11:16:35 node2 kernel: Lustre: Binding irq 169 to CPU 0 with cmd: echo 1 > /proc/irq/169/smp_affinity May 18 11:16:35 node2 kernel: Lustre: Filtering OBD driver; [EMAIL PROTECTED] May 18 11:16:37 node2 kernel: LustreError: 3916:0:(recov_thread.c:473:llog_start_commit_thread()) error starting thread #1: -513 May 18 11:16:37 node2 kernel: LustreError: 3916:0:(llog_obd.c:392:llog_cat_initialize()) rc: -513 May 18 11:16:37 node2 kernel: LustreError: 3916:0:(filter.c:1717:filter_common_setup()) failed to setup llogging subsystems May 18 11:16:37 node2 kernel: LustreError: 3916:0:(obd_config.c:325:class_setup()) setup cubefs-OST0002 failed (-513) May 18 11:16:37 node2 kernel: LustreError: 3916:0:(obd_config.c:1062:class_config_llog_handler()) Err -513 on cfg command: May 18 11:16:37 node2 kernel: Lustre: cmd=cf003 0:cubefs-OST0002 1:dev 2:type 3:f May 18 11:16:37 node2 kernel: LustreError: 15c-8: [EMAIL PROTECTED]: The configuration from log 'cubefs-OST0002' failed (-513). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. May 18 11:16:37 node2 kernel: LustreError: 3916:0:(obd_mount.c:1080:server_start_targets()) failed to start server cubefs-OST0002: -513 May 18 11:16:37 node2 kernel: LustreError: 3916:0:(obd_mount.c:1570:server_fill_super()) Unable to start targets: -513 May 18 11:16:37 node2 kernel: LustreError: 3916:0:(obd_config.c:392:class_cleanup()) Device 2 not setup May 18 11:16:37 node2 kernel: LDISKFS-fs: mballoc: 1 blocks 1 reqs (0 success) May 18 11:16:37 node2 kernel: LDISKFS-fs: mballoc: 1 extents scanned, 0 goal hits, 1 2^N hits, 0 breaks, 0 lost May 18 11:16:37 node2 kernel: LDISKFS-fs: mballoc: 2 generated and it took 107912 May 18 11:16:37 node2 kernel: LDISKFS-fs: mballoc: 256 preallocated, 0 discarded May 18 11:16:37 node2 kernel: Lustre: server umount cubefs-OST0002 complete May 18 11:16:37 node2 kernel: LustreError: 3916:0:(obd_mount.c:1924:lustre_fill_super()) Unable to mount (-513) May 18 11:16:38 node2 kernel: kjournald starting. Commit interval 5 seconds May 18 11:16:38 node2 kernel: LDISKFS FS on drbd1, internal journal May 18 11:16:38 node2 kernel: LDISKFS-fs: mounted filesystem with ordered data mode. May 18 11:16:38 node2 kernel: LustreError: 137-5: UUID 'cubefs-OST0002_UUID' is not available for connect (not set up) May 18 11:16:38 node2 kernel: LustreError: 4024:0:(ldlm_lib.c:1442:target_send_reply_msg()) @@@ processing error (-19) [EMAIL PROTECTED] x85843/t0 o8-><?>@<?>:-1 lens 304/0 ref 0 fl Interpret:/0/0 rc -19/0 I restarted the node, and I could mount it. At this moment it's working, but I see this in logs again: May 18 12:05:13 node2 kernel: Lustre: 3507:0:(filter_io_26.c:698:filter_commitrw_write()) cubefs-OST0002: slow i_mutex 46s May 18 12:05:13 node2 kernel: Lustre: 3507:0:(filter_io_26.c:711:filter_commitrw_write()) cubefs-OST0002: slow brw_start 46s May 18 12:05:13 node2 kernel: Lustre: 3478:0:(lustre_fsfilt.h:240:fsfilt_brw_start_log()) cubefs-OST0002: slow journal start 39s May 18 12:05:13 node2 kernel: Lustre: 3483:0:(lustre_fsfilt.h:205:fsfilt_start_log()) cubefs-OST0002: slow journal start 41s May 18 12:05:13 node2 kernel: Lustre: 3507:0:(filter_io_26.c:711:filter_commitrw_write()) Skipped 1 previous similar message May 18 12:05:13 node2 kernel: Lustre: 3515:0:(filter_io_26.c:698:filter_commitrw_write()) cubefs-OST0002: slow i_mutex 46s May 18 12:05:13 node2 kernel: Lustre: 3515:0:(filter_io_26.c:711:filter_commitrw_write()) cubefs-OST0002: slow brw_start 46s May 18 12:05:13 node2 kernel: Lustre: 3506:0:(lustre_fsfilt.h:296:fsfilt_commit_wait()) cubefs-OST0002: slow journal start 46s May 18 12:05:13 node2 kernel: Lustre: 3506:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0002: slow commitrw commit 46s May 18 12:05:13 node2 kernel: Lustre: 3478:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0002: slow direct_io 39s May 18 12:05:13 node2 kernel: Lustre: 3478:0:(filter_io_26.c:776:filter_commitrw_write()) cubefs-OST0002: slow commitrw commit 39s May 18 12:05:13 node2 kernel: Lustre: 3478:0:(filter_io_26.c:776:filter_commitrw_write()) Skipped 1 previous similar message May 18 12:05:13 node2 kernel: Lustre: 3515:0:(filter_io_26.c:763:filter_commitrw_write()) cubefs-OST0002: slow direct_io 47s May 18 12:05:13 node2 kernel: Lustre: 3515:0:(filter_io_26.c:763:filter_commitrw_write()) Skipped 1 previous similar message _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
