Anil, Your error message show o8->scratch-OST0003-osc-MDT0000@192.168.1.5@o2ib which means it is trying to connect (opcode o8 is OST connect) to OST0003 ost of scratch file system which is hosted in 192.168.1.5@o2ib nid OSS node but the client has lost connection to the OSS node. This seems to me that you are having network issue. You can ping server nid from client and try to troubleshoot network issue:
client# lctl ping 192.168.1.5@o2ib On Tue, Feb 19, 2019 at 11:44 PM Anilkumar Naik <anilkumar.j.n...@gmail.com> wrote: > Dear All, > > Lustre file system goes to hung state and unable to know the exact issue > with lustre. Kindly find below information and help us to know the fixes > for flle system kernerl hung issue. > > Cluster Details: > > Oss node/server is mounted with below mount targets. We could able to > mount the client with home mounts and its works for some time. After > 10-15mins all the clients hangs and oss node get rebooted. Kindly help. > > /dev/mapper/mdt-mgt 19G 446M 17G 3% /mdt-mgt > /dev/mapper/mdt-home 140G 2.8G 128G 3% /mdt-home > /dev/mapper/mdt-scratch 140G 759M 130G 1% /mdt-scratch > /dev/mapper/ost-home 3.7T 2.4T 1.1T 69% /ost-home > > Below Lustre packages has been installed at oss node. > ============== > kernel-devel-2.6.32-431.23.3.el6_lustre.x86_64 > lustre-debuginfo-2.5.3-2.6.32_431.23.3.el6_lustre.x86_64.x86_64 > lustre-2.5.3-2.6.32_431.23.3.el6_lustre.x86_64.x86_64 > kernel-firmware-2.6.32-431.23.3.el6_lustre.x86_64 > lustre-iokit-2.5.3-2.6.32_431.23.3.el6_lustre.x86_64.x86_64 > kernel-2.6.32-431.23.3.el6_lustre.x86_64 > lustre-modules-2.5.3-2.6.32_431.23.3.el6_lustre.x86_64.x86_64 > lustre-tests-2.5.3-2.6.32_431.23.3.el6_lustre.x86_64.x86_64 > kernel-debuginfo-common-x86_64-2.6.32-431.23.3.el6_lustre.x86_64 > lustre-osd-ldiskfs-2.5.3-2.6.32_431.23.3.el6_lustre.x86_64.x86_64 > kernel-debuginfo-2.6.32-431.23.3.el6_lustre.x86_64 > ===================== > > Lustre errors: > ===== > Feb 20 06:22:06 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 17 previous > similar messages > Feb 20 06:29:11 oss1 kernel: LustreError: 137-5: scratch-OST0001_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 06:29:11 oss1 kernel: LustreError: Skipped 16 previous similar > messages > Feb 20 06:29:11 oss1 kernel: LustreError: 11-0: > scratch-OST0001-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 06:29:11 oss1 kernel: LustreError: Skipped 16 previous similar > messages > Feb 20 06:32:42 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > timed out for sent delay: [sent 1550624551/real 0] req@ffff880800be1000 > x1625913123994836/t0(0) o8->scratch-OST0003-osc-MDT0000@192.168.1.5@o2ib:28/4 > lens 400/544 e 0 to 1 dl 1550624562 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 06:32:42 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 15 previous > similar messages > Feb 20 06:39:36 oss1 kernel: LustreError: 137-5: scratch-OST0003_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 06:39:36 oss1 kernel: LustreError: Skipped 17 previous similar > messages > Feb 20 06:39:36 oss1 kernel: LustreError: 11-0: > scratch-OST0003-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 06:39:36 oss1 kernel: LustreError: Skipped 17 previous similar > messages > Feb 20 06:43:12 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > timed out for sent delay: [sent 1550625151/real 0] req@ffff880800dcd000 > x1625913123996040/t0(0) o8->scratch-OST0001-osc-MDT0000@192.168.1.5@o2ib:28/4 > lens 400/544 e 0 to 1 dl 1550625192 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 06:43:12 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 17 previous > similar messages > Feb 20 06:50:01 oss1 kernel: LustreError: 137-5: scratch-OST0003_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 06:50:01 oss1 kernel: LustreError: Skipped 15 previous similar > messages > Feb 20 06:50:01 oss1 kernel: LustreError: 11-0: > scratch-OST0003-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 06:50:01 oss1 kernel: LustreError: Skipped 15 previous similar > messages > Feb 20 06:53:57 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > timed out for sent delay: [sent 1550625826/real 0] req@ffff881005e88800 > x1625913123997352/t0(0) o8->scratch-OST0003-osc-MDT0000@192.168.1.5@o2ib:28/4 > lens 400/544 e 0 to 1 dl 1550625837 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 06:53:57 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 15 previous > similar messages > Feb 20 07:00:51 oss1 kernel: LustreError: 137-5: scratch-OST0003_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 07:00:51 oss1 kernel: LustreError: Skipped 17 previous similar > messages > Feb 20 07:00:51 oss1 kernel: LustreError: 11-0: > scratch-OST0003-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 07:00:51 oss1 kernel: LustreError: Skipped 17 previous similar > messages > Feb 20 07:04:32 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > timed out for sent delay: [sent 1550626426/real 0] req@ffff880fc2b02800 > x1625913123998556/t0(0) o8->scratch-OST0001-osc-MDT0000@192.168.1.5@o2ib:28/4 > lens 400/544 e 0 to 1 dl 1550626472 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 07:04:32 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 17 previous > similar messages > Feb 20 07:12:06 oss1 kernel: LustreError: 137-5: scratch-OST0003_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 07:12:06 oss1 kernel: LustreError: Skipped 15 previous similar > messages > Feb 20 07:12:06 oss1 kernel: LustreError: 11-0: > scratch-OST0003-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 07:12:06 oss1 kernel: LustreError: Skipped 15 previous similar > messages > Feb 20 07:14:42 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > timed out for sent delay: [sent 1550627076/real 0] req@ffff880fb1d50000 > x1625913123999836/t0(0) o8->scratch-OST0001-osc-MDT0000@192.168.1.5@o2ib:28/4 > lens 400/544 e 0 to 1 dl 1550627082 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 07:14:42 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 16 previous > similar messages > Feb 20 07:23:21 oss1 kernel: LustreError: 137-5: scratch-OST0003_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 07:23:21 oss1 kernel: LustreError: Skipped 20 previous similar > messages > Feb 20 07:23:21 oss1 kernel: LustreError: 11-0: > scratch-OST0003-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 07:23:21 oss1 kernel: LustreError: Skipped 20 previous similar > messages > Feb 20 07:25:51 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > failed due to network error: [sent 1550627726/real 1550627751] > req@ffff880fd292c000 x1625913124001124/t0(0) > o8->scratch-OST0003-osc-MDT0000@192.168.1.5@o2ib:28/4 lens 400/544 e 0 to > 1 dl 1550627782 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 07:25:51 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 17 previous > similar messages > Feb 20 07:33:46 oss1 kernel: LustreError: 137-5: scratch-OST0001_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 07:33:46 oss1 kernel: LustreError: Skipped 14 previous similar > messages > Feb 20 07:33:46 oss1 kernel: LustreError: 11-0: > scratch-OST0001-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 07:33:46 oss1 kernel: LustreError: Skipped 14 previous similar > messages > Feb 20 07:35:52 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > timed out for sent delay: [sent 1550628326/real 0] req@ffff880fb23b4c00 > x1625913124002296/t0(0) o8->scratch-OST0003-osc-MDT0000@192.168.1.5@o2ib:28/4 > lens 400/544 e 0 to 1 dl 1550628352 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 07:35:52 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 15 previous > similar messages > Feb 20 07:44:36 oss1 kernel: LustreError: 137-5: scratch-OST0001_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 07:44:36 oss1 kernel: LustreError: Skipped 18 previous similar > messages > Feb 20 07:44:36 oss1 kernel: LustreError: 11-0: > scratch-OST0001-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 07:44:36 oss1 kernel: LustreError: Skipped 18 previous similar > messages > Feb 20 07:47:31 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > failed due to network error: [sent 1550629001/real 1550629051] > req@ffff880fb1fa6000 x1625913124003644/t0(0) > o8->scratch-OST0001-osc-MDT0000@192.168.1.5@o2ib:28/4 lens 400/544 e 0 to > 1 dl 1550629056 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 07:47:31 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 17 previous > similar messages > Feb 20 07:55:01 oss1 kernel: LustreError: 137-5: scratch-OST0003_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 07:55:01 oss1 kernel: LustreError: Skipped 14 previous similar > messages > Feb 20 07:55:01 oss1 kernel: LustreError: 11-0: > scratch-OST0003-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 07:55:01 oss1 kernel: LustreError: Skipped 14 previous similar > messages > Feb 20 07:57:56 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > failed due to network error: [sent 1550629651/real 1550629676] > req@ffff880fe52fd800 x1625913124004920/t0(0) > o8->scratch-OST0003-osc-MDT0000@192.168.1.5@o2ib:28/4 lens 400/544 e 0 to > 1 dl 1550629682 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 07:57:56 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 17 previous > similar messages > Feb 20 08:05:51 oss1 kernel: LustreError: 137-5: scratch-OST0001_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 08:05:51 oss1 kernel: LustreError: Skipped 17 previous similar > messages > Feb 20 08:05:51 oss1 kernel: LustreError: 11-0: > scratch-OST0001-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 08:05:51 oss1 kernel: LustreError: Skipped 18 previous similar > messages > Feb 20 08:08:46 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > failed due to network error: [sent 1550630276/real 1550630326] > req@ffff880fb1f31400 x1625913124006160/t0(0) > o8->scratch-OST0001-osc-MDT0000@192.168.1.5@o2ib:28/4 lens 400/544 e 0 to > 1 dl 1550630332 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 08:08:46 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 15 previous > similar messages > Feb 20 08:16:16 oss1 kernel: LustreError: 137-5: scratch-OST0001_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 08:16:16 oss1 kernel: LustreError: Skipped 15 previous similar > messages > Feb 20 08:16:16 oss1 kernel: LustreError: 11-0: > scratch-OST0001-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 08:16:16 oss1 kernel: LustreError: Skipped 16 previous similar > messages > Feb 20 08:18:47 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > timed out for sent delay: [sent 1550630901/real 0] req@ffff880fb0c00000 > x1625913124007400/t0(0) o8->scratch-OST0001-osc-MDT0000@192.168.1.5@o2ib:28/4 > lens 400/544 e 0 to 1 dl 1550630927 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 08:18:47 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 17 previous > similar messages > Feb 20 08:27:31 oss1 kernel: LustreError: 137-5: scratch-OST0003_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 08:27:31 oss1 kernel: LustreError: Skipped 16 previous similar > messages > Feb 20 08:27:31 oss1 kernel: LustreError: 11-0: > scratch-OST0001-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 08:27:31 oss1 kernel: LustreError: Skipped 16 previous similar > messages > Feb 20 08:30:26 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > failed due to network error: [sent 1550631576/real 1550631626] > req@ffff880fb0ebb400 x1625913124008720/t0(0) > o8->scratch-OST0001-osc-MDT0000@192.168.1.5@o2ib:28/4 lens 400/544 e 0 to > 1 dl 1550631632 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 08:30:26 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 15 previous > similar messages > Feb 20 08:38:21 oss1 kernel: LustreError: 137-5: scratch-OST0001_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 08:38:21 oss1 kernel: LustreError: Skipped 15 previous similar > messages > Feb 20 08:38:21 oss1 kernel: LustreError: 11-0: > scratch-OST0001-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 08:38:21 oss1 kernel: LustreError: Skipped 17 previous similar > messages > Feb 20 08:40:32 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > timed out for sent delay: [sent 1550632201/real 0] req@ffff880fb0fd6000 > x1625913124009960/t0(0) o8->scratch-OST0001-osc-MDT0000@192.168.1.5@o2ib:28/4 > lens 400/544 e 0 to 1 dl 1550632232 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 08:40:32 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 17 previous > similar messages > Feb 20 08:49:36 oss1 kernel: LustreError: 137-5: scratch-OST0003_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 08:49:36 oss1 kernel: LustreError: Skipped 15 previous similar > messages > Feb 20 08:49:36 oss1 kernel: LustreError: 11-0: > scratch-OST0003-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 08:49:36 oss1 kernel: LustreError: Skipped 15 previous similar > messages > Feb 20 08:50:51 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > failed due to network error: [sent 1550632801/real 1550632851] > req@ffff880fab861000 x1625913124011132/t0(0) > o8->scratch-OST0001-osc-MDT0000@192.168.1.5@o2ib:28/4 lens 400/544 e 0 to > 1 dl 1550632857 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 08:50:51 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 13 previous > similar messages > Feb 20 09:00:01 oss1 kernel: LustreError: 137-5: scratch-OST0001_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 09:00:01 oss1 kernel: LustreError: Skipped 17 previous similar > messages > Feb 20 09:00:01 oss1 kernel: LustreError: 11-0: > scratch-OST0001-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 09:00:01 oss1 kernel: LustreError: Skipped 17 previous similar > messages > Feb 20 09:01:02 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > timed out for sent delay: [sent 1550633426/real 0] req@ffff880fb0f20000 > x1625913124012372/t0(0) o8->scratch-OST0001-osc-MDT0000@192.168.1.5@o2ib:28/4 > lens 400/544 e 0 to 1 dl 1550633462 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 09:01:02 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 17 previous > similar messages > Feb 20 09:11:16 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > failed due to network error: [sent 1550634026/real 1550634076] > req@ffff881016296400 x1625913124013544/t0(0) > o8->scratch-OST0001-osc-MDT0000@192.168.1.5@o2ib:28/4 lens 400/544 e 0 to > 1 dl 1550634082 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 09:11:16 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 13 previous > similar messages > Feb 20 09:11:41 oss1 kernel: LustreError: 137-5: scratch-OST0003_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 09:11:41 oss1 kernel: LustreError: Skipped 15 previous similar > messages > Feb 20 09:11:41 oss1 kernel: LustreError: 11-0: > scratch-OST0003-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 09:11:41 oss1 kernel: LustreError: Skipped 15 previous similar > messages > Feb 20 09:21:52 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > timed out for sent delay: [sent 1550634676/real 0] req@ffff880fb1dae400 > x1625913124014832/t0(0) o8->scratch-OST0001-osc-MDT0000@192.168.1.5@o2ib:28/4 > lens 400/544 e 0 to 1 dl 1550634712 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 09:21:52 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 17 previous > similar messages > Feb 20 09:22:06 oss1 kernel: LustreError: 137-5: scratch-OST0003_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 09:22:06 oss1 kernel: LustreError: Skipped 17 previous similar > messages > Feb 20 09:22:06 oss1 kernel: LustreError: 11-0: > scratch-OST0003-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 09:22:06 oss1 kernel: LustreError: Skipped 17 previous similar > messages > Feb 20 09:32:42 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > timed out for sent delay: [sent 1550635351/real 0] req@ffff880fb0e9f800 > x1625913124016140/t0(0) o8->scratch-OST0003-osc-MDT0000@192.168.1.5@o2ib:28/4 > lens 400/544 e 0 to 1 dl 1550635362 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 09:32:42 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 15 previous > similar messages > Feb 20 09:32:56 oss1 kernel: LustreError: 137-5: scratch-OST0003_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 09:32:56 oss1 kernel: LustreError: Skipped 15 previous similar > messages > Feb 20 09:32:56 oss1 kernel: LustreError: 11-0: > scratch-OST0003-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 09:32:56 oss1 kernel: LustreError: Skipped 15 previous similar > messages > Feb 20 09:43:12 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > timed out for sent delay: [sent 1550635951/real 0] req@ffff880fb0e79c00 > x1625913124017344/t0(0) o8->scratch-OST0001-osc-MDT0000@192.168.1.5@o2ib:28/4 > lens 400/544 e 0 to 1 dl 1550635992 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 09:43:12 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 17 previous > similar messages > Feb 20 09:43:21 oss1 kernel: LustreError: 137-5: scratch-OST0001_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 09:43:21 oss1 kernel: LustreError: Skipped 17 previous similar > messages > Feb 20 09:43:21 oss1 kernel: LustreError: 11-0: > scratch-OST0001-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 09:43:21 oss1 kernel: LustreError: Skipped 17 previous similar > messages > Feb 20 09:53:32 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > timed out for sent delay: [sent 1550636601/real 0] req@ffff880fc2466800 > x1625913124018600/t0(0) o8->scratch-OST0003-osc-MDT0000@192.168.1.5@o2ib:28/4 > lens 400/544 e 0 to 1 dl 1550636612 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 09:53:32 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 13 previous > similar messages > Feb 20 09:53:46 oss1 kernel: LustreError: 137-5: scratch-OST0001_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 09:53:46 oss1 kernel: LustreError: 137-5: scratch-OST0003_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 09:53:46 oss1 kernel: LustreError: Skipped 13 previous similar > messages > Feb 20 09:53:46 oss1 kernel: LustreError: 11-0: > scratch-OST0001-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 09:53:46 oss1 kernel: LustreError: Skipped 13 previous similar > messages > Feb 20 10:03:37 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > timed out for sent delay: [sent 1550637176/real 0] req@ffff880fb1f71400 > x1625913124019756/t0(0) o8->scratch-OST0001-osc-MDT0000@192.168.1.5@o2ib:28/4 > lens 400/544 e 0 to 1 dl 1550637217 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 10:03:37 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 17 previous > similar messages > Feb 20 10:04:11 oss1 kernel: LustreError: 137-5: scratch-OST0003_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 10:04:11 oss1 kernel: LustreError: Skipped 17 previous similar > messages > Feb 20 10:04:11 oss1 kernel: LustreError: 11-0: > scratch-OST0003-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 10:04:11 oss1 kernel: LustreError: Skipped 18 previous similar > messages > Feb 20 10:13:57 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > timed out for sent delay: [sent 1550637826/real 0] req@ffff880fb0d45c00 > x1625913124021024/t0(0) o8->scratch-OST0003-osc-MDT0000@192.168.1.5@o2ib:28/4 > lens 400/544 e 0 to 1 dl 1550637837 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 10:13:57 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 15 previous > similar messages > Feb 20 10:15:01 oss1 kernel: LustreError: 137-5: scratch-OST0003_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 10:15:01 oss1 kernel: LustreError: Skipped 14 previous similar > messages > Feb 20 10:15:01 oss1 kernel: LustreError: 11-0: > scratch-OST0003-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 10:15:01 oss1 kernel: LustreError: Skipped 15 previous similar > messages > Feb 20 10:24:32 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > timed out for sent delay: [sent 1550638426/real 0] req@ffff880fb0d2b000 > x1625913124022228/t0(0) o8->scratch-OST0001-osc-MDT0000@192.168.1.5@o2ib:28/4 > lens 400/544 e 0 to 1 dl 1550638472 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 10:24:32 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 17 previous > similar messages > Feb 20 10:26:16 oss1 kernel: LustreError: 137-5: scratch-OST0003_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 10:26:16 oss1 kernel: LustreError: Skipped 18 previous similar > messages > Feb 20 10:26:16 oss1 kernel: LustreError: 11-0: > scratch-OST0003-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 10:26:16 oss1 kernel: LustreError: Skipped 18 previous similar > messages > Feb 20 10:34:42 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > timed out for sent delay: [sent 1550639076/real 0] req@ffff880fb1cd2800 > x1625913124023492/t0(0) o8->scratch-OST0001-osc-MDT0000@192.168.1.5@o2ib:28/4 > lens 400/544 e 0 to 1 dl 1550639082 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 10:34:42 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 14 previous > similar messages > Feb 20 10:36:41 oss1 kernel: LustreError: 137-5: scratch-OST0003_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 10:36:41 oss1 kernel: LustreError: Skipped 15 previous similar > messages > Feb 20 10:36:41 oss1 kernel: LustreError: 11-0: > scratch-OST0003-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 10:36:41 oss1 kernel: LustreError: Skipped 15 previous similar > messages > Feb 20 10:45:01 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > failed due to network error: [sent 1550639676/real 1550639701] > req@ffff880fb0e35000 x1625913124024684/t0(0) > o8->scratch-OST0003-osc-MDT0000@192.168.1.5@o2ib:28/4 lens 400/544 e 0 to > 1 dl 1550639731 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 10:45:01 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 16 previous > similar messages > Feb 20 10:47:06 oss1 kernel: LustreError: 137-5: scratch-OST0003_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 10:47:06 oss1 kernel: LustreError: Skipped 15 previous similar > messages > Feb 20 10:47:06 oss1 kernel: LustreError: 11-0: > scratch-OST0003-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 10:47:06 oss1 kernel: LustreError: Skipped 15 previous similar > messages > Feb 20 10:55:27 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has > timed out for sent delay: [sent 1550640301/real 0] req@ffff880fb0e70000 > x1625913124025896/t0(0) o8->scratch-OST0003-osc-MDT0000@192.168.1.5@o2ib:28/4 > lens 400/544 e 0 to 1 dl 1550640327 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 > Feb 20 10:55:27 oss1 kernel: Lustre: > 6285:0:(client.c:1918:ptlrpc_expire_one_request()) Skipped 14 previous > similar messages > Feb 20 10:57:56 oss1 kernel: LustreError: 137-5: scratch-OST0001_UUID: not > available for connect from 0@lo (no target). If you are running an HA > pair check that the target is mounted on the other server. > Feb 20 10:57:56 oss1 kernel: LustreError: Skipped 16 previous similar > messages > Feb 20 10:57:56 oss1 kernel: LustreError: 11-0: > scratch-OST0001-osc-MDT0000: Communicating with 0@lo, operation > ost_connect failed with -19. > Feb 20 10:57:56 oss1 kernel: LustreError: Skipped 16 previous similar > messages > ================= > > -- > Regards, > A J Naik > Computer Centre, TIFR. > +91 022 22782342 / 2121 > > _______________________________________________ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >
_______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
