Dell Customer Communication - Confidential Ben, please call into the main support line so we can take a look. (If you wait till tomorrow you might get me!)
How long has this been going on? It could definitely be a card issue, does the iDRAC report anything? Firmware, driver, FC switch could also be in play. Just the one server, correct? Is it always 0000:3b:00.1? (Meaning, 0000:3b:00.0 or the other bus/function.) Be sure to have a sosreport and SupportAssist Collection ready. -----Original Message----- From: linux-poweredge-bounces-Lists <[email protected]> On Behalf Of Ben Argyle Sent: Wednesday, November 6, 2019 4:33 AM To: linux-poweredge-Lists Subject: [Linux-PowerEdge] RHEL7 or FC card / SFP / cable issues? [EXTERNAL EMAIL] We have a RHEL7 OS (fully patched as of 2019-11-01, kernel 3.10.0-1062.4.1.el7.x86_64) running on a Dell PowerEdge R640 server with a pair of "QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)" cards to provide FC connectivity to our Dell/Compellent SAN. Frequently we see issues similar to the below. We've tried changing the FC cables, but wonder if perhaps the card(s) are at fault. They're at the most recent Dell-version firmware as of this quarter, available using Dell's 'dsu' application. # multipath -ll vold09 (36000d31000e394000000000000000165) dm-7 COMPELNT,Compellent Vol size=2.9T features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active |- 15:0:2:2 sdk 8:160 active ready running `- 16:0:3:2 sdd 8:48 active ready running vold11 (36000d31000e394000000000000000214) dm-6 COMPELNT,Compellent Vol size=2.9T features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active |- 15:0:2:6 sdo 8:224 active ready running `- 16:0:3:6 sdh 8:112 active ready running vold08 (36000d31000e394000000000000000164) dm-8 COMPELNT,Compellent Vol size=20G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active |- 15:0:1:3 sdj 8:144 active ready running `- 16:0:2:3 sdc 8:32 active ready running vold10 (36000d31000e39400000000000000018d) dm-4 COMPELNT,Compellent Vol size=1000G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active |- 15:0:2:5 sdn 8:208 active ready running `- 16:0:3:5 sdg 8:96 active ready running vold05 (36000d31000e394000000000000000151) dm-5 COMPELNT,Compellent Vol size=1000G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active |- 15:0:2:4 sdm 8:192 active ready running `- 16:0:3:4 sdf 8:80 active ready running vold04 (36000d31000e394000000000000000150) dm-2 COMPELNT,Compellent Vol size=150G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active |- 15:0:2:3 sdl 8:176 active ready running `- 16:0:3:3 sde 8:64 active ready running vold03 (36000d31000e39400000000000000014f) dm-3 COMPELNT,Compellent Vol size=2.0T features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active |- 15:0:1:1 sdi 8:128 active ready running `- 16:0:2:1 sdb 8:16 active ready running # dmesg -T [Fri Nov 1 21:41:09 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:2:1 -- 0 2003. [Fri Nov 1 21:41:09 2019] qla2xxx [0000:3b:00.1]-8009:16: DEVICE RESET ISSUED nexus=16:2:1 cmd=ffff94f5e2ee64c0. [Fri Nov 1 21:41:09 2019] qla2xxx [0000:3b:00.1]-800e:16: DEVICE RESET SUCCEEDED nexus:16:2:1 cmd=ffff94f5e2ee64c0. [Fri Nov 1 21:41:09 2019] sd 15:0:1:1: alua: port group f03d state A non-preferred supports toluSNA [Fri Nov 1 21:41:09 2019] sd 16:0:2:1: alua: port group f03f state A non-preferred supports toluSNA [Sat Nov 2 19:22:14 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Sat Nov 2 19:22:14 2019] qla2xxx [0000:3b:00.1]-8009:16: DEVICE RESET ISSUED nexus=16:3:6 cmd=ffff951007e58700. [Sat Nov 2 19:22:14 2019] qla2xxx [0000:3b:00.1]-800e:16: DEVICE RESET SUCCEEDED nexus:16:3:6 cmd=ffff951007e58700. [Sat Nov 2 19:22:14 2019] sd 16:0:3:6: alua: port group f040 state A non-preferred supports toluSNA [Sat Nov 2 19:22:14 2019] sd 15:0:2:6: alua: port group f03e state A non-preferred supports toluSNA [Sat Nov 2 21:27:54 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:2:1 -- 0 2003. [Sat Nov 2 21:27:56 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:3 -- 0 2003. [Sat Nov 2 21:27:56 2019] qla2xxx [0000:3b:00.1]-8009:16: DEVICE RESET ISSUED nexus=16:2:1 cmd=ffff9505d893b2c0. [Sat Nov 2 21:27:56 2019] qla2xxx [0000:3b:00.1]-800e:16: DEVICE RESET SUCCEEDED nexus:16:2:1 cmd=ffff9505d893b2c0. [Sat Nov 2 21:27:56 2019] qla2xxx [0000:3b:00.1]-8009:16: DEVICE RESET ISSUED nexus=16:3:3 cmd=ffff952b4db38a80. [Sat Nov 2 21:27:56 2019] qla2xxx [0000:3b:00.1]-800e:16: DEVICE RESET SUCCEEDED nexus:16:3:3 cmd=ffff952b4db38a80. [Sat Nov 2 21:27:56 2019] sd 16:0:3:3: alua: port group f040 state A non-preferred supports toluSNA [Sat Nov 2 21:27:56 2019] sd 15:0:2:3: alua: port group f03e state A non-preferred supports toluSNA [Sat Nov 2 21:27:56 2019] sd 16:0:2:1: alua: port group f03f state A non-preferred supports toluSNA [Sat Nov 2 21:27:56 2019] sd 15:0:1:1: alua: port group f03d state A non-preferred supports toluSNA [Sat Nov 2 21:33:26 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Sat Nov 2 21:33:26 2019] qla2xxx [0000:3b:00.1]-8009:16: DEVICE RESET ISSUED nexus=16:3:6 cmd=ffff9517f8f216c0. [Sat Nov 2 21:33:26 2019] qla2xxx [0000:3b:00.1]-800e:16: DEVICE RESET SUCCEEDED nexus:16:3:6 cmd=ffff9517f8f216c0. [Sat Nov 2 21:33:26 2019] sd 16:0:3:6: alua: port group f040 state A non-preferred supports toluSNA [Sat Nov 2 21:33:27 2019] sd 15:0:2:6: alua: port group f03e state A non-preferred supports toluSNA [Sun Nov 3 18:37:40 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Sun Nov 3 18:37:40 2019] qla2xxx [0000:3b:00.1]-8009:16: DEVICE RESET ISSUED nexus=16:3:6 cmd=ffff950f90ec8380. [Sun Nov 3 18:37:40 2019] qla2xxx [0000:3b:00.1]-800e:16: DEVICE RESET SUCCEEDED nexus:16:3:6 cmd=ffff950f90ec8380. [Sun Nov 3 18:37:40 2019] sd 16:0:3:6: alua: port group f040 state A non-preferred supports toluSNA [Sun Nov 3 18:37:40 2019] sd 15:0:2:6: alua: port group f03e state A non-preferred supports toluSNA [Sun Nov 3 18:38:13 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Sun Nov 3 18:38:13 2019] qla2xxx [0000:3b:00.1]-8009:16: DEVICE RESET ISSUED nexus=16:3:6 cmd=ffff94fef063f700. [Sun Nov 3 18:38:13 2019] qla2xxx [0000:3b:00.1]-800e:16: DEVICE RESET SUCCEEDED nexus:16:3:6 cmd=ffff94fef063f700. [Sun Nov 3 18:38:13 2019] sd 16:0:3:6: alua: port group f040 state A non-preferred supports toluSNA [Sun Nov 3 18:38:13 2019] sd 15:0:2:6: alua: port group f03e state A non-preferred supports toluSNA [Sun Nov 3 18:38:44 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Sun Nov 3 18:38:55 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Sun Nov 3 18:38:55 2019] qla2xxx [0000:3b:00.1]-8009:16: DEVICE RESET ISSUED nexus=16:3:6 cmd=ffff94e0c8e81a40. [Sun Nov 3 18:38:55 2019] qla2xxx [0000:3b:00.1]-800e:16: DEVICE RESET SUCCEEDED nexus:16:3:6 cmd=ffff94e0c8e81a40. [Sun Nov 3 18:38:55 2019] sd 16:0:3:6: alua: port group f040 state A non-preferred supports toluSNA [Sun Nov 3 18:38:55 2019] sd 15:0:2:6: alua: port group f03e state A non-preferred supports toluSNA [Sun Nov 3 18:39:31 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Sun Nov 3 18:39:37 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Sun Nov 3 18:39:38 2019] qla2xxx [0000:3b:00.1]-8009:16: DEVICE RESET ISSUED nexus=16:3:6 cmd=ffff95216b7a79c0. [Sun Nov 3 18:39:38 2019] qla2xxx [0000:3b:00.1]-800e:16: DEVICE RESET SUCCEEDED nexus:16:3:6 cmd=ffff95216b7a79c0. [Sun Nov 3 18:39:38 2019] sd 16:0:3:6: alua: port group f040 state A non-preferred supports toluSNA [Sun Nov 3 18:39:38 2019] sd 15:0:2:6: alua: port group f03e state A non-preferred supports toluSNA [Mon Nov 4 08:59:56 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:2:1 -- 0 2003. [Mon Nov 4 08:59:56 2019] qla2xxx [0000:3b:00.1]-8009:16: DEVICE RESET ISSUED nexus=16:2:1 cmd=ffff951e67b23100. [Mon Nov 4 08:59:56 2019] qla2xxx [0000:3b:00.1]-800e:16: DEVICE RESET SUCCEEDED nexus:16:2:1 cmd=ffff951e67b23100. [Mon Nov 4 08:59:56 2019] sd 16:0:2:1: alua: port group f03f state A non-preferred supports toluSNA [Mon Nov 4 08:59:56 2019] sd 15:0:1:1: alua: port group f03d state A non-preferred supports toluSNA [Mon Nov 4 21:51:42 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Mon Nov 4 21:51:42 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:2:1 -- 0 2003. [Mon Nov 4 21:51:43 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:2:1 -- 0 2003. [Mon Nov 4 21:51:43 2019] qla2xxx [0000:3b:00.1]-8009:16: DEVICE RESET ISSUED nexus=16:2:1 cmd=ffff950f903f6840. [Mon Nov 4 21:51:43 2019] qla2xxx [0000:3b:00.1]-800e:16: DEVICE RESET SUCCEEDED nexus:16:2:1 cmd=ffff950f903f6840. [Mon Nov 4 21:51:43 2019] qla2xxx [0000:3b:00.1]-8009:16: DEVICE RESET ISSUED nexus=16:3:6 cmd=ffff9510a16e8a80. [Mon Nov 4 21:51:43 2019] qla2xxx [0000:3b:00.1]-800e:16: DEVICE RESET SUCCEEDED nexus:16:3:6 cmd=ffff9510a16e8a80. [Mon Nov 4 21:51:43 2019] sd 15:0:1:1: alua: port group f03d state A non-preferred supports toluSNA [Mon Nov 4 21:51:43 2019] sd 16:0:2:1: alua: port group f03f state A non-preferred supports toluSNA [Mon Nov 4 21:51:43 2019] sd 16:0:3:6: alua: port group f040 state A non-preferred supports toluSNA [Mon Nov 4 21:51:43 2019] sd 15:0:2:6: alua: port group f03e state A non-preferred supports toluSNA [Tue Nov 5 18:04:18 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 18:04:18 2019] qla2xxx [0000:3b:00.1]-8009:16: DEVICE RESET ISSUED nexus=16:3:6 cmd=ffff9514ba669c00. [Tue Nov 5 18:04:18 2019] qla2xxx [0000:3b:00.1]-800e:16: DEVICE RESET SUCCEEDED nexus:16:3:6 cmd=ffff9514ba669c00. [Tue Nov 5 18:04:18 2019] sd 16:0:3:6: alua: port group f040 state A non-preferred supports toluSNA [Tue Nov 5 18:04:18 2019] sd 15:0:2:6: alua: port group f03e state A non-preferred supports toluSNA [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:20 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:21 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:21 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:21 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:21 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:21 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:21 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:21 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:21 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:21 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:21 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:21 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:21 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:6 -- 0 2003. [Tue Nov 5 19:18:21 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:2:1 -- 0 2003. [Tue Nov 5 19:18:37 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:2:1 -- 0 2003. [Tue Nov 5 19:18:37 2019] qla2xxx [0000:3b:00.1]-8009:16: DEVICE RESET ISSUED nexus=16:2:1 cmd=ffff9506d4e1c000. [Tue Nov 5 19:18:57 2019] qla2xxx [0000:3b:00.1]-8030:16: TM IOCB failed (1). [Tue Nov 5 19:18:57 2019] qla2xxx [0000:3b:00.1]-800c:16: do_reset failed for cmd=ffff9506d4e1c000. [Tue Nov 5 19:18:57 2019] qla2xxx [0000:3b:00.1]-800f:16: DEVICE RESET FAILED: Task management failed nexus=16:2:1 cmd=ffff9506d4e1c000. [Tue Nov 5 19:18:57 2019] qla2xxx [0000:3b:00.1]-8009:16: DEVICE RESET ISSUED nexus=16:2:3 cmd=ffff9536b3aa2300. [Tue Nov 5 19:19:19 2019] qla2xxx [0000:3b:00.1]-800d:16: wait for pending cmds failed for cmd=ffff9536b3aa2300. [Tue Nov 5 19:19:19 2019] qla2xxx [0000:3b:00.1]-800f:16: DEVICE RESET FAILED: Waiting for command completions nexus=16:2:3 cmd=ffff9536b3aa2300. [Tue Nov 5 19:19:19 2019] qla2xxx [0000:3b:00.1]-8009:16: DEVICE RESET ISSUED nexus=16:3:3 cmd=ffff9536b3aa2bc0. [Tue Nov 5 19:19:41 2019] qla2xxx [0000:3b:00.1]-800d:16: wait for pending cmds failed for cmd=ffff9536b3aa2bc0. [Tue Nov 5 19:19:41 2019] qla2xxx [0000:3b:00.1]-800f:16: DEVICE RESET FAILED: Waiting for command completions nexus=16:3:3 cmd=ffff9536b3aa2bc0. [Tue Nov 5 19:19:41 2019] qla2xxx [0000:3b:00.1]-8009:16: DEVICE RESET ISSUED nexus=16:3:6 cmd=ffff9510b2384700. [Tue Nov 5 19:20:01 2019] qla2xxx [0000:3b:00.1]-8030:16: TM IOCB failed (1). [Tue Nov 5 19:20:01 2019] qla2xxx [0000:3b:00.1]-800c:16: do_reset failed for cmd=ffff9510b2384700. [Tue Nov 5 19:20:01 2019] qla2xxx [0000:3b:00.1]-800f:16: DEVICE RESET FAILED: Task management failed nexus=16:3:6 cmd=ffff9510b2384700. [Tue Nov 5 19:20:01 2019] qla2xxx [0000:3b:00.1]-8009:16: TARGET RESET ISSUED nexus=16:3:6 cmd=ffff9510b2384700. [Tue Nov 5 19:20:05 2019] INFO: task jbd2/dm-10-8:4854 blocked for more than 120 seconds. [Tue Nov 5 19:20:05 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Nov 5 19:20:05 2019] jbd2/dm-10-8 D ffff9536b4770000 0 4854 2 0x00000000 [Tue Nov 5 19:20:05 2019] Call Trace: [Tue Nov 5 19:20:05 2019] [<ffffffff9da87042>] ? bio_clone_bioset+0x32/0x70 [Tue Nov 5 19:20:05 2019] [<ffffffffc010c460>] ? dm_use_blk_mq+0x20/0x20 [dm_mod] [Tue Nov 5 19:20:05 2019] [<ffffffff9df7cad0>] ? bit_wait+0x50/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7eb09>] schedule+0x29/0x70 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7c491>] schedule_timeout+0x221/0x2d0 [Tue Nov 5 19:20:05 2019] [<ffffffff9d9048b2>] ? ktime_get_ts64+0x52/0xf0 [Tue Nov 5 19:20:05 2019] [<ffffffff9d95c02f>] ? delayacct_end+0x8f/0xb0 [Tue Nov 5 19:20:05 2019] [<ffffffff9d9048b2>] ? ktime_get_ts64+0x52/0xf0 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7cad0>] ? bit_wait+0x50/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7e07d>] io_schedule_timeout+0xad/0x130 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7e118>] io_schedule+0x18/0x20 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7cae1>] bit_wait_io+0x11/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7c607>] __wait_on_bit+0x67/0x90 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7cad0>] ? bit_wait+0x50/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7c771>] out_of_line_wait_on_bit+0x81/0xb0 [Tue Nov 5 19:20:05 2019] [<ffffffff9d8c6360>] ? wake_bit_function+0x40/0x40 [Tue Nov 5 19:20:05 2019] [<ffffffff9da812ba>] __wait_on_buffer+0x2a/0x30 [Tue Nov 5 19:20:05 2019] [<ffffffffc0205f42>] jbd2_journal_commit_transaction+0xea2/0x19f0 [jbd2] [Tue Nov 5 19:20:05 2019] [<ffffffffc020bee9>] kjournald2+0xc9/0x260 [jbd2] [Tue Nov 5 19:20:05 2019] [<ffffffff9d8c62a0>] ? wake_up_atomic_t+0x30/0x30 [Tue Nov 5 19:20:05 2019] [<ffffffffc020be20>] ? commit_timeout+0x10/0x10 [jbd2] [Tue Nov 5 19:20:05 2019] [<ffffffff9d8c51b1>] kthread+0xd1/0xe0 [Tue Nov 5 19:20:05 2019] [<ffffffff9d8c50e0>] ? insert_kthread_work+0x40/0x40 [Tue Nov 5 19:20:05 2019] [<ffffffff9df8bd1d>] ret_from_fork_nospec_begin+0x7/0x21 [Tue Nov 5 19:20:05 2019] [<ffffffff9d8c50e0>] ? insert_kthread_work+0x40/0x40 [Tue Nov 5 19:20:05 2019] INFO: task jbd2/dm-13-8:4875 blocked for more than 120 seconds. [Tue Nov 5 19:20:05 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Nov 5 19:20:05 2019] jbd2/dm-13-8 D ffff9536b8381070 0 4875 2 0x00000000 [Tue Nov 5 19:20:05 2019] Call Trace: [Tue Nov 5 19:20:05 2019] [<ffffffffc010d12c>] ? dm_old_request_fn+0xcc/0x210 [dm_mod] [Tue Nov 5 19:20:05 2019] [<ffffffff9df7cad0>] ? bit_wait+0x50/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7eb09>] schedule+0x29/0x70 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7c491>] schedule_timeout+0x221/0x2d0 [Tue Nov 5 19:20:05 2019] [<ffffffff9db4d87a>] ? queue_unplugged+0x2a/0xa0 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7cad0>] ? bit_wait+0x50/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7e07d>] io_schedule_timeout+0xad/0x130 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7e118>] io_schedule+0x18/0x20 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7cae1>] bit_wait_io+0x11/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7c607>] __wait_on_bit+0x67/0x90 [Tue Nov 5 19:20:05 2019] [<ffffffff9d9bb061>] wait_on_page_bit+0x81/0xa0 [Tue Nov 5 19:20:05 2019] [<ffffffff9d8c6360>] ? wake_bit_function+0x40/0x40 [Tue Nov 5 19:20:05 2019] [<ffffffff9d9bb191>] __filemap_fdatawait_range+0x111/0x190 [Tue Nov 5 19:20:05 2019] [<ffffffff9d9bb224>] filemap_fdatawait_range+0x14/0x30 [Tue Nov 5 19:20:05 2019] [<ffffffff9d9bb267>] filemap_fdatawait+0x27/0x30 [Tue Nov 5 19:20:05 2019] [<ffffffffc0205b29>] jbd2_journal_commit_transaction+0xa89/0x19f0 [jbd2] [Tue Nov 5 19:20:05 2019] [<ffffffff9d82b59e>] ? __switch_to+0xce/0x580 [Tue Nov 5 19:20:05 2019] [<ffffffffc020bee9>] kjournald2+0xc9/0x260 [jbd2] [Tue Nov 5 19:20:05 2019] [<ffffffff9d8c62a0>] ? wake_up_atomic_t+0x30/0x30 [Tue Nov 5 19:20:05 2019] [<ffffffffc020be20>] ? commit_timeout+0x10/0x10 [jbd2] [Tue Nov 5 19:20:05 2019] [<ffffffff9d8c51b1>] kthread+0xd1/0xe0 [Tue Nov 5 19:20:05 2019] [<ffffffff9d8c50e0>] ? insert_kthread_work+0x40/0x40 [Tue Nov 5 19:20:05 2019] [<ffffffff9df8bd1d>] ret_from_fork_nospec_begin+0x7/0x21 [Tue Nov 5 19:20:05 2019] [<ffffffff9d8c50e0>] ? insert_kthread_work+0x40/0x40 [Tue Nov 5 19:20:05 2019] INFO: task nfsd:5599 blocked for more than 120 seconds. [Tue Nov 5 19:20:05 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Nov 5 19:20:05 2019] nfsd D ffff9536b8385230 0 5599 2 0x00000080 [Tue Nov 5 19:20:05 2019] Call Trace: [Tue Nov 5 19:20:05 2019] [<ffffffff9df7cad0>] ? bit_wait+0x50/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7eb09>] schedule+0x29/0x70 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7c491>] schedule_timeout+0x221/0x2d0 [Tue Nov 5 19:20:05 2019] [<ffffffff9db4d7b9>] ? __blk_run_queue+0x39/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9db4d87a>] ? queue_unplugged+0x2a/0xa0 [Tue Nov 5 19:20:05 2019] [<ffffffff9d9048b2>] ? ktime_get_ts64+0x52/0xf0 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7cad0>] ? bit_wait+0x50/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7e07d>] io_schedule_timeout+0xad/0x130 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7e118>] io_schedule+0x18/0x20 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7cae1>] bit_wait_io+0x11/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7c607>] __wait_on_bit+0x67/0x90 [Tue Nov 5 19:20:05 2019] [<ffffffff9d9bb061>] wait_on_page_bit+0x81/0xa0 [Tue Nov 5 19:20:05 2019] [<ffffffff9d8c6360>] ? wake_bit_function+0x40/0x40 [Tue Nov 5 19:20:05 2019] [<ff ffffff9d9bb191>] __filemap_fdatawait_range+0x111/0x190 [Tue Nov 5 19:20:05 2019] [<ffffffff9d9c92f1>] ? do_writepages+0x21/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9d9bb224>] filemap_fdatawait_range+0x14/0x30 [Tue Nov 5 19:20:05 2019] [<ffffffff9d9bd6e6>] filemap_write_and_wait_range+0x56/0x90 [Tue Nov 5 19:20:05 2019] [<ffffffffc04515da>] ext4_sync_file+0xba/0x320 [ext4] [Tue Nov 5 19:20:05 2019] [<ffffffff9da7e060>] vfs_fsync_range+0x20/0x30 [Tue Nov 5 19:20:05 2019] [<ffffffffc081c639>] nfsd_commit+0xb9/0xe0 [nfsd] [Tue Nov 5 19:20:05 2019] [<ffffffffc0825746>] nfsd3_proc_commit+0x96/0xf0 [nfsd] [Tue Nov 5 19:20:05 2019] [<ffffffffc0816810>] nfsd_dispatch+0xe0/0x290 [nfsd] [Tue Nov 5 19:20:05 2019] [<ffffffffc04f3333>] svc_process_common+0x3d3/0x7c0 [sunrpc] [Tue Nov 5 19:20:05 2019] [<ffffffffc04f3823>] svc_process+0x103/0x190 [sunrpc] [Tue Nov 5 19:20:05 2019] [<ffffffffc081616f>] nfsd+0xdf/0x150 [nfsd] [Tue Nov 5 19:20:05 2019] [<ffffffffc0816090>] ? nfsd_destroy+0x80/0x80 [nfsd] [Tue Nov 5 19:20:05 2019] [<ffffffff9d8c51b1>] kthread+0xd1/0xe0 [Tue Nov 5 19:20:05 2019] [<ffffffff9d8c50e0>] ? insert_kthread_work+0x40/0x40 [Tue Nov 5 19:20:05 2019] [<ffffffff9df8bd1d>] ret_from_fork_nospec_begin+0x7/0x21 [Tue Nov 5 19:20:05 2019] [<ffffffff9d8c50e0>] ? insert_kthread_work+0x40/0x40 [Tue Nov 5 19:20:05 2019] INFO: task ora_dbw1_ufsliv:170243 blocked for more than 120 seconds. [Tue Nov 5 19:20:05 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Nov 5 19:20:05 2019] ora_dbw1_ufsliv D ffff95352cbce2a0 0 170243 1 0x00000080 [Tue Nov 5 19:20:05 2019] Call Trace: [Tue Nov 5 19:20:05 2019] [<ffffffff9df7cad0>] ? bit_wait+0x50/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7eb09>] schedule+0x29/0x70 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7c491>] schedule_timeout+0x221/0x2d0 [Tue Nov 5 19:20:05 2019] [<ffffffff9db7fce5>] ? cpumask_next_and+0x35/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9d8e7514>] ? find_busiest_group+0x144/0x990 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7cad0>] ? bit_wait+0x50/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7e07d>] io_schedule_timeout+0xad/0x130 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7e118>] io_schedule+0x18/0x20 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7cae1>] bit_wait_io+0x11/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7c607>] __wait_on_bit+0x67/0x90 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7cad0>] ? bit_wait+0x50/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7c771>] out_of_line_wait_on_bit+0x81/0xb0 [Tue Nov 5 19:20:05 2019] [<ffffffff9d8c6360>] ? wake_bit_function+0x40/0x40 [Tue Nov 5 19:20:05 2019 ] [<ffffffffc0203cf5>] do_get_write_access+0x285/0x4f0 [jbd2] [Tue Nov 5 19:20:05 2019] [<ffffffff9da8158d>] ? __getblk+0x2d/0x300 [Tue Nov 5 19:20:05 2019] [<ffffffffc0203f87>] jbd2_journal_get_write_access+0x27/0x40 [jbd2] [Tue Nov 5 19:20:05 2019] [<ffffffffc048a37b>] __ext4_journal_get_write_access+0x3b/0x80 [ext4] [Tue Nov 5 19:20:05 2019] [<ffffffffc045a3e0>] ext4_reserve_inode_write+0x70/0xa0 [ext4] [Tue Nov 5 19:20:05 2019] [<ffffffffc045dcc0>] ? ext4_dirty_inode+0x40/0x60 [ext4] [Tue Nov 5 19:20:05 2019] [<ffffffffc045a463>] ext4_mark_inode_dirty+0x53/0x210 [ext4] [Tue Nov 5 19:20:05 2019] [<ffffffffc045dcc0>] ext4_dirty_inode+0x40/0x60 [ext4] [Tue Nov 5 19:20:05 2019] [<ffffffff9da798bd>] __mark_inode_dirty+0x16d/0x270 [Tue Nov 5 19:20:05 2019] [<ffffffff9da66409>] update_time+0x89/0xd0 [Tue Nov 5 19:20:05 2019] [<ffffffff9da664f0>] file_update_time+0xa0/0xf0 [Tue Nov 5 19:20:05 2019] [<ffffffff9d9be1d8>] __generic_file_aio_write+0x198/0x400 [Tue Nov 5 19:20:05 2019] [<ffffffff9d9be499>] generic_file_aio_write+0x59/0xa0 [Tue Nov 5 19:20:05 2019] [<ffffffffc0451352>] ext4_file_write+0xd2/0x1e0 [ext4] [Tue Nov 5 19:20:05 2019] [<ffffffff9da48433>] do_sync_write+0x93/0xe0 [Tue Nov 5 19:20:05 2019] [<ffffffff9da48f20>] vfs_write+0xc0/0x1f0 [Tue Nov 5 19:20:05 2019] [<ffffffff9da49f02>] SyS_pwrite64+0x92/0xc0 [Tue Nov 5 19:20:05 2019] [<ffffffff9df8bede>] system_call_fastpath+0x25/0x2a [Tue Nov 5 19:20:05 2019] INFO: task ora_dbw2_ufsliv:170245 blocked for more than 120 seconds. [Tue Nov 5 19:20:05 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Nov 5 19:20:05 2019] ora_dbw2_ufsliv D ffff9506d268d230 0 170245 1 0x00000080 [Tue Nov 5 19:20:05 2019] Call Trace: [Tue Nov 5 19:20:05 2019] [<ffffffff9d8acca0>] ? __internal_add_timer+0x130/0x130 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7fa29>] schedule_preempt_disabled+0x29/0x70 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7d837>] __mutex_lock_slowpath+0xc7/0x1d0 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7cc0f>] mutex_lock+0x1f/0x2f [Tue Nov 5 19:20:05 2019] [<ffffffff9d9be486>] generic_file_aio_write+0x46/0xa0 [Tue Nov 5 19:20:05 2019] [<ffffffffc0451352>] ext4_file_write+0xd2/0x1e0 [ext4] [Tue Nov 5 19:20:05 2019] [<ffffffff9da48433>] do_sync_write+0x93/0xe0 [Tue Nov 5 19:20:05 2019] [<ffffffff9da48f20>] vfs_write+0xc0/0x1f0 [Tue Nov 5 19:20:05 2019] [<ffffffff9da49f02>] SyS_pwrite64+0x92/0xc0 [Tue Nov 5 19:20:05 2019] [<ffffffff9df8bede>] system_call_fastpath+0x25/0x2a [Tue Nov 5 19:20:05 2019] INFO: task ora_dbw6_ufsliv:170253 blocked for more than 120 seconds. [Tue Nov 5 19:20:05 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Nov 5 19:20:05 2019] ora_dbw6_ufsliv D ffff9506d5ecc1c0 0 170253 1 0x00000080 [Tue Nov 5 19:20:05 2019] Call Trace: [Tue Nov 5 19:20:05 2019] [<ffffffff9df7cad0>] ? bit_wait+0x50/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7eb09>] schedule+0x29/0x70 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7c491>] schedule_timeout+0x221/0x2d0 [Tue Nov 5 19:20:05 2019] [<ffffffff9db7fce5>] ? cpumask_next_and+0x35/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9d8e7514>] ? find_busiest_group+0x144/0x990 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7cad0>] ? bit_wait+0x50/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7e07d>] io_schedule_timeout+0xad/0x130 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7e118>] io_schedule+0x18/0x20 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7cae1>] bit_wait_io+0x11/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7c607>] __wait_on_bit+0x67/0x90 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7cad0>] ? bit_wait+0x50/0x50 [Tue Nov 5 19:20:05 2019] [<ffffffff9df7c771>] out_of_line_wait_on_bit+0x81/0xb0 [Tue Nov 5 19:20:05 2019] [<ffffffff9d8c6360>] ? wake_bit_function+0x40/0x40 [Tue Nov 5 19:20:06 2019 ] [<ffffffffc0203cf5>] do_get_write_access+0x285/0x4f0 [jbd2] [Tue Nov 5 19:20:06 2019] [<ffffffff9da8158d>] ? __getblk+0x2d/0x300 [Tue Nov 5 19:20:06 2019] [<ffffffffc0203f87>] jbd2_journal_get_write_access+0x27/0x40 [jbd2] [Tue Nov 5 19:20:06 2019] [<ffffffffc048a37b>] __ext4_journal_get_write_access+0x3b/0x80 [ext4] [Tue Nov 5 19:20:06 2019] [<ffffffffc045a3e0>] ext4_reserve_inode_write+0x70/0xa0 [ext4] [Tue Nov 5 19:20:06 2019] [<ffffffffc045dcc0>] ? ext4_dirty_inode+0x40/0x60 [ext4] [Tue Nov 5 19:20:06 2019] [<ffffffffc045a463>] ext4_mark_inode_dirty+0x53/0x210 [ext4] [Tue Nov 5 19:20:06 2019] [<ffffffffc045dcc0>] ext4_dirty_inode+0x40/0x60 [ext4] [Tue Nov 5 19:20:06 2019] [<ffffffff9da798bd>] __mark_inode_dirty+0x16d/0x270 [Tue Nov 5 19:20:06 2019] [<ffffffff9da66409>] update_time+0x89/0xd0 [Tue Nov 5 19:20:06 2019] [<ffffffff9da664f0>] file_update_time+0xa0/0xf0 [Tue Nov 5 19:20:06 2019] [<ffffffff9d9be1d8>] __generic_file_aio_write+0x198/0x400 [Tue Nov 5 19:20:06 2019] [<ffffffff9d9be499>] generic_file_aio_write+0x59/0xa0 [Tue Nov 5 19:20:06 2019] [<ffffffffc0451352>] ext4_file_write+0xd2/0x1e0 [ext4] [Tue Nov 5 19:20:06 2019] [<ffffffff9da48433>] do_sync_write+0x93/0xe0 [Tue Nov 5 19:20:06 2019] [<ffffffff9da48f20>] vfs_write+0xc0/0x1f0 [Tue Nov 5 19:20:06 2019] [<ffffffff9da49f02>] SyS_pwrite64+0x92/0xc0 [Tue Nov 5 19:20:06 2019] [<ffffffff9df8bede>] system_call_fastpath+0x25/0x2a [Tue Nov 5 19:20:06 2019] INFO: task ora_lg00_ufsliv:170261 blocked for more than 120 seconds. [Tue Nov 5 19:20:06 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Nov 5 19:20:06 2019] ora_lg00_ufsliv D ffff9506d5ec8000 0 170261 1 0x00000080 [Tue Nov 5 19:20:06 2019] Call Trace: [Tue Nov 5 19:20:06 2019] [<ffffffff9df7cad0>] ? bit_wait+0x50/0x50 [Tue Nov 5 19:20:06 2019] [<ffffffff9df7eb09>] schedule+0x29/0x70 [Tue Nov 5 19:20:06 2019] [<ffffffff9df7c491>] schedule_timeout+0x221/0x2d0 [Tue Nov 5 19:20:06 2019] [<ffffffff9db7fce5>] ? cpumask_next_and+0x35/0x50 [Tue Nov 5 19:20:06 2019] [<ffffffff9d8e7514>] ? find_busiest_group+0x144/0x990 [Tue Nov 5 19:20:06 2019] [<ffffffff9df7cad0>] ? bit_wait+0x50/0x50 [Tue Nov 5 19:20:06 2019] [<ffffffff9df7e07d>] io_schedule_timeout+0xad/0x130 [Tue Nov 5 19:20:06 2019] [<ffffffff9df7e118>] io_schedule+0x18/0x20 [Tue Nov 5 19:20:06 2019] [<ffffffff9df7cae1>] bit_wait_io+0x11/0x50 [Tue Nov 5 19:20:06 2019] [<ffffffff9df7c607>] __wait_on_bit+0x67/0x90 [Tue Nov 5 19:20:06 2019] [<ffffffff9df7cad0>] ? bit_wait+0x50/0x50 [Tue Nov 5 19:20:06 2019] [<ffffffff9df7c771>] out_of_line_wait_on_bit+0x81/0xb0 [Tue Nov 5 19:20:06 2019] [<ffffffff9d8c6360>] ? wake_bit_function+0x40/0x40 [Tue Nov 5 19:20:06 2019 ] [<ffffffffc0203cf5>] do_get_write_access+0x285/0x4f0 [jbd2] [Tue Nov 5 19:20:06 2019] [<ffffffff9da8158d>] ? __getblk+0x2d/0x300 [Tue Nov 5 19:20:06 2019] [<ffffffffc0203f87>] jbd2_journal_get_write_access+0x27/0x40 [jbd2] [Tue Nov 5 19:20:06 2019] [<ffffffffc048a37b>] __ext4_journal_get_write_access+0x3b/0x80 [ext4] [Tue Nov 5 19:20:06 2019] [<ffffffffc045a3e0>] ext4_reserve_inode_write+0x70/0xa0 [ext4] [Tue Nov 5 19:20:06 2019] [<ffffffffc045dcc0>] ? ext4_dirty_inode+0x40/0x60 [ext4] [Tue Nov 5 19:20:06 2019] [<ffffffffc045a463>] ext4_mark_inode_dirty+0x53/0x210 [ext4] [Tue Nov 5 19:20:06 2019] [<ffffffffc045dcc0>] ext4_dirty_inode+0x40/0x60 [ext4] [Tue Nov 5 19:20:06 2019] [<ffffffff9da798bd>] __mark_inode_dirty+0x16d/0x270 [Tue Nov 5 19:20:06 2019] [<ffffffff9da66409>] update_time+0x89/0xd0 [Tue Nov 5 19:20:06 2019] [<ffffffff9da664f0>] file_update_time+0xa0/0xf0 [Tue Nov 5 19:20:06 2019] [<ffffffff9d9be1d8>] __generic_file_aio_write+0x198/0x400 [Tue Nov 5 19:20:06 2019] [<ffffffff9d9be499>] generic_file_aio_write+0x59/0xa0 [Tue Nov 5 19:20:06 2019] [<ffffffffc0451352>] ext4_file_write+0xd2/0x1e0 [ext4] [Tue Nov 5 19:20:06 2019] [<ffffffff9da48433>] do_sync_write+0x93/0xe0 [Tue Nov 5 19:20:06 2019] [<ffffffff9da48f20>] vfs_write+0xc0/0x1f0 [Tue Nov 5 19:20:06 2019] [<ffffffff9da49f02>] SyS_pwrite64+0x92/0xc0 [Tue Nov 5 19:20:06 2019] [<ffffffff9df8bede>] system_call_fastpath+0x25/0x2a [Tue Nov 5 19:20:21 2019] qla2xxx [0000:3b:00.1]-8030:16: TM IOCB failed (1). [Tue Nov 5 19:20:21 2019] qla2xxx [0000:3b:00.1]-800c:16: do_reset failed for cmd=ffff9510b2384700. [Tue Nov 5 19:20:21 2019] qla2xxx [0000:3b:00.1]-800f:16: TARGET RESET FAILED: Task management failed nexus=16:3:6 cmd=ffff9510b2384700. [Tue Nov 5 19:20:21 2019] qla2xxx [0000:3b:00.1]-8009:16: TARGET RESET ISSUED nexus=16:2:1 cmd=ffff9506d4e1c000. [Tue Nov 5 19:20:41 2019] qla2xxx [0000:3b:00.1]-8030:16: TM IOCB failed (1). [Tue Nov 5 19:20:41 2019] qla2xxx [0000:3b:00.1]-800c:16: do_reset failed for cmd=ffff9506d4e1c000. [Tue Nov 5 19:20:41 2019] qla2xxx [0000:3b:00.1]-800f:16: TARGET RESET FAILED: Task management failed nexus=16:2:1 cmd=ffff9506d4e1c000. [Tue Nov 5 19:20:41 2019] qla2xxx [0000:3b:00.1]-8012:16: BUS RESET ISSUED nexus=16:2:3. [Tue Nov 5 19:22:02 2019] qla2xxx [0000:3b:00.1]-802b:16: BUS RESET SUCCEEDED nexus=16:2:3. [Tue Nov 5 19:22:06 2019] INFO: task multipathd:136912 blocked for more than 120 seconds. [Tue Nov 5 19:22:06 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Nov 5 19:22:06 2019] multipathd D ffff94e0fd100000 0 136912 1 0x00000080 [Tue Nov 5 19:22:06 2019] Call Trace: [Tue Nov 5 19:22:06 2019] [<ffffffff9df7eb09>] schedule+0x29/0x70 [Tue Nov 5 19:22:06 2019] [<ffffffff9dce09b5>] scsi_block_when_processing_errors+0xc5/0x140 [Tue Nov 5 19:22:06 2019] [<ffffffff9d8c62a0>] ? wake_up_atomic_t+0x30/0x30 [Tue Nov 5 19:22:06 2019] [<ffffffff9dcde44c>] scsi_nonblockable_ioctl+0xcc/0xf0 [Tue Nov 5 19:22:06 2019] [<ffffffffc027a68e>] sd_ioctl+0x6e/0x140 [sd_mod] [Tue Nov 5 19:22:06 2019] [<ffffffff9db61b6a>] blkdev_ioctl+0x28a/0xa20 [Tue Nov 5 19:22:06 2019] [<ffffffff9d89fa62>] ? release_task+0x392/0x490 [Tue Nov 5 19:22:06 2019] [<ffffffff9da88fe1>] block_ioctl+0x41/0x50 [Tue Nov 5 19:22:06 2019] [<ffffffff9da5e270>] do_vfs_ioctl+0x3a0/0x5a0 [Tue Nov 5 19:22:06 2019] [<ffffffff9df7eb09>] ? schedule+0x29/0x70 [Tue Nov 5 19:22:06 2019] [<ffffffff9da5e511>] SyS_ioctl+0xa1/0xc0 [Tue Nov 5 19:22:06 2019] [<ffffffff9df8bede>] system_call_fastpath+0x25/0x2a [Tue Nov 5 19:22:06 2019] INFO: task multipathd:137671 blocked for more than 120 seconds. [Tue Nov 5 19:22:06 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Nov 5 19:22:06 2019] multipathd D ffff95346b35c1c0 0 137671 1 0x00000080 [Tue Nov 5 19:22:06 2019] Call Trace: [Tue Nov 5 19:22:06 2019] [<ffffffff9df7eb09>] schedule+0x29/0x70 [Tue Nov 5 19:22:06 2019] [<ffffffff9dce09b5>] scsi_block_when_processing_errors+0xc5/0x140 [Tue Nov 5 19:22:06 2019] [<ffffffff9d8c62a0>] ? wake_up_atomic_t+0x30/0x30 [Tue Nov 5 19:22:06 2019] [<ffffffff9dcde44c>] scsi_nonblockable_ioctl+0xcc/0xf0 [Tue Nov 5 19:22:06 2019] [<ffffffffc027a68e>] sd_ioctl+0x6e/0x140 [sd_mod] [Tue Nov 5 19:22:06 2019] [<ffffffff9db61b6a>] blkdev_ioctl+0x28a/0xa20 [Tue Nov 5 19:22:06 2019] [<ffffffff9da88fe1>] block_ioctl+0x41/0x50 [Tue Nov 5 19:22:06 2019] [<ffffffff9da5e270>] do_vfs_ioctl+0x3a0/0x5a0 [Tue Nov 5 19:22:06 2019] [<ffffffff9da5e511>] SyS_ioctl+0xa1/0xc0 [Tue Nov 5 19:22:06 2019] [<ffffffff9df8bede>] system_call_fastpath+0x25/0x2a [Tue Nov 5 19:22:06 2019] INFO: task multipathd:137672 blocked for more than 120 seconds. [Tue Nov 5 19:22:06 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Tue Nov 5 19:22:06 2019] multipathd D ffff95346b35b150 0 137672 1 0x00000080 [Tue Nov 5 19:22:06 2019] Call Trace: [Tue Nov 5 19:22:06 2019] [<ffffffff9df7eb09>] schedule+0x29/0x70 [Tue Nov 5 19:22:06 2019] [<ffffffff9dce09b5>] scsi_block_when_processing_errors+0xc5/0x140 [Tue Nov 5 19:22:06 2019] [<ffffffff9d8c62a0>] ? wake_up_atomic_t+0x30/0x30 [Tue Nov 5 19:22:06 2019] [<ffffffff9dcde44c>] scsi_nonblockable_ioctl+0xcc/0xf0 [Tue Nov 5 19:22:06 2019] [<ffffffffc027a68e>] sd_ioctl+0x6e/0x140 [sd_mod] [Tue Nov 5 19:22:06 2019] [<ffffffff9db61b6a>] blkdev_ioctl+0x28a/0xa20 [Tue Nov 5 19:22:06 2019] [<ffffffff9da88fe1>] block_ioctl+0x41/0x50 [Tue Nov 5 19:22:06 2019] [<ffffffff9da5e270>] do_vfs_ioctl+0x3a0/0x5a0 [Tue Nov 5 19:22:06 2019] [<ffffffff9da5e511>] SyS_ioctl+0xa1/0xc0 [Tue Nov 5 19:22:06 2019] [<ffffffff9df8bede>] system_call_fastpath+0x25/0x2a [Tue Nov 5 19:22:12 2019] sd 16:0:2:3: [sdc] tag#44 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK [Tue Nov 5 19:22:12 2019] sd 16:0:2:3: [sdc] tag#44 CDB: Test Unit Ready 00 00 00 00 00 00 [Tue Nov 5 19:22:12 2019] sd 16:0:2:1: [sdb] tag#43 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK [Tue Nov 5 19:22:12 2019] sd 16:0:2:1: [sdb] tag#43 CD B: Test Unit Ready 00 00 00 00 00 00 [Tue Nov 5 19:22:12 2019] sd 16:0:3:3: [sde] tag#41 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK [Tue Nov 5 19:22:12 2019] sd 16:0:3:3: [sde] tag#41 CDB: Test Unit Ready 00 00 00 00 00 00 [Tue Nov 5 19:22:12 2019] sd 16:0:3:6: [sdh] tag#0 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK [Tue Nov 5 19:22:12 2019] sd 16:0:3:6: [sdh] tag#0 CDB: Test Unit Ready 00 00 00 00 00 00 [Tue Nov 5 19:22:12 2019] sd 16:0:3:3: timing out command, waited 180s [Tue Nov 5 19:22:12 2019] sd 16:0:3:3: [sde] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [Tue Nov 5 19:22:12 2019] sd 16:0:3:3: [sde] tag#0 Sense Key : Unit Attention [current] [Tue Nov 5 19:22:12 2019] sd 16:0:3:3: [sde] tag#0 Add. Sense: I_T nexus loss occurred [Tue Nov 5 19:22:12 2019] sd 16:0:3:3: [sde] tag#0 CDB: Write(10) 2a 00 11 a8 01 4f 00 00 08 00 [Tue Nov 5 19:22:12 2019] blk_update_request: I/O error, dev sde, sector 2962230 55 [Tue Nov 5 19:22:12 2019] sd 16:0:3:3: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical [Tue Nov 5 19:22:12 2019] sd 16:0:3:3: timing out command, waited 180s [Tue Nov 5 19:22:12 2019] sd 16:0:3:3: [sde] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [Tue Nov 5 19:22:12 2019] sd 16:0:3:3: [sde] tag#1 Sense Key : Unit Attention [current] [Tue Nov 5 19:22:12 2019] sd 16:0:3:3: [sde] tag#1 Add. Sense: Reported luns data has changed [Tue Nov 5 19:22:13 2019] sd 16:0:3:3: [sde] tag#1 CDB: Write(10) 2a 00 09 47 6c 97 00 00 10 00 [Tue Nov 5 19:22:13 2019] blk_update_request: I/O error, dev sde, sector 155675799 [Tue Nov 5 19:22:13 2019] sd 16:0:2:1: timing out command, waited 180s [Tue Nov 5 19:22:13 2019] sd 16:0:2:1: [sdb] tag#7 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [Tue Nov 5 19:22:13 2019] sd 16:0:2:1: [sdb] tag#7 Sense Key : Unit Attention [current] [Tue Nov 5 19 :22:13 2019] sd 16:0:2:1: [sdb] tag#7 Add. Sense: I_T nexus loss occurred [Tue Nov 5 19:22:13 2019] sd 16:0:2:1: [sdb] tag#7 CDB: Read(10) 28 00 bd 24 ae bf 00 10 00 00 [Tue Nov 5 19:22:13 2019] blk_update_request: I/O error, dev sdb, sector 3173297855 [Tue Nov 5 19:22:13 2019] sd 16:0:2:1: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical [Tue Nov 5 19:22:13 2019] sd 16:0:2:1: timing out command, waited 180s [Tue Nov 5 19:22:13 2019] sd 16:0:2:1: [sdb] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [Tue Nov 5 19:22:13 2019] sd 16:0:2:1: [sdb] tag#12 Sense Key : Unit Attention [current] [Tue Nov 5 19:22:13 2019] sd 16:0:2:1: [sdb] tag#12 Add. Sense: Reported luns data has changed [Tue Nov 5 19:22:13 2019] sd 16:0:2:1: [sdb] tag#12 CDB: Write(10) 2a 00 7c c5 fb df 00 10 00 00 [Tue Nov 5 19:22:13 2019] blk_update_request: I/O error, dev sdb, sector 2093349855 [Tue Nov 5 19:22:13 2019 ] device-mapper: multipath: Failing path 8:64. [Tue Nov 5 19:22:13 2019] device-mapper: multipath: Failing path 8:16. [Tue Nov 5 19:22:13 2019] device-mapper: multipath: Reinstating path 8:16. [Tue Nov 5 19:22:13 2019] device-mapper: multipath: Reinstating path 8:64. [Tue Nov 5 19:33:49 2019] qla2xxx [0000:3b:00.1]-801c:16: Abort command issued nexus=16:3:3 -- 0 2003. [Tue Nov 5 19:33:49 2019] qla2xxx [0000:3b:00.1]-8009:16: DEVICE RESET ISSUED nexus=16:3:3 cmd=ffff94f4f66d3b80. [Tue Nov 5 19:33:49 2019] qla2xxx [0000:3b:00.1]-800e:16: DEVICE RESET SUCCEEDED nexus:16:3:3 cmd=ffff94f4f66d3b80. [Tue Nov 5 19:33:49 2019] sd 16:0:3:3: alua: port group f040 state A non-preferred supports toluSNA [Tue Nov 5 19:33:49 2019] sd 15:0:2:3: alua: port group f03e state A non-preferred supports toluSNA Any thoughts on what might be wrong, and how we might fix it? With thanks, Ben _______________________________________________ Linux-PowerEdge mailing list [email protected] https://lists.us.dell.com/mailman/listinfo/linux-poweredge _______________________________________________ Linux-PowerEdge mailing list [email protected] https://lists.us.dell.com/mailman/listinfo/linux-poweredge
