At the time in which a storage controller is failed, /var/log/syslog and
journalctl look identical:
Apr 7 11:45:28 ICTM1608S01H1 kernel: [586649.657080] lpfc 0000:af:00.1:
5:(0):6172 NVME rescanned DID x3d0a00 port_state x2
Apr 7 11:45:28 ICTM1608S01H1 kernel: [586649.657268] lpfc 0000:18:00.1:
1:(0):6172 NVME rescanned DID x3d0a00 port_state x2
Apr 7 11:45:28 ICTM1608S01H1 kernel: [586649.658064] nvme nvme5: NVME-FC{4}:
controller connectivity lost. Awaiting Reconnect
Apr 7 11:45:28 ICTM1608S01H1 kernel: [586649.659036] nvme nvme1: NVME-FC{0}:
controller connectivity lost. Awaiting Reconnect
Apr 7 11:45:28 ICTM1608S01H1 systemd-udevd[2895178]: fc_udev_device: Process
'systemctl --no-block start
nvmf-connect@--device=none\t--transport=fc\t--traddr=nn-0x200200a098d8580e:pn-0x202200a098d8580e\t--trsvcid=none\t--host-traddr=nn-0x20000090fadcc5ce:pn-0x10000090fadcc5ce.service'
failed with exit code 1.
Apr 7 11:45:28 ICTM1608S01H1 systemd-udevd[2895178]: fc_udev_device: Process
'systemctl --no-block start
nvmf-connect@--device=none\t--transport=fc\t--traddr=nn-0x200200a098d8580e:pn-0x202200a098d8580e\t--trsvcid=none\t--host-traddr=nn-0x200000109b8f2b8e:pn-0x100000109b8f2b8e.service'
failed with exit code 1.
Apr 7 11:45:28 ICTM1608S01H1 kernel: [586649.680671] nvme nvme5: NVME-FC{4}:
io failed due to lldd error 6
Apr 7 11:45:28 ICTM1608S01H1 kernel: [586649.703918] nvme nvme1: NVME-FC{0}:
io failed due to lldd error 6
Apr 7 11:45:29 ICTM1608S01H1 kernel: [586650.469693] lpfc 0000:af:00.0:
4:(0):6172 NVME rescanned DID x011400 port_state x2
Apr 7 11:45:29 ICTM1608S01H1 kernel: [586650.469715] lpfc 0000:18:00.0:
0:(0):6172 NVME rescanned DID x011400 port_state x2
Apr 7 11:45:29 ICTM1608S01H1 kernel: [586650.470629] nvme nvme4: NVME-FC{1}:
controller connectivity lost. Awaiting Reconnect
Apr 7 11:45:29 ICTM1608S01H1 kernel: [586650.471611] nvme nvme8: NVME-FC{5}:
controller connectivity lost. Awaiting Reconnect
Apr 7 11:45:29 ICTM1608S01H1 systemd-udevd[2895178]: fc_udev_device: Process
'systemctl --no-block start
nvmf-connect@--device=none\t--transport=fc\t--traddr=nn-0x200200a098d8580e:pn-0x201200a098d8580e\t--trsvcid=none\t--host-traddr=nn-0x20000090fadcc5cd:pn-0x10000090fadcc5cd.service'
failed with exit code 1.
Apr 7 11:45:29 ICTM1608S01H1 systemd-udevd[2895178]: fc_udev_device: Process
'systemctl --no-block start
nvmf-connect@--device=none\t--transport=fc\t--traddr=nn-0x200200a098d8580e:pn-0x201200a098d8580e\t--trsvcid=none\t--host-traddr=nn-0x200000109b8f2b8d:pn-0x100000109b8f2b8d.service'
failed with exit code 1.
Apr 7 11:45:29 ICTM1608S01H1 kernel: [586650.493222] nvme nvme4: NVME-FC{1}:
io failed due to lldd error 6
Apr 7 11:45:29 ICTM1608S01H1 kernel: [586650.516848] nvme nvme8: NVME-FC{5}:
io failed due to lldd error 6
Apr 7 11:45:59 ICTM1608S01H1 kernel: [586680.663369] rport-10:0-9: blocked FC
remote port time out: removing rport
Apr 7 11:45:59 ICTM1608S01H1 kernel: [586680.663373] rport-16:0-9: blocked FC
remote port time out: removing rport
Apr 7 11:45:59 ICTM1608S01H1 kernel: [586680.663377] rport-15:0-9: blocked FC
remote port time out: removing rport
Apr 7 11:45:59 ICTM1608S01H1 kernel: [586680.663383] rport-12:0-9: blocked FC
remote port time out: removing rport
Apr 7 11:46:28 ICTM1608S01H1 kernel: [586709.847350] nvme nvme5: NVME-FC{4}:
dev_loss_tmo (60) expired while waiting for remoteport connectivity.
Apr 7 11:46:28 ICTM1608S01H1 kernel: [586709.847363] nvme nvme5: Removing
ctrl: NQN "nqn.1992-08.com.netapp:5700.600a098000d8580e000000005c0136a2"
Apr 7 11:46:28 ICTM1608S01H1 kernel: [586709.847385] nvme nvme1: NVME-FC{0}:
dev_loss_tmo (60) expired while waiting for remoteport connectivity.
Apr 7 11:46:28 ICTM1608S01H1 kernel: [586709.847395] nvme nvme1: Removing
ctrl: NQN "nqn.1992-08.com.netapp:5700.600a098000d8580e000000005c0136a2"
Apr 7 11:46:29 ICTM1608S01H1 kernel: [586710.615343] nvme nvme4: NVME-FC{1}:
dev_loss_tmo (60) expired while waiting for remoteport connectivity.
Apr 7 11:46:29 ICTM1608S01H1 kernel: [586710.615357] nvme nvme4: Removing
ctrl: NQN "nqn.1992-08.com.netapp:5700.600a098000d8580e000000005c0136a2"
Apr 7 11:46:29 ICTM1608S01H1 kernel: [586710.615375] nvme nvme8: NVME-FC{5}:
dev_loss_tmo (60) expired while waiting for remoteport connectivity.
Apr 7 11:46:29 ICTM1608S01H1 kernel: [586710.615389] nvme nvme8: Removing
ctrl: NQN "nqn.1992-08.com.netapp:5700.600a098000d8580e000000005c0136a2"
Apr 7 11:47:07 ICTM1608S01H1 systemd-udevd[2896874]: fc_udev_device: Process
'systemctl --no-block start
nvmf-connect@--device=none\t--transport=fc\t--traddr=nn-0x200200a098d8580e:pn-0x201200a098d8580e\t--trsvcid=none\t--host-traddr=nn-0x20000090fadcc5cd:pn-0x10000090fadcc5cd.service'
failed with exit code 1.
Apr 7 11:47:07 ICTM1608S01H1 systemd-udevd[2896874]: fc_udev_device: Process
'systemctl --no-block start
nvmf-connect@--device=none\t--transport=fc\t--traddr=nn-0x200200a098d8580e:pn-0x201200a098d8580e\t--trsvcid=none\t--host-traddr=nn-0x200000109b8f2b8d:pn-0x100000109b8f2b8d.service'
failed with exit code 1.
Apr 7 11:47:08 ICTM1608S01H1 systemd-udevd[2896872]: fc_udev_device: Process
'systemctl --no-block start
nvmf-connect@--device=none\t--transport=fc\t--traddr=nn-0x200200a098d8580e:pn-0x202200a098d8580e\t--trsvcid=none\t--host-traddr=nn-0x20000090fadcc5ce:pn-0x10000090fadcc5ce.service'
failed with exit code 1.
Apr 7 11:47:08 ICTM1608S01H1 systemd-udevd[2896874]: fc_udev_device: Process
'systemctl --no-block start
nvmf-connect@--device=none\t--transport=fc\t--traddr=nn-0x200200a098d8580e:pn-0x202200a098d8580e\t--trsvcid=none\t--host-traddr=nn-0x200000109b8f2b8e:pn-0x100000109b8f2b8e.service'
failed with exit code 1.
I've attached /var/log/syslog and the entire output of journalctl. Is
there something in particular that would help indicate why the
subsystems aren't rediscovered?
** Attachment added: "syslog-journalctl"
https://bugs.launchpad.net/ubuntu/+source/nvme-cli/+bug/1874270/+attachment/5485243/+files/syslog-journalctl.zip
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1874270
Title:
NVMe/FC connections fail to reestablish after controller is reset
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nvme-cli/+bug/1874270/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
