Anyone facing this.. Could you please make sure to propagate I/O errors properly to filesystem in order for it to call its automatically shutdown procedure. After that, the mount won't reference the super block in kernel and that will allow the device mapper to be closed/destroyed (this can be seen by checking existence of device-mapper node in /sys/fs/{xfs,ext3,ext4}/node, saying that the super block is still being referenced).
I had a recent case where not propagating the error would cause the filesystem to hang forever, causing lockups and not allowing the device- mapper tables to be flushed by multipath. In this case, if the network was turned off before the iscsi was logged, the multipath (dm) and iscsi (transport) layers HAVE TO propagate the I/O error to the filesystem for it to shutdown. This is can be done by: node.session.timeo.replacement_timeout = 0 (iscsid.conf) USE: iscsiadm -m node -T $target_name -p $target_ip:$port -o update -n \ node.session.timeo.replacement_timeout -v $timeout_value on already logged/discovered LUNs. This parameter might be persistent on already discovered LUNs. AND: dev_loss_tmo 10 (multipath.conf) OBS: node.session.timeo.replacement_timeout has to be changed per The fact that the network was removed BEFORE the iscsi logout could happen is NOT an issue if ROOT device is not ISCSI, you just have to allow the block device <-> device mapper <-> transport layer logic shutdown to work. Pay attention because if filesystem is umounted and iscsi is gone, the timeout will have to be waited before the error is propagated to other layer. So if filesystem is umounted a last superblock inode update will be attempted (if no cache in pagecache). This will make the I/O error for the superblock inode to fail (after time timeout you configured). This will make the filesystem to be shutdown (a half unmounted state). After shutdown, the logic trying to umount (synchronously) the filesystem will continue (possibly in systemd). If the problem is because of you tried to flush the device mapper and the filesystem was still referencing the device mapper super block, this will fade away if the error is propagated and the filesystem is shutdown, you can attempt doing the flush of device-mapper right after filesystem shutdown happened. Could you please try setting the timeouts properly for lower timings and let me know if that mitigated the issue ? Thank you -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets Status in systemd package in Ubuntu: Confirmed Status in systemd source package in Xenial: Confirmed Bug description: I have 4 servers running the latest 16.04 updates from the development branch (as of right now). Each server is connected to NetApp storage using iscsi software initiator. There are a total of 56 volumes spread across two NetApp arrays. Each volume has 4 paths available to it which are being managed by device mapper. While logged into the iscsi sessions all I have to do is reboot the server and I get a hang. I see a message that says: "Reached target Shutdown" followed by "systemd-shutdown[1]: Failed to finalize DM devices, ignoring" and then I see 8 lines that say: "connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection5:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection6:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection7:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection8:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" NOTE: the actual values of the *'s differ for each line above. This seems like a bug somewhere but I am unaware of any additional logging that I could turn on to pinpoint the problem. Note I also have similar setups that are not doing iscsi and they don't have this problem. Here is a screenshot of what I see on the shell when I try to reboot: (https://launchpadlibrarian.net/291303059/Screenshot.jpg) This is being tracked in NetApp bug tracker CQ number 860251. If I log out of all iscsi sessions before rebooting then I do not experience the hang: iscsiadm -m node -U all We are wondering if this could be some kind of shutdown ordering problem. Like the network devices have already disappeared and then iscsi tries to perform some operation (hence the ping timeouts). To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp