I'm failing trusty's patch because: 1) It was not backported fully, missing the debug message at its bottom. That led me to initially think, when debugging, that the patch hasn't been included at all. Later I could check kernel tree and saw the it was manually merged, missing that debug message part.
2) Since functional piece of code was backported, it should, still, have worked, and it did not. Something in trusty is not making it to work like expected. I'm still investigating it through a kdump taken during a shutdown hang on trusty, but, I don't want to hold the kernel release because of that: This is the output for my tests: https://pastebin.ubuntu.com/p/B9fp5y5gqK/ And you can check (1) with: [ 144.862511] session4: iscsi_eh_cmd_timed_out return nh It should be "or shutdown" there, together. Differently than other tests, in trusty's kernel (3.13) you can see: [ 144.860010] session4: iscsi_eh_cmd_timed_out scsi cmd ffff880037a9a100 timedout [ 144.861400] session4: iscsi_eh_cmd_timed_out sc on shutdown, handled [ 144.862511] session4: iscsi_eh_cmd_timed_out return nh [ 144.863483] session4: iscsi_queuecommand iscsi: cmd 0x3 is not queued (2) [ 144.864794] session4: iscsi_eh_device_reset LU Reset [sc ffff880037a9a100 lun 1] [ 144.865907] session4: iscsi_eh_device_reset dev reset result = FAILED [ 144.866875] session4: iscsi_eh_target_reset tgt Reset [sc ffff880037a9a100 tgt iqn.2017.tgtd:tid5.lun] [ 144.868343] session4: iscsi_eh_target_reset tgt iqn.2017.tgtd:tid5.lun reset result = FAILED [ 144.869580] session4: iscsi_eh_session_reset wait for relogin The iscsi transport layer, did not queue the other commands, after the first one timed out, but instead of warning the upper layer, it proceeded with a "device reset", causing the "relogin" logic to be stuck since the transport layer was already gone. This is the hang backtrace: crash> bt PID: 5980 TASK: ffff880057e19800 CPU: 0 COMMAND: "halt" #0 [ffff880037b53a50] __schedule at ffffffff8173af59 #1 [ffff880037b53ab8] schedule at ffffffff8173b3e9 #2 [ffff880037b53ac8] schedule_timeout at ffffffff8173a5d5 #3 [ffff880037b53b70] io_schedule_timeout at ffffffff8173bacb #4 [ffff880037b53ba0] wait_for_completion_io_timeout at ffffffff8173c1e6 #5 [ffff880037b53bf8] blk_execute_rq at ffffffff8134b9fb #6 [ffff880037b53ca8] scsi_execute at ffffffff814f1a77 #7 [ffff880037b53cf0] scsi_execute_req_flags at ffffffff814f2cfc #8 [ffff880037b53d58] sd_sync_cache at ffffffff81500626 #9 [ffff880037b53dd0] sd_shutdown at ffffffff81500bb9 #10 [ffff880037b53df0] device_shutdown at ffffffff814a4495 #11 [ffff880037b53e20] kernel_power_off at ffffffff81096b75 #12 [ffff880037b53e30] SYSC_reboot at ffffffff81096d4b #13 [ffff880037b53f70] sys_reboot at ffffffff81096ebe #14 [ffff880037b53f80] system_call_fastpath at ffffffff81748170 RIP: 00007ff48aa45bc6 RSP: 00007ffef9bc4be8 RFLAGS: 00010246 RAX: 00000000000000a9 RBX: ffffffff81748170 RCX: 000000000000001e RDX: 000000004321fedc RSI: 0000000028121969 RDI: fffffffffee1dead RBP: 0000000000000000 R8: fefefefefefefeff R9: 0000000000000000 R10: 00007ff48ad13c8c R11: 0000000000000202 R12: ffffffff81096ebe R13: ffff880037b53f78 R14: 0000000000000000 R15: 0000000000000000 ORIG_RAX: 00000000000000a9 CS: 0033 SS: 002b Taken by a watchdog timeout on a hang kvm system that had network interface shutdown before the transport layer could log out. ** Tags removed: verification-needed-trusty ** Tags added: verification-failed-trusty -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets Status in linux package in Ubuntu: Fix Released Status in open-iscsi package in Ubuntu: Opinion Status in linux source package in Trusty: Fix Committed Status in open-iscsi source package in Trusty: Opinion Status in linux source package in Xenial: Fix Released Status in open-iscsi source package in Xenial: Opinion Status in linux source package in Artful: Fix Released Status in open-iscsi source package in Artful: Opinion Status in linux source package in Bionic: Fix Released Status in open-iscsi source package in Bionic: Opinion Bug description: [Impact] * open-iscsi users might face hangs during OS shutdown. * hangs can be caused by manual iscsi configuration/setup. * hangs can also be caused by bad systemd unit ordering. * if transport layer interface vanishes before lun is disconnected, then the hang will happen. * check comment #89 for the fix decision. [Test Case] * a simple way of reproducing the kernel hang is to disable the open-iscsi logouts. this simulates a situation when a service has shutdown the network interface, used by the transport layer, before proper iscsi logout was done. $ log into all iscsi luns $ systemctl edit --full open-iscsi.service ... #ExecStop=/lib/open-iscsi/logout-all.sh ... $ sudo reboot # this will make server to hang forever # on shutdown [Regression Potential] * the regression is low because the change acts on the iscsi transport layer code ONLY when the server is in shutdown state. * any error in logic would only appear during shutdown and would not cause any harm to data. [Other Info] * ORIGINAL BUG DESCRIPTION I have 4 servers running the latest 16.04 updates from the development branch (as of right now). Each server is connected to NetApp storage using iscsi software initiator. There are a total of 56 volumes spread across two NetApp arrays. Each volume has 4 paths available to it which are being managed by device mapper. While logged into the iscsi sessions all I have to do is reboot the server and I get a hang. I see a message that says: "Reached target Shutdown" followed by "systemd-shutdown[1]: Failed to finalize DM devices, ignoring" and then I see 8 lines that say: "connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection5:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection6:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection7:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection8:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" NOTE: the actual values of the *'s differ for each line above. This seems like a bug somewhere but I am unaware of any additional logging that I could turn on to pinpoint the problem. Note I also have similar setups that are not doing iscsi and they don't have this problem. Here is a screenshot of what I see on the shell when I try to reboot: (https://launchpadlibrarian.net/291303059/Screenshot.jpg) This is being tracked in NetApp bug tracker CQ number 860251. If I log out of all iscsi sessions before rebooting then I do not experience the hang: iscsiadm -m node -U all We are wondering if this could be some kind of shutdown ordering problem. Like the network devices have already disappeared and then iscsi tries to perform some operation (hence the ping timeouts). To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
