[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug was fixed in the package linux - 3.13.0-144.193 --- linux (3.13.0-144.193) trusty; urgency=medium * linux: 3.13.0-144.193 -proposed tracker (LP: #1755227) * CVE-2017-12762 - isdn/i4l: fix buffer overflow * CVE-2017-17807 - KEYS: add missing permission check

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug was fixed in the package linux - 3.13.0-144.193 --- linux (3.13.0-144.193) trusty; urgency=medium * linux: 3.13.0-144.193 -proposed tracker (LP: #1755227) * CVE-2017-12762 - isdn/i4l: fix buffer overflow * CVE-2017-17807 - KEYS: add missing permission check

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

I'm failing trusty's patch because: 1) It was not backported fully, missing the debug message at its bottom. That led me to initially think, when debugging, that the patch hasn't been included at all. Later I could check kernel tree and saw the it was manually merged, missing that debug message

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

I'm verifying for trusty and its taking sometime because looks like system still hangs on shutdown. I have generated a kdump (from a watchdog timer) in the exact moment trusty is hanging on shutdown to see why BUT crash can't handle a kdump from this new kernel and I still don't know why. Testing

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- trusty' to 'verification-done-trusty'. If the problem still exists, change the tag

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug was fixed in the package linux - 4.15.0-10.11 --- linux (4.15.0-10.11) bionic; urgency=medium * linux: 4.15.0-10.11 -proposed tracker (LP: #1749250) * "swiotlb: coherent allocation failed" dmesg spam with linux 4.15.0-9.10 (LP: #1749202) - swiotlb: suppress

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug was fixed in the package linux - 4.4.0-116.140 --- linux (4.4.0-116.140) xenial; urgency=medium * linux: 4.4.0-116.140 -proposed tracker (LP: #1748990) * BUG: unable to handle kernel NULL pointer dereference at 0009 (LP: #1748671) - SAUCE: net: ipv4:

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug was fixed in the package linux - 4.13.0-36.40 --- linux (4.13.0-36.40) artful; urgency=medium * linux: 4.13.0-36.40 -proposed tracker (LP: #1750010) * Rebuild without "CVE-2017-5754 ARM64 KPTI fixes" patch set linux (4.13.0-35.39) artful; urgency=medium * linux:

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug was fixed in the package linux - 4.13.0-36.40 --- linux (4.13.0-36.40) artful; urgency=medium * linux: 4.13.0-36.40 -proposed tracker (LP: #1750010) * Rebuild without "CVE-2017-5754 ARM64 KPTI fixes" patch set linux (4.13.0-35.39) artful; urgency=medium * linux:

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug was fixed in the package linux - 4.4.0-116.140 --- linux (4.4.0-116.140) xenial; urgency=medium * linux: 4.4.0-116.140 -proposed tracker (LP: #1748990) * BUG: unable to handle kernel NULL pointer dereference at 0009 (LP: #1748671) - SAUCE: net: ipv4:

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

# xenial - 4.4.0-112 [ OK ] Reached target Shutdown. [ 70.272171] connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4294907361, last ping 4294908612, now 4294909864 [ 70.274189] connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4294907361, last

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Checking if Trusty's fix is in -proposed (no comment given from Kernel Team about it) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

# xenial - 4.4.0-112 [ OK ] Reached target Shutdown. [ 70.272171] connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4294907361, last ping 4294908612, now 4294909864 [ 70.274189] connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4294907361, last

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Hello Kleber, Thanks for the feedback. Working in verification. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets Status in linux

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- artful' to 'verification-done-artful'. If the problem still exists, change the tag

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- xenial' to 'verification-done-xenial'. If the problem still exists, change the tag

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Yep, soon kernel team will advise (here) about this patch been released in a specific kernel version that is in -proposed and will call out for testing/verification. I'll give all feedback here also and will mark, if appropriate, case as "verification-done" so new kernel is marked as solving this

Re: [Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Thanks, and will notifications appear here in the bug during these stages? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

BTW, kernel team usually updates all affecting cases with the version that was built that address this issue (so you know what to text in -proposed). -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Hello Matt, this is part of the SRU procedure (https://wiki.ubuntu.com/StableReleaseUpdates). For the kernel, the SRU is a bit different (https://wiki.ubuntu.com/Kernel/Dev/StablePatchFormat, https://wiki.ubuntu.com/Kernel/StableReleaseCadence). Now that the bug has been marked as "Fix Committed"

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

I apologize for my ignorance, but now that tall these check ins have occurred, how do I know when an installable package has been built and is available in the various repos? And thank you @inaddy for the diligence getting this one resolved. -- You received this bug notification because you are

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Changed in: linux (Ubuntu Artful) Status: In Progress => Fix Committed ** Changed in: linux (Ubuntu Xenial) Status: In Progress => Fix Committed ** Changed in: linux (Ubuntu Trusty) Status: In Progress => Fix Committed -- You received this bug notification because you

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Commit has been merged upstream: commit d754941225a7dbc61f6dd2173fa9498049f9a7ee Author: Rafael David Tinoco Date: Thu Dec 7 19:59:13 2017 -0200 scsi: libiscsi: Allow sd_shutdown on bad transport If, for any reason, userland shuts down iscsi transport

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Changed in: linux (Ubuntu Bionic) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

I have just submitted the SRU proposal to kernel team mailing list. I'll wait for the SRU to be made and this public bug to be marked as Fix Committed before verifying the fix again for the final release. -- You received this bug notification because you are a member of Kernel Packages, which is

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Changed in: open-iscsi (Ubuntu Trusty) Status: New => Opinion ** Changed in: linux (Ubuntu Trusty) Status: New => In Progress ** Changed in: linux (Ubuntu Trusty) Assignee: (unassigned) => Rafael David Tinoco (inaddy) ** Changed in: linux (Ubuntu Trusty) Importance:

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Also affects: open-iscsi (Ubuntu Bionic) Importance: Medium Assignee: Rafael David Tinoco (inaddy) Status: Opinion ** Also affects: linux (Ubuntu Bionic) Importance: Medium Assignee: Rafael David Tinoco (inaddy) Status: In Progress ** Also affects: open-iscsi

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Description changed: + [Impact] + + * open-iscsi users might face hangs during OS shutdown. + * hangs can be caused by manual iscsi configuration/setup. + * hangs can also be caused by bad systemd unit ordering. + * if transport layer interface vanishes before lun is +disconnected,

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Tags added: sts -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets Status in linux package in Ubuntu: In Progress Status in

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

The scsi/iscsi team has accepted by fix proposal and merged the patch for 4.16 merge window: https://www.mail-archive.com/open-iscsi@googlegroups.com/msg10111.html I'll propose the cherry-pick as SRU for Ubuntu kernels as soon as it is merged. -- You received this bug notification because you

Re: [Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

That's awesome sir, thank you for your diligence. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets Status in linux package in

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

I have proposed a fix to the kernel iscsi transport code. You can follow upstream discussion, if any, in the following address: https://marc.info/?t=15126839768=1=2 The patch proposal is here: https://marc.info/?l=linux-scsi=151268396227285=2 If upstream chooses to accept my proposal,

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Long story short -> https://goo.gl/vPyh8C Basically the block device enqueues the last request (a SYNC scsi command coming from sd_shutdown) for every scsi device there is on the system. Unfortunately, since the OS is shutting down, in between the block request and its execution, we have userland

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Hello Amanda, I'm actively working into this, will provide better feedback soon. The script is just a workaround for now, so people can shut down properly, if they are facing this. Despite what userland does, the kernel will always potentially hang with iscsi sessions left opened during

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

I followed all of the setup steps, but was still encountering the ping timeout issue at reboot. I then ran the script http://pastebin.ubuntu.com/25894592/ you provided and am no longer hitting the issue. Was the script supposed to fix the issue? Please let me know. Thank you! -- You received

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Hello Matthijs Unfortunately the best way to make this not to happen is by fixing the kernel hang situation, when kernel calls sd_sync_cache() to every configured device before the shutdown. There is a single I/O cmd hanging in all scsi paths and the I/O error is never propagated to block layer

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Even with the script a shutdown/reboot takes a very long time. Any expectations when a fix will be ready? I hit this all the time on 2 installed servers. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Tags added: id-598b6543459f8ccf5dfbc04c -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets Status in linux package in Ubuntu:

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

I created a workaround for this issue while its being worked on: http://pastebin.ubuntu.com/25519820/ Create a file: /lib/systemd/system-shutdown/debug.sh and place the contents in it. This workaround (based on some ideas from sil2100 and slangasek for iscsi / umount) is basically bringing the

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Tags added: rls-aa-notfixing -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets Status in linux package in Ubuntu: In Progress

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Changed in: open-iscsi (Ubuntu Xenial) Assignee: (unassigned) => Rafael David Tinoco (inaddy) ** Changed in: open-iscsi (Ubuntu Zesty) Assignee: (unassigned) => Rafael David Tinoco (inaddy) ** Changed in: open-iscsi (Ubuntu Artful) Assignee: (unassigned) => Rafael David Tinoco

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

2 things for previous comment #75: 1) I pasted wrong pastebin URL. I'll describe the fix: its a service unit file that runs at the end of shutdown, right before it, re-using /run (tmpfs). Service has a preparation script that copies binaries from initrd image - so there is a minimum execution

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Unrelated to systemd. Related to open-iscsi systemd unit files and kernel iscsi transport logout during shutdown. ** Changed in: systemd (Ubuntu Artful) Importance: High => Medium ** Changed in: systemd (Ubuntu Artful) Status: Confirmed => Triaged ** Changed in: systemd (Ubuntu

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Unrelated to systemd. Related to open-iscsi systemd unit files and kernel iscsi transport logout during shutdown. ** Changed in: systemd (Ubuntu Zesty) Importance: Undecided => Medium ** Changed in: systemd (Ubuntu Zesty) Status: New => Triaged -- You received this bug notification

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Unrelated to systemd. Related to open-iscsi systemd unit files and kernel iscsi transport logout during shutdown. ** Changed in: open-iscsi (Ubuntu Xenial) Importance: Undecided => Medium ** Changed in: open-iscsi (Ubuntu Xenial) Status: New => In Progress ** Changed in: open-iscsi

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

## SUMMARY 1 Affects open-iscsi (NOT systemd): * ISCSID NEEDS TO BE UP & RUNNING (AND LOGGED TO PORTAL) FOR THE LOGOUT TO WORK https://pastebin.canonical.com/197359/ Daemon shutdown documentation in iscsiadm command says (about stopping iscsid daemon with iscsiadm -k): """ ... This will

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Also affects: open-iscsi (Ubuntu Artful) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Artful) Importance: Undecided Status: New ** Also affects: systemd (Ubuntu Artful) Importance: High Assignee: Nish Aravamudan (nacc) Status: Confirmed

[Kernel-packages] [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Also affects: open-iscsi (Ubuntu) Importance: Undecided Status: New ** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.