[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug was fixed in the package linux - 3.13.0-144.193 --- linux (3.13.0-144.193) trusty; urgency=medium * linux: 3.13.0-144.193 -proposed tracker (LP: #1755227) * CVE-2017-12762 - isdn/i4l: fix buffer overflow * CVE-2017-17807 - KEYS: add missing permission check

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug was fixed in the package linux - 3.13.0-144.193 --- linux (3.13.0-144.193) trusty; urgency=medium * linux: 3.13.0-144.193 -proposed tracker (LP: #1755227) * CVE-2017-12762 - isdn/i4l: fix buffer overflow * CVE-2017-17807 - KEYS: add missing permission check

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

I'm failing trusty's patch because: 1) It was not backported fully, missing the debug message at its bottom. That led me to initially think, when debugging, that the patch hasn't been included at all. Later I could check kernel tree and saw the it was manually merged, missing that debug message

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

I'm verifying for trusty and its taking sometime because looks like system still hangs on shutdown. I have generated a kdump (from a watchdog timer) in the exact moment trusty is hanging on shutdown to see why BUT crash can't handle a kdump from this new kernel and I still don't know why. Testing

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- trusty' to 'verification-done-trusty'. If the problem still exists, change the tag

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug was fixed in the package linux - 4.15.0-10.11 --- linux (4.15.0-10.11) bionic; urgency=medium * linux: 4.15.0-10.11 -proposed tracker (LP: #1749250) * "swiotlb: coherent allocation failed" dmesg spam with linux 4.15.0-9.10 (LP: #1749202) - swiotlb: suppress

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug was fixed in the package linux - 4.4.0-116.140 --- linux (4.4.0-116.140) xenial; urgency=medium * linux: 4.4.0-116.140 -proposed tracker (LP: #1748990) * BUG: unable to handle kernel NULL pointer dereference at 0009 (LP: #1748671) - SAUCE: net: ipv4:

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug was fixed in the package linux - 4.4.0-116.140 --- linux (4.4.0-116.140) xenial; urgency=medium * linux: 4.4.0-116.140 -proposed tracker (LP: #1748990) * BUG: unable to handle kernel NULL pointer dereference at 0009 (LP: #1748671) - SAUCE: net: ipv4:

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug was fixed in the package linux - 4.13.0-36.40 --- linux (4.13.0-36.40) artful; urgency=medium * linux: 4.13.0-36.40 -proposed tracker (LP: #1750010) * Rebuild without "CVE-2017-5754 ARM64 KPTI fixes" patch set linux (4.13.0-35.39) artful; urgency=medium * linux:

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug was fixed in the package linux - 4.13.0-36.40 --- linux (4.13.0-36.40) artful; urgency=medium * linux: 4.13.0-36.40 -proposed tracker (LP: #1750010) * Rebuild without "CVE-2017-5754 ARM64 KPTI fixes" patch set linux (4.13.0-35.39) artful; urgency=medium * linux:

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Checking if Trusty's fix is in -proposed (no comment given from Kernel Team about it) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets To

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

# xenial - 4.4.0-112 [ OK ] Reached target Shutdown. [ 70.272171] connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4294907361, last ping 4294908612, now 4294909864 [ 70.274189] connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4294907361, last

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

# xenial - 4.4.0-112 [ OK ] Reached target Shutdown. [ 70.272171] connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4294907361, last ping 4294908612, now 4294909864 [ 70.274189] connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4294907361, last

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Hello Kleber, Thanks for the feedback. Working in verification. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets To manage notifications

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- xenial' to 'verification-done-xenial'. If the problem still exists, change the tag

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- artful' to 'verification-done-artful'. If the problem still exists, change the tag

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Yep, soon kernel team will advise (here) about this patch been released in a specific kernel version that is in -proposed and will call out for testing/verification. I'll give all feedback here also and will mark, if appropriate, case as "verification-done" so new kernel is marked as solving this

RE: [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Thanks, and will notifications appear here in the bug during these stages? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets To manage

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Hello Matt, this is part of the SRU procedure (https://wiki.ubuntu.com/StableReleaseUpdates). For the kernel, the SRU is a bit different (https://wiki.ubuntu.com/Kernel/Dev/StablePatchFormat, https://wiki.ubuntu.com/Kernel/StableReleaseCadence). Now that the bug has been marked as "Fix Committed"

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

BTW, kernel team usually updates all affecting cases with the version that was built that address this issue (so you know what to text in -proposed). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

I apologize for my ignorance, but now that tall these check ins have occurred, how do I know when an installable package has been built and is available in the various repos? And thank you @inaddy for the diligence getting this one resolved. -- You received this bug notification because you are

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Changed in: linux (Ubuntu Artful) Status: In Progress => Fix Committed ** Changed in: linux (Ubuntu Xenial) Status: In Progress => Fix Committed ** Changed in: linux (Ubuntu Trusty) Status: In Progress => Fix Committed -- You received this bug notification because you

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Commit has been merged upstream: commit d754941225a7dbc61f6dd2173fa9498049f9a7ee Author: Rafael David Tinoco Date: Thu Dec 7 19:59:13 2017 -0200 scsi: libiscsi: Allow sd_shutdown on bad transport If, for any reason, userland shuts down iscsi transport

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Changed in: linux (Ubuntu Bionic) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets To manage

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

I have just submitted the SRU proposal to kernel team mailing list. I'll wait for the SRU to be made and this public bug to be marked as Fix Committed before verifying the fix again for the final release. -- You received this bug notification because you are a member of Ubuntu Bugs, which is

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Changed in: open-iscsi (Ubuntu Trusty) Status: New => Opinion ** Changed in: linux (Ubuntu Trusty) Status: New => In Progress ** Changed in: linux (Ubuntu Trusty) Assignee: (unassigned) => Rafael David Tinoco (inaddy) ** Changed in: linux (Ubuntu Trusty) Importance:

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Also affects: open-iscsi (Ubuntu Bionic) Importance: Medium Assignee: Rafael David Tinoco (inaddy) Status: Opinion ** Also affects: linux (Ubuntu Bionic) Importance: Medium Assignee: Rafael David Tinoco (inaddy) Status: In Progress ** Also affects: open-iscsi

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Description changed: + [Impact] + + * open-iscsi users might face hangs during OS shutdown. + * hangs can be caused by manual iscsi configuration/setup. + * hangs can also be caused by bad systemd unit ordering. + * if transport layer interface vanishes before lun is +disconnected,

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Changed open-iscsi to opinion since I choose to fix the kernel instead of fixing Userland. No matter what you do in userland, the kernel had space for freezing and hanging in different scenarios depending on how userland disconnected the transport layer. I kept the "linux" source package as being

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Tags added: sts -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets To manage notifications about this bug go to:

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

The scsi/iscsi team has accepted by fix proposal and merged the patch for 4.16 merge window: https://www.mail-archive.com/open-iscsi@googlegroups.com/msg10111.html I'll propose the cherry-pick as SRU for Ubuntu kernels as soon as it is merged. -- You received this bug notification because you

RE: [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

That's awesome sir, thank you for your diligence. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets To manage notifications about this bug go

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

I have proposed a fix to the kernel iscsi transport code. You can follow upstream discussion, if any, in the following address: https://marc.info/?t=15126839768=1=2 The patch proposal is here: https://marc.info/?l=linux-scsi=151268396227285=2 If upstream chooses to accept my proposal,

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Long story short -> https://goo.gl/vPyh8C Basically the block device enqueues the last request (a SYNC scsi command coming from sd_shutdown) for every scsi device there is on the system. Unfortunately, since the OS is shutting down, in between the block request and its execution, we have userland

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Hello Amanda, I'm actively working into this, will provide better feedback soon. The script is just a workaround for now, so people can shut down properly, if they are facing this. Despite what userland does, the kernel will always potentially hang with iscsi sessions left opened during

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

I followed all of the setup steps, but was still encountering the ping timeout issue at reboot. I then ran the script http://pastebin.ubuntu.com/25894592/ you provided and am no longer hitting the issue. Was the script supposed to fix the issue? Please let me know. Thank you! -- You received

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Hello Matthijs Unfortunately the best way to make this not to happen is by fixing the kernel hang situation, when kernel calls sd_sync_cache() to every configured device before the shutdown. There is a single I/O cmd hanging in all scsi paths and the I/O error is never propagated to block layer

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Even with the script a shutdown/reboot takes a very long time. Any expectations when a fix will be ready? I hit this all the time on 2 installed servers. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Tags added: id-598b6543459f8ccf5dfbc04c -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets To manage notifications about this bug go to:

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

I created a workaround for this issue while its being worked on: http://pastebin.ubuntu.com/25519820/ Create a file: /lib/systemd/system-shutdown/debug.sh and place the contents in it. This workaround (based on some ideas from sil2100 and slangasek for iscsi / umount) is basically bringing the

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Tags added: rls-aa-notfixing -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets To manage notifications about this bug go to:

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Changed in: open-iscsi (Ubuntu Xenial) Assignee: (unassigned) => Rafael David Tinoco (inaddy) ** Changed in: open-iscsi (Ubuntu Zesty) Assignee: (unassigned) => Rafael David Tinoco (inaddy) ** Changed in: open-iscsi (Ubuntu Artful) Assignee: (unassigned) => Rafael David Tinoco

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

2 things for previous comment #75: 1) I pasted wrong pastebin URL. I'll describe the fix: its a service unit file that runs at the end of shutdown, right before it, re-using /run (tmpfs). Service has a preparation script that copies binaries from initrd image - so there is a minimum execution

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Unrelated to systemd. Related to open-iscsi systemd unit files and kernel iscsi transport logout during shutdown. ** Changed in: systemd (Ubuntu Artful) Importance: High => Medium ** Changed in: systemd (Ubuntu Artful) Status: Confirmed => Triaged ** Changed in: systemd (Ubuntu

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Unrelated to systemd. Related to open-iscsi systemd unit files and kernel iscsi transport logout during shutdown. ** Changed in: open-iscsi (Ubuntu Xenial) Importance: Undecided => Medium ** Changed in: open-iscsi (Ubuntu Xenial) Status: New => In Progress ** Changed in: open-iscsi

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Unrelated to systemd. Related to open-iscsi systemd unit files and kernel iscsi transport logout during shutdown. ** Changed in: systemd (Ubuntu Zesty) Importance: Undecided => Medium ** Changed in: systemd (Ubuntu Zesty) Status: New => Triaged -- You received this bug notification

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

## SUMMARY 1 Affects open-iscsi (NOT systemd): * ISCSID NEEDS TO BE UP & RUNNING (AND LOGGED TO PORTAL) FOR THE LOGOUT TO WORK https://pastebin.canonical.com/197359/ Daemon shutdown documentation in iscsiadm command says (about stopping iscsid daemon with iscsiadm -k): """ ... This will

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Also affects: open-iscsi (Ubuntu Artful) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Artful) Importance: Undecided Status: New ** Also affects: systemd (Ubuntu Artful) Importance: High Assignee: Nish Aravamudan (nacc) Status: Confirmed

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Also affects: open-iscsi (Ubuntu) Importance: Undecided Status: New ** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Okay, I removed almost everything out of equation: - removed networking interfaces from systemd - removed open-iscsi login/logout logic from systemd - used a single network interface for the logins, on a single iscsi portal was still able to reproduce the issue by: - doing a simple login after

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

For now, the workaround for this bug is this: inaddy@iscsihang:~$ sudo su - # disconnect what iscsid has mapped root@iscsihang:~$ iscsiadm -m node --logoutall=all # disconnect all leftovers root@iscsihang:~$ for file in `find /sys/devices -name *delete`; do echo 1 > $file; done # shutdown

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Excellent, thank you for the details. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets To manage notifications about this bug go to:

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

This is what I discovered: This systemd issue is caused because there are iscsi connections still logged in (not because of multipath, bonding, vlans, etc). One way of reproducing this is by simply editing the open-iscsi.service unit and disabling the logout script. It will, for me in all times,

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

As I said I don't have one up right now, but Comment 3 above has my interfaces for one of the recreates. Nothing bothered me in any way. I wanted to make sure that you knew that I wasn't one guy who was looking to fix his one system, that's all. You said "Usually those type of "partnership"

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Might be unrelated, but I discovered that: ## root namespace root@iscsihang:~$ mount | egrep -E "sd.*(xfs|ext4)" /dev/sda1 on /ext4 type ext4 (rw,relatime,stripe=32,data=ordered,_netdev) /dev/sdb1 on /xfs type xfs (rw,relatime,attr2,inode64,noquota,_netdev) ## specific mnt namespace

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Hypothesis, Test (2) - The error is propagated to upper layers after X seconds. In this case I'm testing: $ sudo iscsiadm -m node -o show | grep node.session.timeo.replac node.session.timeo.replacement_timeout = 60 node.session.timeo.replacement_timeout = 60 And same "locking" behavior is

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Hypothesis, Test (1) - The error is NEVER propagated to upper layers: # xfs and ext4 mounted automatically inaddy@iscsihang:~$ mount | grep _netde /dev/sda1 on /ext4 type ext4 (rw,relatime,stripe=32,data=ordered,_netdev) /dev/sdb1 on /xfs type xfs (rw,relatime,attr2,inode64,noquota,_netdev) #

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Please let me explain who I am and what we do to possibly shed some light on this. I am an Interoperability engineer at NetApp. We test our storage products with many popular operating systems, adapters, protocols and switches, in varying combinations. We test software initiator iscsi

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Hello Matt, Good to read your feedback. When you have multipath AND iscsi, the error propagation timeout can be given by one or both, that is why I explained both. This issue is not an iscsi or multipath problem, it is a systemd problem related to the fact that the network is removed before the

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

I have node.session.timeo.replacement_timeout = 20, which seems completely reasonable in my book. We have already established that this occurs without multipath even enabled, thus it is not a multipath settings problem. Also I have no containers. -- You received this bug notification because

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Changed in: systemd (Ubuntu Xenial) Assignee: (unassigned) => Rafael David Tinoco (inaddy) ** Changed in: systemd (Ubuntu Xenial) Status: Confirmed => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Changed in: systemd (Ubuntu Xenial) Assignee: Dimitri John Ledkov (xnox) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Another thing. Please check for containers referencing the filesystems in question. When you have _netdev filesystems, it is possible that the "mount --shared" characteristics of containers (different mount namescapes) are holding references to the filesystem in question. A good test is to test

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Anyone facing this.. Could you please make sure to propagate I/O errors properly to filesystem in order for it to call its automatically shutdown procedure. After that, the mount won't reference the super block in kernel and that will allow the device mapper to be destroyed (this can be seen by

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Anyone facing this.. Could you please make sure to propagate I/O errors properly to filesystem in order for it to call its automatically shutdown procedure. After that, the mount won't reference the super block in kernel and that will allow the device mapper to be closed/destroyed (this can be

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Is it possible to reproduce this problem with upstart or sysvinit-core? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets To manage

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

I can confirm that the daily 17.10 build does not exhibit the problem. Interestingly, the first time we tried we forgot to stop running IO to our iSCSI volumes and it _looked_ like we reproduced the problem, then the IOs timed out and our application stopped and we were able to complete the

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: systemd (Ubuntu Xenial) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

@nacc: Yes. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets To manage notifications about this bug go to:

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

@kai-holthaus: after editing the service file, did you issue a `sudo systemctl daemon-reload` ? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

@nacc: The system was originally installed with Ubuntu Server 14.04. It has been upgraded to 16.04. apt policy open-iscsi: open-iscsi: Installed: 2.0.873+git0.3b4b4500-14ubuntu3.3 Candidate: 2.0.873+git0.3b4b4500-14ubuntu3.3 Version table: *** 2.0.873+git0.3b4b4500-14ubuntu3.3 500

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

@Kai: Hrm, that is odd -- was your 16.04 system the fresh install or an upgraded one? The lines I pasted in #49 are supposed to be present in 16.04 (afaict). If they don't, can you provide `apt policy open-iscsi` in a pastebin? And can you test if changing those lines to match fixes the issue

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

@nacc: No, my lines do not like that. On my xenial system, I have: Wants=network-online.target remote-fs-pre.target After=network-online.target Which is probably why on my system the iscsid daemon shuts down before the logout. -- You received this bug notification because you are a member of

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

@nacc: Add-on: On my Debian systems (Debian jessie and stretch) that also experience this issue, however, the lines do look like what you described - and still the system hangs at reboot / shutdown. In a previous message it sounded like you had identified an issue and a fix - did I misinterpret

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

@Kai: In xenial, open-iscsi.service should have the following lines alreadY Wants=network-online.target remote-fs-pre.target iscsid.service After=network-online.target iscsid.service Can you confirm that is the case on your system? -- You received this bug notification because you are a member

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

I've installed a system with the daily artful server image, and the error is not present for me. However, I'm not sure how useful that test is, given that I didn't experience the error during a fresh install of xenial. I've attached the journal from the artful install, and it looks to me like the

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

@kai-holthause or @gimpeystrada: would either of you be able to test with the daily artful (17.10) ISO? @kai-holthause: I think you did find a real ordering issue, nicely done :) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Also affects: systemd (Ubuntu Xenial) Importance: Undecided Status: New ** Changed in: systemd (Ubuntu Xenial) Assignee: (unassigned) => Dimitri John Ledkov (xnox) ** Changed in: systemd (Ubuntu Xenial) Importance: Undecided => High -- You received this bug notification

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Thanks, @nacc. So, here's what I've done in the meantime: - I've built another server with 14.04. Clean install, only OpenSSH server. Connected to an iscsi target. Reboot works. - Upgraded the server to 16.04.02. Reboot still works. Guess my hunch was wrong. :( - I've attached the journal as

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

@kai-holthaus: Thank you for the excellent data point, again, and for your willingness to provide the data! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

@kai-holthaus: also, afaict, that might mean you are hitting a *different* bug (yay for so many bugs) in open-iscsi relative to @gimpeystrada, as @gimpeystrada and others in this channel have said they experience this issue with a fresh install of 16.04 (16.10 and 17.04 as well). That's ok, we can

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

@nacc: I can certainly do that - but it won't be quite as fast as my response yesterday, as the machine that exhibits this error is in use during the day, and I need to schedule the downtime. In the meantime, I have installed a test machine from scratch, using the standard Ubuntu LTS server

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

@kai-holthaus: thank you for responding so quickly. Can you set up a persistent journal as described in c#8 and provide the journal as mentioned in that comment for the failed shutdown? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

I checked my system (16.04.02) which is also exhibiting this issue (my Debian systems do as well). On both OS, the open-iscsi service is in "active (exited)" state after booting (and not in failed state): ● open-iscsi.service - Login to default iSCSI targets Loaded: loaded

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

My test configuration has long since been re-tasked. I will eventually be able to come back and test as you ask, but it may take a few weeks. If anyone else on the bug has their systems up and available, please test and reply. -- You received this bug notification because you are a member of

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

@gimpeystrada: Thank you for providing your journal logs (in the future, providing the raw journal file is better, as then we can run `journalctl` on it locally). From the logs: Apr 26 08:52:13 ICTM1612S02H1 iscsiadm[3234]: iscsiadm: initiator reported error (8 - connection timed out) Apr 26

Re: [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

On Wed, May 24, 2017 at 1:45 PM, Scott <1569...@bugs.launchpad.net> wrote: > Fresh install 16.04 download ISO from ubuntu > Can confirm this issue as well. > If I manually log out of the iscsi session reboot no issue Interesting data point. > If I am logged into iscsi session reboot hangs. Need

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Fresh install 16.04 download ISO from ubuntu Can confirm this issue as well. If I manually log out of the iscsi session reboot no issue If I am logged into iscsi session reboot hangs. Need to physically power cycle server -- You received this bug notification because you are a member of Ubuntu

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Didn't take as long as I thought. Installed the most recent build of Zesty (zesty-server-amd64.iso 2017-02-22 06:54676M). I was able to: 1. Confirm the issue still exists. 2. Confirm that the issue is not caused by multipath (i.e. it still occurs during reboot when multipathing is

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

No problem -- also, you can just test in 16.10 without multipath. I can spend time reproducing it, etc. later to verify whatever fix we come up with is correct. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

It is possible to test without multipath, this will take some time as I no longer have anything with 16.04 installed. For fun I may go ahead and try with 17.04 so we can figure out if it still exists then we'll kind of kill two birds with one stone. -- You received this bug notification because

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

@gimpeystrada: thank you for the prompt response! Err, right on multipath. Is it at all possible to test without multipath in use? To help narrow down if it's really in iSCSI or if it's in the multipath layer? Note additional bugs aren't required (I understand at some point this one may have

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

iSCSI disks are not the root/boot disks. Multipath IS in use. Hence the /dev/mapper at the beginning of the disk names. Yes I have tried 16.10, it is still present and I have another bug for that release. https://bugs.launchpad.net/bugs/1636862 -- You received this bug notification because

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

So let's separate out two issues in this bug. @gimpeystrada, I believe your issue was always with iSCSI disks that were not in-use as the root/boot disk, correct? Looking at your fstab, I think also you don't have multipath over iSCSI, correct? For anyone else indicating this bug is also

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Any idea on this one folks? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets To manage notifications about this bug go to:

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Seeing a similar issue here with so called "diskless" clients. That is, all of their filesystems are network mounts including the root filesystem. These systems use iscsi targets for the root filesystem and during shutdown the network is turned off before the file I/O operations are completed

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

New bug for 16.10: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1636862 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets To manage

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Changed in: systemd (Ubuntu) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets To manage notifications

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Attachment added: "Screenshot.jpg" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+attachment/4769386/+files/Screenshot.jpg ** Description changed: I have 4 servers running the latest 16.04 updates from the development branch (as of right now). Each server is

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

** Changed in: systemd (Ubuntu) Status: Expired => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets To manage notifications

  1   2   >