Unrelated to systemd. Related to open-iscsi systemd unit files and
kernel iscsi transport logout during shutdown.
** Changed in: open-iscsi (Ubuntu Xenial)
Importance: Undecided => Medium
** Changed in: open-iscsi (Ubuntu Xenial)
Status: New => In Progress
** Changed in: open-iscsi
## SUMMARY 1
Affects open-iscsi (NOT systemd):
* ISCSID NEEDS TO BE UP & RUNNING (AND LOGGED TO PORTAL) FOR THE LOGOUT
TO WORK
https://pastebin.canonical.com/197359/
Daemon shutdown documentation in iscsiadm command says (about stopping iscsid
daemon with iscsiadm -k):
""" ... This will
** Also affects: open-iscsi (Ubuntu Artful)
Importance: Undecided
Status: New
** Also affects: linux (Ubuntu Artful)
Importance: Undecided
Status: New
** Also affects: systemd (Ubuntu Artful)
Importance: High
Assignee: Nish Aravamudan (nacc)
Status: Confirmed
** Also affects: open-iscsi (Ubuntu)
Importance: Undecided
Status: New
** Also affects: linux (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
Okay, I removed almost everything out of equation:
- removed networking interfaces from systemd
- removed open-iscsi login/logout logic from systemd
- used a single network interface for the logins, on a single iscsi portal
was still able to reproduce the issue by:
- doing a simple login after
For now, the workaround for this bug is this:
inaddy@iscsihang:~$ sudo su -
# disconnect what iscsid has mapped
root@iscsihang:~$ iscsiadm -m node --logoutall=all
# disconnect all leftovers
root@iscsihang:~$ for file in `find /sys/devices -name *delete`; do echo 1 >
$file; done
# shutdown
Excellent, thank you for the details.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1569925
Title:
Shutdown hang on 16.04 with iscsi targets
Status in systemd package
This is what I discovered:
This systemd issue is caused because there are iscsi connections still
logged in (not because of multipath, bonding, vlans, etc). One way of
reproducing this is by simply editing the open-iscsi.service unit and
disabling the logout script. It will, for me in all times,
As I said I don't have one up right now, but Comment 3 above has my
interfaces for one of the recreates.
Nothing bothered me in any way. I wanted to make sure that you knew
that I wasn't one guy who was looking to fix his one system, that's all.
You said "Usually those type of "partnership"
Okay, besides all crap I posted - which are definitely right and will
possibly be addressed in other bugs, and also show other problems that
could make systemd to hang because of iscsi disks - I was able to cause
the same symptom of this bug:
[ OK ] Stopped LSB: ebtables ruleset management.
[
Might be unrelated, but I discovered that:
## root namespace
root@iscsihang:~$ mount | egrep -E "sd.*(xfs|ext4)"
/dev/sda1 on /ext4 type ext4 (rw,relatime,stripe=32,data=ordered,_netdev)
/dev/sdb1 on /xfs type xfs (rw,relatime,attr2,inode64,noquota,_netdev)
## specific mnt namespace
Hypothesis,
Test (2) - The error is propagated to upper layers after X seconds.
In this case I'm testing:
$ sudo iscsiadm -m node -o show | grep node.session.timeo.replac
node.session.timeo.replacement_timeout = 60
node.session.timeo.replacement_timeout = 60
And same "locking" behavior is
Hypothesis,
Test (1) - The error is NEVER propagated to upper layers:
# xfs and ext4 mounted automatically
inaddy@iscsihang:~$ mount | grep _netde
/dev/sda1 on /ext4 type ext4 (rw,relatime,stripe=32,data=ordered,_netdev)
/dev/sdb1 on /xfs type xfs (rw,relatime,attr2,inode64,noquota,_netdev)
#
Please let me explain who I am and what we do to possibly shed some
light on this.
I am an Interoperability engineer at NetApp. We test our storage
products with many popular operating systems, adapters, protocols and
switches, in varying combinations.
We test software initiator iscsi
Hello Matt,
Good to read your feedback. When you have multipath AND iscsi, the error
propagation timeout can be given by one or both, that is why I explained
both. This issue is not an iscsi or multipath problem, it is a systemd
problem related to the fact that the network is removed before the
I have node.session.timeo.replacement_timeout = 20, which seems
completely reasonable in my book.
We have already established that this occurs without multipath even
enabled, thus it is not a multipath settings problem.
Also I have no containers.
--
You received this bug notification because
** Changed in: systemd (Ubuntu Xenial)
Assignee: (unassigned) => Rafael David Tinoco (inaddy)
** Changed in: systemd (Ubuntu Xenial)
Status: Confirmed => In Progress
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed
** Changed in: systemd (Ubuntu Xenial)
Assignee: Dimitri John Ledkov (xnox) => (unassigned)
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1569925
Title:
Shutdown
Another thing. Please check for containers referencing the filesystems
in question. When you have _netdev filesystems, it is possible that the
"mount --shared" characteristics of containers (different mount
namescapes) are holding references to the filesystem in question.
A good test is to test
Anyone facing this..
Could you please make sure to propagate I/O errors properly to
filesystem in order for it to call its automatically shutdown procedure.
After that, the mount won't reference the super block in kernel and that
will allow the device mapper to be destroyed (this can be seen by
Anyone facing this..
Could you please make sure to propagate I/O errors properly to
filesystem in order for it to call its automatically shutdown procedure.
After that, the mount won't reference the super block in kernel and that
will allow the device mapper to be closed/destroyed (this can be
Is it possible to reproduce this problem with upstart or sysvinit-core?
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1569925
Title:
Shutdown hang on 16.04 with iscsi
I can confirm that the daily 17.10 build does not exhibit the problem.
Interestingly, the first time we tried we forgot to stop running IO to
our iSCSI volumes and it _looked_ like we reproduced the problem, then
the IOs timed out and our application stopped and we were able to
complete the
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: systemd (Ubuntu Xenial)
Status: New => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
@nacc: Yes.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1569925
Title:
Shutdown hang on 16.04 with iscsi targets
Status in systemd package in Ubuntu:
Confirmed
@kai-holthaus: after editing the service file, did you issue a `sudo
systemctl daemon-reload` ?
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1569925
Title:
Shutdown
@nacc: The system was originally installed with Ubuntu Server 14.04. It
has been upgraded to 16.04.
apt policy open-iscsi:
open-iscsi:
Installed: 2.0.873+git0.3b4b4500-14ubuntu3.3
Candidate: 2.0.873+git0.3b4b4500-14ubuntu3.3
Version table:
*** 2.0.873+git0.3b4b4500-14ubuntu3.3 500
@Kai: Hrm, that is odd -- was your 16.04 system the fresh install or an
upgraded one?
The lines I pasted in #49 are supposed to be present in 16.04 (afaict).
If they don't, can you provide `apt policy open-iscsi` in a pastebin?
And can you test if changing those lines to match fixes the issue
@nacc: Add-on: On my Debian systems (Debian jessie and stretch) that
also experience this issue, however, the lines do look like what you
described - and still the system hangs at reboot / shutdown.
In a previous message it sounded like you had identified an issue and a
fix - did I misinterpret
@nacc: No, my lines do not like that. On my xenial system, I have:
Wants=network-online.target remote-fs-pre.target
After=network-online.target
Which is probably why on my system the iscsid daemon shuts down before
the logout.
--
You received this bug notification because you are a member of
@Kai: In xenial, open-iscsi.service should have the following lines
alreadY
Wants=network-online.target remote-fs-pre.target iscsid.service
After=network-online.target iscsid.service
Can you confirm that is the case on your system?
--
You received this bug notification because you are a member
I've installed a system with the daily artful server image, and the
error is not present for me. However, I'm not sure how useful that test
is, given that I didn't experience the error during a fresh install of
xenial.
I've attached the journal from the artful install, and it looks to me
like the
@kai-holthause or @gimpeystrada: would either of you be able to test
with the daily artful (17.10) ISO?
@kai-holthause: I think you did find a real ordering issue, nicely done
:)
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed
** Also affects: systemd (Ubuntu Xenial)
Importance: Undecided
Status: New
** Changed in: systemd (Ubuntu Xenial)
Assignee: (unassigned) => Dimitri John Ledkov (xnox)
** Changed in: systemd (Ubuntu Xenial)
Importance: Undecided => High
--
You received this bug notification
Thanks, @nacc.
So, here's what I've done in the meantime:
- I've built another server with 14.04. Clean install, only OpenSSH server.
Connected to an iscsi target. Reboot works.
- Upgraded the server to 16.04.02. Reboot still works. Guess my hunch was
wrong. :(
- I've attached the journal as
@kai-holthaus: also, afaict, that might mean you are hitting a
*different* bug (yay for so many bugs) in open-iscsi relative to
@gimpeystrada, as @gimpeystrada and others in this channel have said
they experience this issue with a fresh install of 16.04 (16.10 and
17.04 as well). That's ok, we can
@kai-holthaus: Thank you for the excellent data point, again, and for
your willingness to provide the data!
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1569925
Title:
@nacc: I can certainly do that - but it won't be quite as fast as my response
yesterday, as the machine that exhibits this error is in use during the day,
and I need to schedule the downtime.
In the meantime, I have installed a test machine from scratch, using the
standard Ubuntu LTS server
@kai-holthaus: thank you for responding so quickly. Can you set up a
persistent journal as described in c#8 and provide the journal as
mentioned in that comment for the failed shutdown?
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is
I checked my system (16.04.02) which is also exhibiting this issue (my Debian
systems do as well). On both OS, the open-iscsi service is in "active (exited)"
state after booting (and not in failed state):
● open-iscsi.service - Login to default iSCSI targets
Loaded: loaded
My test configuration has long since been re-tasked. I will eventually
be able to come back and test as you ask, but it may take a few weeks.
If anyone else on the bug has their systems up and available, please
test and reply.
--
You received this bug notification because you are a member of
@gimpeystrada: Thank you for providing your journal logs (in the future,
providing the raw journal file is better, as then we can run
`journalctl` on it locally). From the logs:
Apr 26 08:52:13 ICTM1612S02H1 iscsiadm[3234]: iscsiadm: initiator reported
error (8 - connection timed out)
Apr 26
On Wed, May 24, 2017 at 1:45 PM, Scott <1569...@bugs.launchpad.net> wrote:
> Fresh install 16.04 download ISO from ubuntu
> Can confirm this issue as well.
> If I manually log out of the iscsi session reboot no issue
Interesting data point.
> If I am logged into iscsi session reboot hangs. Need
Fresh install 16.04 download ISO from ubuntu
Can confirm this issue as well.
If I manually log out of the iscsi session reboot no issue
If I am logged into iscsi session reboot hangs. Need to physically power cycle
server
--
You received this bug notification because you are a member of Ubuntu
Didn't take as long as I thought. Installed the most recent build of
Zesty (zesty-server-amd64.iso 2017-02-22 06:54676M).
I was able to:
1. Confirm the issue still exists.
2. Confirm that the issue is not caused by multipath (i.e. it still occurs
during reboot when multipathing is
No problem -- also, you can just test in 16.10 without multipath. I can
spend time reproducing it, etc. later to verify whatever fix we come up
with is correct.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in
It is possible to test without multipath, this will take some time as I
no longer have anything with 16.04 installed. For fun I may go ahead
and try with 17.04 so we can figure out if it still exists then we'll
kind of kill two birds with one stone.
--
You received this bug notification because
@gimpeystrada: thank you for the prompt response!
Err, right on multipath. Is it at all possible to test without multipath
in use? To help narrow down if it's really in iSCSI or if it's in the
multipath layer?
Note additional bugs aren't required (I understand at some point this
one may have
iSCSI disks are not the root/boot disks.
Multipath IS in use. Hence the /dev/mapper at the beginning of the disk
names.
Yes I have tried 16.10, it is still present and I have another bug for
that release. https://bugs.launchpad.net/bugs/1636862
--
You received this bug notification because
So let's separate out two issues in this bug.
@gimpeystrada, I believe your issue was always with iSCSI disks that
were not in-use as the root/boot disk, correct? Looking at your fstab, I
think also you don't have multipath over iSCSI, correct?
For anyone else indicating this bug is also
Any idea on this one folks?
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1569925
Title:
Shutdown hang on 16.04 with iscsi targets
Status in systemd package in Ubuntu:
Seeing a similar issue here with so called "diskless" clients. That is,
all of their filesystems are network mounts including the root
filesystem. These systems use iscsi targets for the root filesystem and
during shutdown the network is turned off before the file I/O operations
are completed
New bug for 16.10:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1636862
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1569925
Title:
Shutdown hang on 16.04
** Changed in: systemd (Ubuntu)
Importance: Undecided => High
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1569925
Title:
Shutdown hang on 16.04 with iscsi targets
** Attachment added: "Screenshot.jpg"
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+attachment/4769386/+files/Screenshot.jpg
** Description changed:
I have 4 servers running the latest 16.04 updates from the development
branch (as of right now).
Each server is
** Changed in: systemd (Ubuntu)
Status: Expired => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1569925
Title:
Shutdown hang on 16.04 with iscsi
For completeness, I upgraded to 16.04.1, the issue is still present.
I upgraded to the latest xenial-updates packages as of 10/24/16 and the
issue is still present.
I then re-installed with 16.10 and the issue is still present. I guess
I'll open a new bug for 16.10 and refer back to this one.
If this is still a problem, what do we do now? Do I just keep testing
with new releases and file new bugs?
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1569925
Title:
[Expired for systemd (Ubuntu) because there has been no activity for 60
days.]
** Changed in: systemd (Ubuntu)
Status: Incomplete => Expired
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
There is clearly something wrong here.
FWIW This issue did not exist in 14.04.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1569925
Title:
Shutdown hang on 16.04 with
I got same preblom
look iscsi not logout before Network-manager cut the network
so iscsi can not logout.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1569925
Title:
Seeing what I believe is this bug on a 16.04 install that boots from
iSCSI LUN. When I shut down, it goes just past shutting down the network
interfaces, then slows down and starts throwing task blocked messages
from the kernel. It eventually gets down to "Reached Target Shutdown"
before getting
rc_micha: Your volumes aren't mounted, but the iscsi sessions are logged
in? If you log out of your iscsi sessions does it shutdown clean?
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
I have the same problem with my multipath iscsi session. I just logged in
without mounting the volumes.
While rebooting I get always the message "systemd-shutdown[1]: Failed to
finalize DM devices, ignoring" but in most cases the server just ignores that
an reboots.
The iscsi multipath volumes
I have finally been able to verify that this is reproducible with the
released 16.04 from the ISO.
What else can be provided here to move this bug along?
(This is being tracked internally in CQ 860251)
--
You received this bug notification because you are a member of Ubuntu
Touch seeded
I definitely have remote filesystems mounted (using _netdev if that
matters) when I experience this problem.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1569925
Title:
OK, I'm trying reboot and reboot and now I think this is kernel bug and
it is multipath related...
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1569925
Title:
Shutdown
Because it is not always reproduced, I guess this is dependency problem.
Looked into /etc/init.d/iscsid
# Required-Stop: $network $local_fs sendsigs
I guess we need $remote_fs here?
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is
>Looks like iscsi not always stopped before network interfaces are down.
No, just reproduced this, iscsi was logged out...
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
I have the same problem , target is HPE MSA, and problem happens not every
time, about in 1/5 of reboots- I run 16.04 , so I just tested...
Looks like iscsi not always stopped before network interfaces are down.
--
You received this bug notification because you are a member of Ubuntu
Touch
Contents of fstab:
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
#
# / was
This contains the configuration of my iscsi sessions. It is a tar of
the entire /etc/iscsi folder, it should show what we've got going on.
** Attachment added: "iscsi.tar.gz"
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+attachment/4648082/+files/iscsi.tar.gz
--
You
And just for good measure this is my /etc/network/interfaces file:
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet
Here is the requested journal.txt file.
I had a bit of difficulty after performing the requested operations, my
server kept booting into the (initramfs) prompt.
The journal log may contain more than one reboot attempt.
** Attachment added: "journal.txt"
Unfortunately rsyslog gets shut down too early, so we don't see much
from this. Can you please re-try with enabling persistent journal:
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
then reboot once to actually enable on-disk journal, then reboot
Nothing for the system is on iscsi. They are just extra volumes that I
use for generating block or filesystem IO to our storage.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
Here is the requested syslog.
Reboot happens after:
Apr 14 09:49:42 ICTM1612S02H1 root: MDS: Rebooting Now
** Attachment added: "requested syslog"
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+attachment/4636878/+files/syslog
--
You received this bug notification because
Do you have your root partition on iscsi, or just some auxiliary ones
like /opt?
If it's the root partition, then you can't put that on an interface
which is "auto" in /etc/network/interfaces -- ifupdown will tear it down
during shutdown and sever the connection, explaining the hang. In this
In the above, the two NICs used for iSCSI traffic are the enp130s0 and
the enp4s0f0.
In case you were wondering I have another server running ipv4 and it
also hangs.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in
root@ICTM1612S02H1:/etc/network# cat interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The
My first suspicion is that the interface that the iscsi device is on
lives in /etc/network/interfaces (or interfaces.d/*) somewhere and thus
is being shut down by networking.service. Ordinarily this network
interface is being set up in initramfs by open-iscsi and should not
otherwise be
** Tags added: xenial
** Package changed: ubuntu => systemd (Ubuntu)
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1569925
Title:
Shutdown hang on 16.04 with iscsi
82 matches
Mail list logo
