[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Launchpad Bug Tracker]

This bug was fixed in the package linux - 3.13.0-144.193

---
linux (3.13.0-144.193) trusty; urgency=medium

  * linux: 3.13.0-144.193 -proposed tracker (LP: #1755227)

  * CVE-2017-12762
- isdn/i4l: fix buffer overflow

  * CVE-2017-17807
- KEYS: add missing permission check for request_key() destination

  * bnx2x_attn_int_deasserted3:4323 MC assert! (LP: #1715519) //
CVE-2018-126
- net: Add ndo_gso_check
- net: create skb_gso_validate_mac_len()
- bnx2x: disable GSO where gso_size is too big for hardware

  * CVE-2017-17448
- netfilter: nfnetlink_cthelper: Add missing permission checks

  * CVE-2017-11089
- cfg80211: Define nla_policy for NL80211_ATTR_LOCAL_MESH_POWER_MODE

  * CVE-2018-5332
- RDS: Heap OOB write in rds_message_alloc_sgs()

  * ppc64el: Do not call ibm,os-term on panic (LP: #1736954)
- powerpc: Do not call ppc_md.panic in fadump panic notifier

  * CVE-2017-17805
- crypto: salsa20 - fix blkcipher_walk API usage

  * [Hyper-V] storvsc: do not assume SG list is continuous when doing bounce
buffers (LP: #1742480)
- SAUCE: storvsc: do not assume SG list is continuous when doing bounce
  buffers

  * Shutdown hang on 16.04 with iscsi targets (LP: #1569925)
- scsi: libiscsi: Allow sd_shutdown on bad transport

  * CVE-2017-17741
- KVM: Fix stack-out-of-bounds read in write_mmio

  * CVE-2017-5715 (Spectre v2 Intel)
- [Packaging] pull in retpoline files

 -- Stefan Bader   Thu, 15 Mar 2018 15:08:03
+0100

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Launchpad Bug Tracker]

This bug was fixed in the package linux - 3.13.0-144.193

---
linux (3.13.0-144.193) trusty; urgency=medium

  * linux: 3.13.0-144.193 -proposed tracker (LP: #1755227)

  * CVE-2017-12762
- isdn/i4l: fix buffer overflow

  * CVE-2017-17807
- KEYS: add missing permission check for request_key() destination

  * bnx2x_attn_int_deasserted3:4323 MC assert! (LP: #1715519) //
CVE-2018-126
- net: Add ndo_gso_check
- net: create skb_gso_validate_mac_len()
- bnx2x: disable GSO where gso_size is too big for hardware

  * CVE-2017-17448
- netfilter: nfnetlink_cthelper: Add missing permission checks

  * CVE-2017-11089
- cfg80211: Define nla_policy for NL80211_ATTR_LOCAL_MESH_POWER_MODE

  * CVE-2018-5332
- RDS: Heap OOB write in rds_message_alloc_sgs()

  * ppc64el: Do not call ibm,os-term on panic (LP: #1736954)
- powerpc: Do not call ppc_md.panic in fadump panic notifier

  * CVE-2017-17805
- crypto: salsa20 - fix blkcipher_walk API usage

  * [Hyper-V] storvsc: do not assume SG list is continuous when doing bounce
buffers (LP: #1742480)
- SAUCE: storvsc: do not assume SG list is continuous when doing bounce
  buffers

  * Shutdown hang on 16.04 with iscsi targets (LP: #1569925)
- scsi: libiscsi: Allow sd_shutdown on bad transport

  * CVE-2017-17741
- KVM: Fix stack-out-of-bounds read in write_mmio

  * CVE-2017-5715 (Spectre v2 Intel)
- [Packaging] pull in retpoline files

 -- Stefan Bader   Thu, 15 Mar 2018 15:08:03
+0100

** Changed in: linux (Ubuntu Trusty)
   Status: Fix Committed => Fix Released

** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2017-11089

** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2017-12762

** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2017-17448

** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2017-17741

** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2017-17805

** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2017-17807

** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2018-5332

** Changed in: linux (Ubuntu Trusty)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

I'm failing trusty's patch because:

1) It was not backported fully, missing the debug message at its bottom.
That led me to initially think, when debugging, that the patch hasn't
been included at all. Later I could check kernel tree and saw the it was
manually merged, missing that debug message part.

2) Since functional piece of code was backported, it should, still, have
worked, and it did not. Something in trusty is not making it to work
like expected. I'm still investigating it through a kdump taken during a
shutdown hang on trusty, but, I don't want to hold the kernel release
because of that:

This is the output for my tests:

https://pastebin.ubuntu.com/p/B9fp5y5gqK/

And you can check (1) with:

[  144.862511]  session4: iscsi_eh_cmd_timed_out return nh

It should be "or shutdown" there, together.

Differently than other tests, in trusty's kernel (3.13) you can see:

[  144.860010]  session4: iscsi_eh_cmd_timed_out scsi cmd 880037a9a100 
timedout
[  144.861400]  session4: iscsi_eh_cmd_timed_out sc on shutdown, handled
[  144.862511]  session4: iscsi_eh_cmd_timed_out return nh
[  144.863483]  session4: iscsi_queuecommand iscsi: cmd 0x3 is not queued (2)

[  144.864794]  session4: iscsi_eh_device_reset LU Reset [sc 880037a9a100 
lun 1]
[  144.865907]  session4: iscsi_eh_device_reset dev reset result = FAILED
[  144.866875]  session4: iscsi_eh_target_reset tgt Reset [sc 880037a9a100 
tgt iqn.2017.tgtd:tid5.lun]
[  144.868343]  session4: iscsi_eh_target_reset tgt iqn.2017.tgtd:tid5.lun 
reset result = FAILED
[  144.869580]  session4: iscsi_eh_session_reset wait for relogin

The iscsi transport layer, did not queue the other commands, after the
first one timed out, but instead of warning the upper layer, it
proceeded with a "device reset", causing the "relogin" logic to be stuck
since the transport layer was already gone.

This is the hang backtrace:

crash> bt
PID: 5980   TASK: 880057e19800  CPU: 0   COMMAND: "halt"
 #0 [880037b53a50] __schedule at 8173af59
 #1 [880037b53ab8] schedule at 8173b3e9
 #2 [880037b53ac8] schedule_timeout at 8173a5d5
 #3 [880037b53b70] io_schedule_timeout at 8173bacb
 #4 [880037b53ba0] wait_for_completion_io_timeout at 8173c1e6
 #5 [880037b53bf8] blk_execute_rq at 8134b9fb
 #6 [880037b53ca8] scsi_execute at 814f1a77
 #7 [880037b53cf0] scsi_execute_req_flags at 814f2cfc
 #8 [880037b53d58] sd_sync_cache at 81500626
 #9 [880037b53dd0] sd_shutdown at 81500bb9
#10 [880037b53df0] device_shutdown at 814a4495
#11 [880037b53e20] kernel_power_off at 81096b75
#12 [880037b53e30] SYSC_reboot at 81096d4b
#13 [880037b53f70] sys_reboot at 81096ebe
#14 [880037b53f80] system_call_fastpath at 81748170
RIP: 7ff48aa45bc6  RSP: 7ffef9bc4be8  RFLAGS: 00010246
RAX: 00a9  RBX: 81748170  RCX: 001e
RDX: 4321fedc  RSI: 28121969  RDI: fee1dead
RBP:    R8: fefefefefefefeff   R9: 
R10: 7ff48ad13c8c  R11: 0202  R12: 81096ebe
R13: 880037b53f78  R14:   R15: 
ORIG_RAX: 00a9  CS: 0033  SS: 002b

Taken by a watchdog timeout on a hang kvm system that had network
interface shutdown before the transport layer could log out.

** Tags removed: verification-needed-trusty
** Tags added: verification-failed-trusty

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

I'm verifying for trusty and its taking sometime because looks like
system still hangs on shutdown. I have generated a kdump (from a
watchdog timer) in the exact moment trusty is hanging on shutdown to see
why BUT crash can't handle a kdump from this new kernel and I still
don't know why. Testing latest crash tool to see if its because of
recent meltdown/spectre patches or something like it.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Stefan Bader]

This bug is awaiting verification that the kernel in -proposed solves
the problem. Please test the kernel and update this bug with the
results. If the problem is solved, change the tag 'verification-needed-
trusty' to 'verification-done-trusty'. If the problem still exists,
change the tag 'verification-needed-trusty' to 'verification-failed-
trusty'.

If verification is not done by 5 working days from today, this fix will
be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how
to enable and use -proposed. Thank you!


** Tags added: verification-needed-trusty

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Launchpad Bug Tracker]

This bug was fixed in the package linux - 4.15.0-10.11

---
linux (4.15.0-10.11) bionic; urgency=medium

  * linux: 4.15.0-10.11 -proposed tracker (LP: #1749250)

  * "swiotlb: coherent allocation failed" dmesg spam with linux 4.15.0-9.10
(LP: #1749202)
- swiotlb: suppress warning when __GFP_NOWARN is set
- drm/ttm: specify DMA_ATTR_NO_WARN for huge page pools

  * linux-tools: perf incorrectly linking libbfd (LP: #1748922)
- SAUCE: tools -- add ability to disable libbfd
- [Packaging] correct disablement of libbfd

  * [Artful] Realtek ALC225: 2 secs noise when a headset plugged in
(LP: #1744058)
- ALSA: hda/realtek - update ALC225 depop optimize

  * [Artful] Support headset mode for DELL WYSE (LP: #1723913)
- SAUCE: ALSA: hda/realtek - Add support headset mode for DELL WYSE

  * headset mic can't be detected on two Dell machines (LP: #1748807)
- ALSA: hda/realtek - Support headset mode for ALC215/ALC285/ALC289
- ALSA: hda - Fix headset mic detection problem for two Dell machines

  * Bionic update to v4.15.3 stable release (LP: #1749191)
- ip6mr: fix stale iterator
- net: igmp: add a missing rcu locking section
- qlcnic: fix deadlock bug
- qmi_wwan: Add support for Quectel EP06
- r8169: fix RTL8168EP take too long to complete driver initialization.
- tcp: release sk_frag.page in tcp_disconnect
- vhost_net: stop device during reset owner
- ipv6: addrconf: break critical section in addrconf_verify_rtnl()
- ipv6: change route cache aging logic
- Revert "defer call to mem_cgroup_sk_alloc()"
- net: ipv6: send unsolicited NA after DAD
- rocker: fix possible null pointer dereference in
  rocker_router_fib_event_work
- tcp_bbr: fix pacing_gain to always be unity when using lt_bw
- cls_u32: add missing RCU annotation.
- ipv6: Fix SO_REUSEPORT UDP socket with implicit sk_ipv6only
- soreuseport: fix mem leak in reuseport_add_sock()
- net_sched: get rid of rcu_barrier() in tcf_block_put_ext()
- net: sched: fix use-after-free in tcf_block_put_ext
- media: mtk-vcodec: add missing MODULE_LICENSE/DESCRIPTION
- media: soc_camera: soc_scale_crop: add missing
  MODULE_DESCRIPTION/AUTHOR/LICENSE
- media: tegra-cec: add missing MODULE_DESCRIPTION/AUTHOR/LICENSE
- gpio: uniphier: fix mismatch between license text and MODULE_LICENSE
- crypto: tcrypt - fix S/G table for test_aead_speed()
- Linux 4.15.3

  * bnx2x_attn_int_deasserted3:4323 MC assert! (LP: #1715519) //
CVE-2018-126
- net: create skb_gso_validate_mac_len()
- bnx2x: disable GSO where gso_size is too big for hardware

  * ethtool -p fails to light NIC LED on HiSilicon D05 systems (LP: #1748567)
- net: hns: add ACPI mode support for ethtool -p

  * CVE-2017-5715 (Spectre v2 Intel)
- [Packaging] retpoline files must be sorted
- [Packaging] pull in retpoline files

  * [Feature] PXE boot with Intel Omni-Path (LP: #1712031)
- d-i: Add hfi1 to nic-modules

  * CVE-2017-5715 (Spectre v2 retpoline)
- [Packaging] retpoline -- add call site validation
- [Config] disable retpoline checks for first upload

  * Do not duplicate changelog entries assigned to more than one bug or CVE
(LP: #1743383)
- [Packaging] git-ubuntu-log -- handle multiple bugs/cves better

 -- Seth Forshee   Tue, 13 Feb 2018 11:33:58
-0600

** Changed in: linux (Ubuntu Bionic)
   Status: Fix Committed => Fix Released

** CVE added: https://cve.mitre.org/cgi-
bin/cvename.cgi?name=2018-126

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Launchpad Bug Tracker]

This bug was fixed in the package linux - 4.4.0-116.140

---
linux (4.4.0-116.140) xenial; urgency=medium

  * linux: 4.4.0-116.140 -proposed tracker (LP: #1748990)

  * BUG: unable to handle kernel NULL pointer dereference at 0009
(LP: #1748671)
- SAUCE: net: ipv4: fix for a race condition in raw_sendmsg -- fix backport

linux (4.4.0-115.139) xenial; urgency=medium

  * linux: 4.4.0-115.138 -proposed tracker (LP: #1748745)

  * CVE-2017-5715 (Spectre v2 Intel)
- Revert "UBUNTU: SAUCE: turn off IBPB when full retpoline is present"
- SAUCE: turn off IBRS when full retpoline is present
- [Packaging] retpoline files must be sorted
- [Packaging] pull in retpoline files

linux (4.4.0-114.137) xenial; urgency=medium

  * linux: 4.4.0-114.137 -proposed tracker (LP: #1748484)

  * ALSA backport missing NVIDIA GPU codec IDs to patch table to
Ubuntu 16.04 LTS Kernel (LP: #1744117)
- ALSA: hda - Add missing NVIDIA GPU codec IDs to patch table

  * Shutdown hang on 16.04 with iscsi targets (LP: #1569925)
- scsi: libiscsi: Allow sd_shutdown on bad transport

  * libata: apply MAX_SEC_1024 to all LITEON EP1 series devices (LP: #1743053)
- libata: apply MAX_SEC_1024 to all LITEON EP1 series devices

  * KVM patches for s390x to provide facility bits 81 (ppa15) and 82 (bpb)
(LP: #1747090)
- KVM: s390: wire up bpb feature
- KVM: s390: Enable all facility bits that are known good for passthrough

  * CVE-2017-5715 (Spectre v2 Intel)
- SAUCE: drop lingering gmb() macro
- x86/feature: Enable the x86 feature to control Speculation
- x86/feature: Report presence of IBPB and IBRS control
- x86/enter: MACROS to set/clear IBRS and set IBPB
- x86/enter: Use IBRS on syscall and interrupts
- x86/idle: Disable IBRS entering idle and enable it on wakeup
- x86/idle: Disable IBRS when offlining cpu and re-enable on wakeup
- x86/mm: Set IBPB upon context switch
- x86/mm: Only set IBPB when the new thread cannot ptrace current thread
- x86/kvm: add MSR_IA32_SPEC_CTRL and MSR_IA32_PRED_CMD to kvm
- x86/kvm: Set IBPB when switching VM
- x86/kvm: Toggle IBRS on VM entry and exit
- x86/spec_ctrl: Add sysctl knobs to enable/disable SPEC_CTRL feature
- x86/spec_ctrl: Add lock to serialize changes to ibrs and ibpb control
- x86/cpu/amd, kvm: Satisfy guest kernel reads of IC_CFG MSR
- x86/cpu/AMD: Add speculative control support for AMD
- x86/microcode: Extend post microcode reload to support IBPB feature
- KVM: SVM: Do not intercept new speculative control MSRs
- x86/svm: Set IBRS value on VM entry and exit
- x86/svm: Set IBPB when running a different VCPU
- KVM: x86: Add speculative control CPUID support for guests
- SAUCE: Fix spec_ctrl support in KVM
- SAUCE: turn off IBPB when full retpoline is present

linux (4.4.0-113.136) xenial; urgency=low

  * linux: 4.4.0-113.136 -proposed tracker (LP: #1746936)

  [ Stefan Bader ]
  * Missing install-time driver for QLogic QED 25/40/100Gb Ethernet NIC
(LP: #1743638)
- [d-i] Add qede to nic-modules udeb

  * CVE-2017-5753 (Spectre v1 Intel)
- x86/cpu/AMD: Make the LFENCE instruction serialized
- x86/cpu/AMD: Remove now unused definition of MFENCE_RDTSC feature
- SAUCE: reinstate MFENCE_RDTSC feature definition
- locking/barriers: introduce new observable speculation barrier
- bpf: prevent speculative execution in eBPF interpreter
- x86, bpf, jit: prevent speculative execution when JIT is enabled
- SAUCE: FIX: x86, bpf, jit: prevent speculative execution when JIT is 
enabled
- carl9170: prevent speculative execution
- qla2xxx: prevent speculative execution
- Thermal/int340x: prevent speculative execution
- ipv4: prevent speculative execution
- ipv6: prevent speculative execution
- fs: prevent speculative execution
- net: mpls: prevent speculative execution
- udf: prevent speculative execution
- userns: prevent speculative execution
- SAUCE: claim mitigation via observable speculation barrier
- SAUCE: powerpc: add osb barrier
- SAUCE: s390/spinlock: add osb memory barrier
- SAUCE: arm64: no osb() implementation yet
- SAUCE: arm: no osb() implementation yet

  * CVE-2017-5715 (Spectre v2 retpoline)
- x86/cpuid: Provide get_scattered_cpuid_leaf()
- x86/cpu: Factor out application of forced CPU caps
- x86/cpufeatures: Make CPU bugs sticky
- x86/cpufeatures: Add X86_BUG_CPU_INSECURE
- x86/cpu, x86/pti: Do not enable PTI on AMD processors
- x86/pti: Rename BUG_CPU_INSECURE to BUG_CPU_MELTDOWN
- x86/cpufeatures: Add X86_BUG_SPECTRE_V[12]
- x86/cpu: Merge bugs.c and bugs_64.c
- sysfs/cpu: Add vulnerability folder
- x86/cpu: Implement CPU vulnerabilites sysfs functions
- x86/alternatives: Add missing '\n' at end of ALTERNATIVE inline asm
- x86/mm/32: Move setup_clear_cpu_cap(X86_FEATURE_PCID) earlier
- 

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Launchpad Bug Tracker]

This bug was fixed in the package linux - 4.4.0-116.140

---
linux (4.4.0-116.140) xenial; urgency=medium

  * linux: 4.4.0-116.140 -proposed tracker (LP: #1748990)

  * BUG: unable to handle kernel NULL pointer dereference at 0009
(LP: #1748671)
- SAUCE: net: ipv4: fix for a race condition in raw_sendmsg -- fix backport

linux (4.4.0-115.139) xenial; urgency=medium

  * linux: 4.4.0-115.138 -proposed tracker (LP: #1748745)

  * CVE-2017-5715 (Spectre v2 Intel)
- Revert "UBUNTU: SAUCE: turn off IBPB when full retpoline is present"
- SAUCE: turn off IBRS when full retpoline is present
- [Packaging] retpoline files must be sorted
- [Packaging] pull in retpoline files

linux (4.4.0-114.137) xenial; urgency=medium

  * linux: 4.4.0-114.137 -proposed tracker (LP: #1748484)

  * ALSA backport missing NVIDIA GPU codec IDs to patch table to
Ubuntu 16.04 LTS Kernel (LP: #1744117)
- ALSA: hda - Add missing NVIDIA GPU codec IDs to patch table

  * Shutdown hang on 16.04 with iscsi targets (LP: #1569925)
- scsi: libiscsi: Allow sd_shutdown on bad transport

  * libata: apply MAX_SEC_1024 to all LITEON EP1 series devices (LP: #1743053)
- libata: apply MAX_SEC_1024 to all LITEON EP1 series devices

  * KVM patches for s390x to provide facility bits 81 (ppa15) and 82 (bpb)
(LP: #1747090)
- KVM: s390: wire up bpb feature
- KVM: s390: Enable all facility bits that are known good for passthrough

  * CVE-2017-5715 (Spectre v2 Intel)
- SAUCE: drop lingering gmb() macro
- x86/feature: Enable the x86 feature to control Speculation
- x86/feature: Report presence of IBPB and IBRS control
- x86/enter: MACROS to set/clear IBRS and set IBPB
- x86/enter: Use IBRS on syscall and interrupts
- x86/idle: Disable IBRS entering idle and enable it on wakeup
- x86/idle: Disable IBRS when offlining cpu and re-enable on wakeup
- x86/mm: Set IBPB upon context switch
- x86/mm: Only set IBPB when the new thread cannot ptrace current thread
- x86/kvm: add MSR_IA32_SPEC_CTRL and MSR_IA32_PRED_CMD to kvm
- x86/kvm: Set IBPB when switching VM
- x86/kvm: Toggle IBRS on VM entry and exit
- x86/spec_ctrl: Add sysctl knobs to enable/disable SPEC_CTRL feature
- x86/spec_ctrl: Add lock to serialize changes to ibrs and ibpb control
- x86/cpu/amd, kvm: Satisfy guest kernel reads of IC_CFG MSR
- x86/cpu/AMD: Add speculative control support for AMD
- x86/microcode: Extend post microcode reload to support IBPB feature
- KVM: SVM: Do not intercept new speculative control MSRs
- x86/svm: Set IBRS value on VM entry and exit
- x86/svm: Set IBPB when running a different VCPU
- KVM: x86: Add speculative control CPUID support for guests
- SAUCE: Fix spec_ctrl support in KVM
- SAUCE: turn off IBPB when full retpoline is present

linux (4.4.0-113.136) xenial; urgency=low

  * linux: 4.4.0-113.136 -proposed tracker (LP: #1746936)

  [ Stefan Bader ]
  * Missing install-time driver for QLogic QED 25/40/100Gb Ethernet NIC
(LP: #1743638)
- [d-i] Add qede to nic-modules udeb

  * CVE-2017-5753 (Spectre v1 Intel)
- x86/cpu/AMD: Make the LFENCE instruction serialized
- x86/cpu/AMD: Remove now unused definition of MFENCE_RDTSC feature
- SAUCE: reinstate MFENCE_RDTSC feature definition
- locking/barriers: introduce new observable speculation barrier
- bpf: prevent speculative execution in eBPF interpreter
- x86, bpf, jit: prevent speculative execution when JIT is enabled
- SAUCE: FIX: x86, bpf, jit: prevent speculative execution when JIT is 
enabled
- carl9170: prevent speculative execution
- qla2xxx: prevent speculative execution
- Thermal/int340x: prevent speculative execution
- ipv4: prevent speculative execution
- ipv6: prevent speculative execution
- fs: prevent speculative execution
- net: mpls: prevent speculative execution
- udf: prevent speculative execution
- userns: prevent speculative execution
- SAUCE: claim mitigation via observable speculation barrier
- SAUCE: powerpc: add osb barrier
- SAUCE: s390/spinlock: add osb memory barrier
- SAUCE: arm64: no osb() implementation yet
- SAUCE: arm: no osb() implementation yet

  * CVE-2017-5715 (Spectre v2 retpoline)
- x86/cpuid: Provide get_scattered_cpuid_leaf()
- x86/cpu: Factor out application of forced CPU caps
- x86/cpufeatures: Make CPU bugs sticky
- x86/cpufeatures: Add X86_BUG_CPU_INSECURE
- x86/cpu, x86/pti: Do not enable PTI on AMD processors
- x86/pti: Rename BUG_CPU_INSECURE to BUG_CPU_MELTDOWN
- x86/cpufeatures: Add X86_BUG_SPECTRE_V[12]
- x86/cpu: Merge bugs.c and bugs_64.c
- sysfs/cpu: Add vulnerability folder
- x86/cpu: Implement CPU vulnerabilites sysfs functions
- x86/alternatives: Add missing '\n' at end of ALTERNATIVE inline asm
- x86/mm/32: Move setup_clear_cpu_cap(X86_FEATURE_PCID) earlier
- 

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Launchpad Bug Tracker]

This bug was fixed in the package linux - 4.13.0-36.40

---
linux (4.13.0-36.40) artful; urgency=medium

  * linux: 4.13.0-36.40 -proposed tracker (LP: #1750010)

  * Rebuild without "CVE-2017-5754 ARM64 KPTI fixes" patch set

linux (4.13.0-35.39) artful; urgency=medium

  * linux: 4.13.0-35.39 -proposed tracker (LP: #1748743)

  * CVE-2017-5715 (Spectre v2 Intel)
- Revert "UBUNTU: SAUCE: turn off IBPB when full retpoline is present"
- SAUCE: turn off IBRS when full retpoline is present
- [Packaging] retpoline files must be sorted
- [Packaging] pull in retpoline files

linux (4.13.0-34.37) artful; urgency=medium

  * linux: 4.13.0-34.37 -proposed tracker (LP: #1748475)

  * libata: apply MAX_SEC_1024 to all LITEON EP1 series devices (LP: #1743053)
- libata: apply MAX_SEC_1024 to all LITEON EP1 series devices

  * KVM patches for s390x to provide facility bits 81 (ppa15) and 82 (bpb)
(LP: #1747090)
- KVM: s390: wire up bpb feature

  * artful 4.13 i386 kernels crash after memory hotplug remove (LP: #1747069)
- Revert "mm, memory_hotplug: do not associate hotadded memory to zones 
until
  online"

  * CVE-2017-5715 (Spectre v2 Intel)
- x86/feature: Enable the x86 feature to control Speculation
- x86/feature: Report presence of IBPB and IBRS control
- x86/enter: MACROS to set/clear IBRS and set IBPB
- x86/enter: Use IBRS on syscall and interrupts
- x86/idle: Disable IBRS entering idle and enable it on wakeup
- x86/idle: Disable IBRS when offlining cpu and re-enable on wakeup
- x86/mm: Set IBPB upon context switch
- x86/mm: Only set IBPB when the new thread cannot ptrace current thread
- x86/entry: Stuff RSB for entry to kernel for non-SMEP platform
- x86/kvm: add MSR_IA32_SPEC_CTRL and MSR_IA32_PRED_CMD to kvm
- x86/kvm: Set IBPB when switching VM
- x86/kvm: Toggle IBRS on VM entry and exit
- x86/spec_ctrl: Add sysctl knobs to enable/disable SPEC_CTRL feature
- x86/spec_ctrl: Add lock to serialize changes to ibrs and ibpb control
- x86/cpu/AMD: Add speculative control support for AMD
- x86/microcode: Extend post microcode reload to support IBPB feature
- KVM: SVM: Do not intercept new speculative control MSRs
- x86/svm: Set IBRS value on VM entry and exit
- x86/svm: Set IBPB when running a different VCPU
- KVM: x86: Add speculative control CPUID support for guests
- SAUCE: turn off IBPB when full retpoline is present

  * Artful 4.13 fixes for tun (LP: #1748846)
- tun: call dev_get_valid_name() before register_netdevice()
- tun: allow positive return values on dev_get_valid_name() call
- tun/tap: sanitize TUNSETSNDBUF input

  * boot failure on AMD Raven + WestonXT (LP: #1742759)
- SAUCE: drm/amdgpu: add atpx quirk handling (v2)

linux (4.13.0-33.36) artful; urgency=low

  * linux: 4.13.0-33.36 -proposed tracker (LP: #1746903)

  [ Stefan Bader ]
  * starting VMs causing retpoline4 to reboot (LP: #1747507) // CVE-2017-5715
(Spectre v2 retpoline)
- x86/retpoline: Fill RSB on context switch for affected CPUs
- x86/retpoline: Add LFENCE to the retpoline/RSB filling RSB macros
- x86/retpoline: Optimize inline assembler for vmexit_fill_RSB
- x86/retpoline: Remove the esp/rsp thunk
- x86/retpoline: Simplify vmexit_fill_RSB()

  * Missing install-time driver for QLogic QED 25/40/100Gb Ethernet NIC
(LP: #1743638)
- [d-i] Add qede to nic-modules udeb

  * hisi_sas: driver robustness fixes (LP: #1739807)
- scsi: hisi_sas: fix reset and port ID refresh issues
- scsi: hisi_sas: avoid potential v2 hw interrupt issue
- scsi: hisi_sas: fix v2 hw underflow residual value
- scsi: hisi_sas: add v2 hw DFX feature
- scsi: hisi_sas: add irq and tasklet cleanup in v2 hw
- scsi: hisi_sas: service interrupt ITCT_CLR interrupt in v2 hw
- scsi: hisi_sas: fix internal abort slot timeout bug
- scsi: hisi_sas: us start_phy in PHY_FUNC_LINK_RESET
- scsi: hisi_sas: fix NULL check in SMP abort task path
- scsi: hisi_sas: fix the risk of freeing slot twice
- scsi: hisi_sas: kill tasklet when destroying irq in v3 hw
- scsi: hisi_sas: complete all tasklets prior to host reset

  * [Artful/Zesty] ACPI APEI error handling bug fixes (LP: #1732990)
- ACPI: APEI: fix the wrong iteration of generic error status block
- ACPI / APEI: clear error status before acknowledging the error

  * [Zesty/Artful] On ARM64 PCIE physical function passthrough guest fails to
boot (LP: #1732804)
- vfio/pci: Virtualize Maximum Payload Size
- vfio/pci: Virtualize Maximum Read Request Size

  * hisi_sas: Add ATA command support for SMR disks (LP: #1739891)
- scsi: hisi_sas: support zone management commands

  * thunderx2: i2c driver PEC and ACPI clock fixes (LP: #1738073)
- ACPI / APD: Add clock frequency for ThunderX2 I2C controller
- i2c: xlp9xx: Get clock frequency with clk API
- i2c: xlp9xx: Handle 

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Launchpad Bug Tracker]

This bug was fixed in the package linux - 4.13.0-36.40

---
linux (4.13.0-36.40) artful; urgency=medium

  * linux: 4.13.0-36.40 -proposed tracker (LP: #1750010)

  * Rebuild without "CVE-2017-5754 ARM64 KPTI fixes" patch set

linux (4.13.0-35.39) artful; urgency=medium

  * linux: 4.13.0-35.39 -proposed tracker (LP: #1748743)

  * CVE-2017-5715 (Spectre v2 Intel)
- Revert "UBUNTU: SAUCE: turn off IBPB when full retpoline is present"
- SAUCE: turn off IBRS when full retpoline is present
- [Packaging] retpoline files must be sorted
- [Packaging] pull in retpoline files

linux (4.13.0-34.37) artful; urgency=medium

  * linux: 4.13.0-34.37 -proposed tracker (LP: #1748475)

  * libata: apply MAX_SEC_1024 to all LITEON EP1 series devices (LP: #1743053)
- libata: apply MAX_SEC_1024 to all LITEON EP1 series devices

  * KVM patches for s390x to provide facility bits 81 (ppa15) and 82 (bpb)
(LP: #1747090)
- KVM: s390: wire up bpb feature

  * artful 4.13 i386 kernels crash after memory hotplug remove (LP: #1747069)
- Revert "mm, memory_hotplug: do not associate hotadded memory to zones 
until
  online"

  * CVE-2017-5715 (Spectre v2 Intel)
- x86/feature: Enable the x86 feature to control Speculation
- x86/feature: Report presence of IBPB and IBRS control
- x86/enter: MACROS to set/clear IBRS and set IBPB
- x86/enter: Use IBRS on syscall and interrupts
- x86/idle: Disable IBRS entering idle and enable it on wakeup
- x86/idle: Disable IBRS when offlining cpu and re-enable on wakeup
- x86/mm: Set IBPB upon context switch
- x86/mm: Only set IBPB when the new thread cannot ptrace current thread
- x86/entry: Stuff RSB for entry to kernel for non-SMEP platform
- x86/kvm: add MSR_IA32_SPEC_CTRL and MSR_IA32_PRED_CMD to kvm
- x86/kvm: Set IBPB when switching VM
- x86/kvm: Toggle IBRS on VM entry and exit
- x86/spec_ctrl: Add sysctl knobs to enable/disable SPEC_CTRL feature
- x86/spec_ctrl: Add lock to serialize changes to ibrs and ibpb control
- x86/cpu/AMD: Add speculative control support for AMD
- x86/microcode: Extend post microcode reload to support IBPB feature
- KVM: SVM: Do not intercept new speculative control MSRs
- x86/svm: Set IBRS value on VM entry and exit
- x86/svm: Set IBPB when running a different VCPU
- KVM: x86: Add speculative control CPUID support for guests
- SAUCE: turn off IBPB when full retpoline is present

  * Artful 4.13 fixes for tun (LP: #1748846)
- tun: call dev_get_valid_name() before register_netdevice()
- tun: allow positive return values on dev_get_valid_name() call
- tun/tap: sanitize TUNSETSNDBUF input

  * boot failure on AMD Raven + WestonXT (LP: #1742759)
- SAUCE: drm/amdgpu: add atpx quirk handling (v2)

linux (4.13.0-33.36) artful; urgency=low

  * linux: 4.13.0-33.36 -proposed tracker (LP: #1746903)

  [ Stefan Bader ]
  * starting VMs causing retpoline4 to reboot (LP: #1747507) // CVE-2017-5715
(Spectre v2 retpoline)
- x86/retpoline: Fill RSB on context switch for affected CPUs
- x86/retpoline: Add LFENCE to the retpoline/RSB filling RSB macros
- x86/retpoline: Optimize inline assembler for vmexit_fill_RSB
- x86/retpoline: Remove the esp/rsp thunk
- x86/retpoline: Simplify vmexit_fill_RSB()

  * Missing install-time driver for QLogic QED 25/40/100Gb Ethernet NIC
(LP: #1743638)
- [d-i] Add qede to nic-modules udeb

  * hisi_sas: driver robustness fixes (LP: #1739807)
- scsi: hisi_sas: fix reset and port ID refresh issues
- scsi: hisi_sas: avoid potential v2 hw interrupt issue
- scsi: hisi_sas: fix v2 hw underflow residual value
- scsi: hisi_sas: add v2 hw DFX feature
- scsi: hisi_sas: add irq and tasklet cleanup in v2 hw
- scsi: hisi_sas: service interrupt ITCT_CLR interrupt in v2 hw
- scsi: hisi_sas: fix internal abort slot timeout bug
- scsi: hisi_sas: us start_phy in PHY_FUNC_LINK_RESET
- scsi: hisi_sas: fix NULL check in SMP abort task path
- scsi: hisi_sas: fix the risk of freeing slot twice
- scsi: hisi_sas: kill tasklet when destroying irq in v3 hw
- scsi: hisi_sas: complete all tasklets prior to host reset

  * [Artful/Zesty] ACPI APEI error handling bug fixes (LP: #1732990)
- ACPI: APEI: fix the wrong iteration of generic error status block
- ACPI / APEI: clear error status before acknowledging the error

  * [Zesty/Artful] On ARM64 PCIE physical function passthrough guest fails to
boot (LP: #1732804)
- vfio/pci: Virtualize Maximum Payload Size
- vfio/pci: Virtualize Maximum Read Request Size

  * hisi_sas: Add ATA command support for SMR disks (LP: #1739891)
- scsi: hisi_sas: support zone management commands

  * thunderx2: i2c driver PEC and ACPI clock fixes (LP: #1738073)
- ACPI / APD: Add clock frequency for ThunderX2 I2C controller
- i2c: xlp9xx: Get clock frequency with clk API
- i2c: xlp9xx: Handle 

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Checking if Trusty's fix is in -proposed (no comment given from Kernel
Team about it)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

# xenial - 4.4.0-112

[  OK  ] Reached target Shutdown.
[   70.272171]  connection3:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294907361, last ping 4294908612, now 4294909864
[   70.274189]  connection4:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294907361, last ping 4294908612, now 4294909864
[   70.288082]  connection5:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294907362, last ping 4294908614, now 4294909868
[   70.290066]  connection2:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294907363, last ping 4294908614, now 4294909868
[   70.292039]  connection1:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294907362, last ping 4294908614, now 4294909868



# xenial - 4.4.0-116

[  OK  ] Reached target Shutdown.
[   38.384175]  connection3:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294899386, last ping 4294900640, now 4294901892
[   38.386107]  connection1:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294899387, last ping 4294900640, now 4294901892
[   38.388023]  connection5:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294899387, last ping 4294900640, now 4294901892
[   38.388159]  connection4:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294899386, last ping 4294900640, now 4294901892
[   38.388159]  connection2:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294899387, last ping 4294900640, now 4294901892
[   89.825203] reboot: Restarting system

-> xenial-verification-done



# artful - 4.13.0-32

[  OK  ] Reached target Shutdown.
[  194.277712]  connection4:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294938304, last ping 4294939584, now 4294940864
[  194.279684]  connection2:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294938304, last ping 4294939584, now 4294940864
[  194.281483]  connection1:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294938304, last ping 4294939584, now 4294940864
[  194.283265]  connection5:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294938304, last ping 4294939584, now 4294940864
[  194.285067]  connection3:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294938304, last ping 4294939584, now 4294940864



# artful - 4.13.0-35

[  OK  ] Reached target Shutdown.
[  416.109989]  connection2:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294993856, last ping 4294995136, now 4294996416
[  416.111922]  connection1:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294993856, last ping 4294995136, now 4294996416
[  416.113869]  connection4:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294993856, last ping 4294995136, now 4294996416
[  416.115741]  connection3:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294993856, last ping 4294995136, now 4294996416
[  416.117650]  connection5:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294993856, last ping 4294995136, now 4294996416
[  469.614166] reboot: Restarting system

-> artful-verification-done

** Tags removed: verification-needed-artful
** Tags added: verification-done-artful

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

# xenial - 4.4.0-112

[  OK  ] Reached target Shutdown.
[   70.272171]  connection3:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294907361, last ping 4294908612, now 4294909864
[   70.274189]  connection4:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294907361, last ping 4294908612, now 4294909864
[   70.288082]  connection5:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294907362, last ping 4294908614, now 4294909868
[   70.290066]  connection2:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294907363, last ping 4294908614, now 4294909868
[   70.292039]  connection1:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294907362, last ping 4294908614, now 4294909868



# xenial - 4.4.0-116

[  OK  ] Reached target Shutdown.
[   38.384175]  connection3:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294899386, last ping 4294900640, now 4294901892
[   38.386107]  connection1:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294899387, last ping 4294900640, now 4294901892
[   38.388023]  connection5:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294899387, last ping 4294900640, now 4294901892
[   38.388159]  connection4:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294899386, last ping 4294900640, now 4294901892
[   38.388159]  connection2:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294899387, last ping 4294900640, now 4294901892
[   89.825203] reboot: Restarting system

-> xenial-verification-done


** Tags removed: verification-needed-xenial
** Tags added: verification-done-xenial

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Hello Kleber,

Thanks for the feedback. Working in verification.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Kleber Sacilotto de Souza]

This bug is awaiting verification that the kernel in -proposed solves
the problem. Please test the kernel and update this bug with the
results. If the problem is solved, change the tag 'verification-needed-
xenial' to 'verification-done-xenial'. If the problem still exists,
change the tag 'verification-needed-xenial' to 'verification-failed-
xenial'.

If verification is not done by 5 working days from today, this fix will
be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how
to enable and use -proposed. Thank you!

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Kleber Sacilotto de Souza]

This bug is awaiting verification that the kernel in -proposed solves
the problem. Please test the kernel and update this bug with the
results. If the problem is solved, change the tag 'verification-needed-
artful' to 'verification-done-artful'. If the problem still exists,
change the tag 'verification-needed-artful' to 'verification-failed-
artful'.

If verification is not done by 5 working days from today, this fix will
be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how
to enable and use -proposed. Thank you!


** Tags added: verification-needed-artful

** Tags added: verification-needed-xenial

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Yep, soon kernel team will advise (here) about this patch been released
in a specific kernel version that is in -proposed and will call out for
testing/verification. I'll give all feedback here also and will mark, if
appropriate, case as "verification-done" so new kernel is marked as
solving this issue.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

RE: [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Matt Schulte]

Thanks, and will notifications appear here in the bug during these
stages?

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Hello Matt, this is part of the SRU procedure
(https://wiki.ubuntu.com/StableReleaseUpdates). For the kernel, the SRU
is a bit different
(https://wiki.ubuntu.com/Kernel/Dev/StablePatchFormat,
https://wiki.ubuntu.com/Kernel/StableReleaseCadence). Now that the bug
has been marked as "Fix Committed" is means that the patch has been
merged in the "master-next" branch of every one of the following Ubuntu
versions:

Changed in linux (Ubuntu Artful):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Xenial):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Trusty):
status: In Progress → Fix Committed

When the new kernel package is built, it will be placed into -proposed
for each of the Ubuntu versions and you will be able to test the kernel
and provide feedback here. Since I was the one that fixed it, I'll
verify each of the kernels to make sure it fixed the issue. After
verification, and at least 7 days in -proposed, the packages shall be
moved to -updates.

Hope it helps you understand the fixing steps. Cheers o/

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

BTW, kernel team usually updates all affecting cases with the version
that was built that address this issue (so you know what to text in
-proposed).

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Matt Schulte]

I apologize for my ignorance, but now that tall these check ins have
occurred, how do I know when an installable package has been built and
is available in the various repos?

And thank you @inaddy for the diligence getting this one resolved.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Khaled El Mously]

** Changed in: linux (Ubuntu Artful)
   Status: In Progress => Fix Committed

** Changed in: linux (Ubuntu Xenial)
   Status: In Progress => Fix Committed

** Changed in: linux (Ubuntu Trusty)
   Status: In Progress => Fix Committed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Commit has been merged upstream:

commit d754941225a7dbc61f6dd2173fa9498049f9a7ee
Author: Rafael David Tinoco 
Date:   Thu Dec 7 19:59:13 2017 -0200

scsi: libiscsi: Allow sd_shutdown on bad transport

If, for any reason, userland shuts down iscsi transport interfaces
before proper logouts - like when logging in to LUNs manually, without
logging out on server shutdown, or when automated scripts can't
umount/logout from logged LUNs - kernel will hang forever on its
sd_sync_cache() logic, after issuing the SYNCHRONIZE_CACHE cmd to all
still existent paths.

PID: 1 TASK: 8801a69b8000 CPU: 1 COMMAND: "systemd-shutdow"
 #0 [8801a69c3a30] __schedule at 8183e9ee
 #1 [8801a69c3a80] schedule at 8183f0d5
 #2 [8801a69c3a98] schedule_timeout at 81842199
 #3 [8801a69c3b40] io_schedule_timeout at 8183e604
 #4 [8801a69c3b70] wait_for_completion_io_timeout at 8183fc6c
 #5 [8801a69c3bd0] blk_execute_rq at 813cfe10
 #6 [8801a69c3c88] scsi_execute at 815c3fc7
 #7 [8801a69c3cc8] scsi_execute_req_flags at 815c60fe
 #8 [8801a69c3d30] sd_sync_cache at 815d37d7
 #9 [8801a69c3da8] sd_shutdown at 815d3c3c

This happens because iscsi_eh_cmd_timed_out(), the transport layer
timeout helper, would tell the queue timeout function (scsi_times_out)
to reset the request timer over and over, until the session state is
back to logged in state. Unfortunately, during server shutdown, this
might never happen again.

Other option would be "not to handle" the issue in the transport
layer. That would trigger the error handler logic, which would also need
the session state to be logged in again.

Best option, for such case, is to tell upper layers that the command was
handled during the transport layer error handler helper, marking it as
DID_NO_CONNECT, which will allow completion and inform about the
problem.

After the session was marked as ISCSI_STATE_FAILED, due to the first
timeout during the server shutdown phase, all subsequent cmds will fail
to be queued, allowing upper logic to fail faster.

Signed-off-by: Rafael David Tinoco 
Reviewed-by: Lee Duncan 
Signed-off-by: Martin K. Petersen 

Applied in Bionic:

https://lists.ubuntu.com/archives/kernel-team/2018-January/089509.html

And Fix Committed.

Ack-ed for Trusty, Xenial, Artful:

https://lists.ubuntu.com/archives/kernel-team/2018-January/089774.html
https://lists.ubuntu.com/archives/kernel-team/2018-February/089809.html

Waiting to be Fix Committed there. A soon as it is released to -proposed
I'll verify it and provide feedback so it can be Fix Released.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Seth Forshee]

** Changed in: linux (Ubuntu Bionic)
   Status: In Progress => Fix Committed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

I have just submitted the SRU proposal to kernel team mailing list. I'll
wait for the SRU to be made and this public bug to be marked as Fix
Committed before verifying the fix again for the final release.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

** Changed in: open-iscsi (Ubuntu Trusty)
   Status: New => Opinion

** Changed in: linux (Ubuntu Trusty)
   Status: New => In Progress

** Changed in: linux (Ubuntu Trusty)
 Assignee: (unassigned) => Rafael David Tinoco (inaddy)

** Changed in: linux (Ubuntu Trusty)
   Importance: Undecided => Medium

** Changed in: open-iscsi (Ubuntu Trusty)
   Importance: Undecided => Medium

** Changed in: open-iscsi (Ubuntu Trusty)
 Assignee: (unassigned) => Rafael David Tinoco (inaddy)

** No longer affects: open-iscsi (Ubuntu Zesty)

** No longer affects: linux (Ubuntu Zesty)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Dan Streetman]

** Also affects: open-iscsi (Ubuntu Bionic)
   Importance: Medium
 Assignee: Rafael David Tinoco (inaddy)
   Status: Opinion

** Also affects: linux (Ubuntu Bionic)
   Importance: Medium
 Assignee: Rafael David Tinoco (inaddy)
   Status: In Progress

** Also affects: open-iscsi (Ubuntu Trusty)
   Importance: Undecided
   Status: New

** Also affects: linux (Ubuntu Trusty)
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

** Description changed:

+ [Impact]
+ 
+  * open-iscsi users might face hangs during OS shutdown.
+  * hangs can be caused by manual iscsi configuration/setup.
+  * hangs can also be caused by bad systemd unit ordering.
+  * if transport layer interface vanishes before lun is
+disconnected, then the hang will happen.
+  * check comment #89 for the fix decision.
+  
+ [Test Case]
+ 
+  * a simple way of reproducing the kernel hang is to disable
+the open-iscsi logouts. this simulates a situation when
+a service has shutdown the network interface, used by
+the transport layer, before proper iscsi logout was done.
+ 
+$ log into all iscsi luns
+   
+$ systemctl edit --full open-iscsi.service
+...
+#ExecStop=/lib/open-iscsi/logout-all.sh
+...
+   
+$ sudo reboot # this will make server to hang forever
+  # on shutdown
+ 
+ [Regression Potential]
+ 
+  * the regression is low because the change acts on the iscsi
+transport layer code ONLY when the server is in shutdown 
+state.
+ 
+  * any error in logic would only appear during shutdown and
+would not cause any harm to data.
+ 
+ [Other Info]
+  
+  * ORIGINAL BUG DESCRIPTION
+ 
  I have 4 servers running the latest 16.04 updates from the development
  branch (as of right now).
  
  Each server is connected to NetApp storage using iscsi software
  initiator.  There are a total of 56 volumes spread across two NetApp
  arrays.  Each volume has 4 paths available to it which are being managed
  by device mapper.
  
  While logged into the iscsi sessions all I have to do is reboot the
  server and I get a hang.
  
  I see a message that says:
  
    "Reached target Shutdown"
  
  followed by
  
    "systemd-shutdown[1]: Failed to finalize DM devices, ignoring"
  
  and then I see 8 lines that say:
  
    "connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
    "connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
    "connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
    "connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
    "connection5:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
    "connection6:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
    "connection7:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
    "connection8:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
    NOTE: the actual values of the *'s differ for each line above.
  
  This seems like a bug somewhere but I am unaware of any additional
  logging that I could turn on to pinpoint the problem.
  
  Note I also have similar setups that are not doing iscsi and they don't
  have this problem.
  
  Here is a screenshot of what I see on the shell when I try to reboot:
  
  (https://launchpadlibrarian.net/291303059/Screenshot.jpg)
  
  This is being tracked in NetApp bug tracker CQ number 860251.
  
  If I log out of all iscsi sessions before rebooting then I do not
  experience the hang:
  
  iscsiadm -m node -U all
  
  We are wondering if this could be some kind of shutdown ordering
  problem.  Like the network devices have already disappeared and then
  iscsi tries to perform some operation (hence the ping timeouts).

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Changed open-iscsi to opinion since I choose to fix the kernel instead
of fixing Userland. No matter what you do in userland, the kernel had
space for freezing and hanging in different scenarios depending on how
userland disconnected the transport layer. I kept the "linux" source
package as being affected and will SRU it through Ubuntu Kernel Team.

** Changed in: open-iscsi (Ubuntu Xenial)
   Status: In Progress => Opinion

** Changed in: open-iscsi (Ubuntu)
   Status: In Progress => Opinion

** Changed in: open-iscsi (Ubuntu Zesty)
   Status: In Progress => Opinion

** Changed in: open-iscsi (Ubuntu Artful)
   Status: In Progress => Opinion

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

** Tags added: sts

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

The scsi/iscsi team has accepted by fix proposal and merged the patch
for 4.16 merge window:

https://www.mail-archive.com/open-iscsi@googlegroups.com/msg10111.html

I'll propose the cherry-pick as SRU for Ubuntu kernels as soon as it is
merged.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

RE: [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Matt Schulte]

That's awesome sir, thank you for your diligence.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

I have proposed a fix to the kernel iscsi transport code. You can follow
upstream discussion, if any, in the following address:

https://marc.info/?t=15126839768=1=2

The patch proposal is here:

https://marc.info/?l=linux-scsi=151268396227285=2

If upstream chooses to accept my proposal, which fixes the issue like
demonstrated here:

https://marc.info/?l=linux-scsi=151274369910906=2

Then I'll propose Canonical kernel-team a SRU based on the upstream
patch.

For now this case is on hold for upstream discussion/decision.

Thank you
-Rafael

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Long story short -> https://goo.gl/vPyh8C

Basically the block device enqueues the last request (a SYNC scsi
command coming from sd_shutdown) for every scsi device there is on the
system. Unfortunately, since the OS is shutting down, in between the
block request and its execution, we have userland (systemd) killing
iscsid, without proper logout, and/or removing the network.

What happens next is that the mid layer (SCSI) tries to deliver the
request through the transport layer (iscsi_tcp_sw) but it fails since
the transport layer checks the session status and finds out that the
session is not in LOGIN state.

The default behaviour of the transport layer (iscsi_tcp_sw) in such
situation is to tell the mid-layer to keep resetting the request timeout
timer while it tries to recover (something that will never happen
because the network is gone).

Changing that default behavior to state that the scsi command was NOT
handled by the transport layer (iscsi_tcp_sw) implies in making the scsi
timeout function to try to "abort" the scsi command, which also creates
other commands that will timeout because of the transport layer.

Best scenario so far was to change BLK_EH_NOT_HANDLED for BLK_EH_HANDLED
in the scsi_times_out function and make the kernel to be able to
shutdown. By doing that, I'm confirming to block device something that
DID NOT happen, meaning that the command never left the transport layer.

This might be ONE of possible ways to fix this: I can mark in the
transport layer that I have timed-out DURING the shutdown procedure,
cancelling all the block device requests without having to invoke the
scsi error handling mechanism, generating more traffic in transport
layer (what would also cause more timeouts, causing a loop in the
problematic sequence).

Anyway, I'll get back to this next week and hopefully identify best
course of action.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Hello Amanda,

I'm actively working into this, will provide better feedback soon. The
script is just a workaround for now, so people can shut down properly,
if they are facing this.

Despite what userland does, the kernel will always potentially hang with
iscsi sessions left opened during shutdown. If, after following the
setup steps, you still faced the issue, its possible that your shutdown
systemd unit ordering is wrong (your mount points are being unmounted
before they are released). I'm working in the kernel issue, so, it shall
cover all the userland-created situations.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Amanda Gartner]

I followed all of the setup steps, but was still encountering the ping
timeout issue at reboot. I then ran the script
http://pastebin.ubuntu.com/25894592/ you provided and am no longer
hitting the issue. Was the script supposed to fix the issue? Please let
me know. Thank you!

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Hello Matthijs

Unfortunately the best way to make this not to happen is by fixing the
kernel hang situation, when kernel calls sd_sync_cache() to every
configured device before the shutdown. There is a single I/O cmd hanging
in all scsi paths and the I/O error is never propagated to block layer
(despite iscsi having proper I/O error settings). I'm finishing
analysing some kernel dumps so I can finally understand what is
happening in the transport layer (this happens with more recent kernels
also).

The workaround was to create a script that would restore the iscsi
connection, wait for the login to happen again and the paths are back
online, and cleanly logout, allowing the sd_sync_cache() operation to be
finalized.

If you are facing this problem, I know for sure that your iscsi
connections are not being finalized before the network is off. This
means that you have to pay attention on how you configured your iscsi
disks:

- guarantee that iscsiadm was configured with "interfaces" so it works
on startup:

  sudo iscsiadm -m iface -I ens4 --op=new -n iface.hwaddress -v 
52:54:00:b4:21:bb
  sudo iscsiadm -m iface -I ens7 --op=new -n iface.hwaddress -v 
52:54:00:c2:34:1b

- the discovery/login has to be made AFTER the iscsiadm had interfaces
added

  sudo iscsiadm -m discovery --op=new --op=del --type sendtargets --portal 
$SERVER1
  sudo iscsiadm -m discovery --op=new --op=del --type sendtargets --portal 
$SERVER2

  # iscsiadm -m node --loginall=automatic HAS TO WORK or else init
scripts will fail

  http://pastebin.ubuntu.com/25894472/

- configure the volumes in /etc/fstab with "_netdev" parameter for
systemd unit ordering

  LABEL=BLUE /blue ext4 defaults,_netdev 0 1
  LABEL=GREEN /green ext4 defaults,_netdev 0 1
  LABEL=PURPLE /purple ext4 defaults,_netdev 0 1
  LABEL=RED /red ext4 defaults,_netdev 0 1
  LABEL=YELLOW /yellow ext4 defaults,_netdev 0 1

You have to make sure open-iscsi and iscsid systemd units are started
after the network is available and are stopped before they disappear.
That might be your problem, if configuration above is correct.

inaddy@iscsihang:~$ systemctl edit --full iscsid.service
inaddy@iscsihang:~$ systemctl edit --full open-iscsi.service

The defaults are:

[Unit]
Description=iSCSI initiator daemon (iscsid)
Documentation=man:iscsid(8)
Wants=network-online.target remote-fs-pre.target
Before=remote-fs-pre.target
After=network.target network-online.target

and

[Unit]
Description=Login to default iSCSI targets
Documentation=man:iscsiadm(8) man:iscsid(8)
Wants=network-online.target remote-fs-pre.target iscsid.service
After=network-online.target iscsid.service
Before=remote-fs-pre.target

So you can see that iscsid.service runs BEFORE open-iscsi.service.  In
my case, I'm configuring network using rc-local.service (since this is
my lab) and I had to guarantee the ordering also:

If, after configuring your system like this, you still face problems,
you can use this script:

http://pastebin.ubuntu.com/25894592/

And provide me the DEBUG=/.shutdown.log file, created after its
execution, attached to this launchpad case. Its likely that you will
have hang iscsi connections for some reason (services ordering, lack of
volumes in fstab so umounts are not done, etc).

Hope it helps for now.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Matthijs Hoekstra]

Even with the script a shutdown/reboot takes a very long time. Any
expectations when a fix will be ready? I hit this all the time on 2
installed servers.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Francis Ginther]

** Tags added: id-598b6543459f8ccf5dfbc04c

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

I created a workaround for this issue while its being worked on:

http://pastebin.ubuntu.com/25519820/

Create a file: /lib/systemd/system-shutdown/debug.sh and place the
contents in it.

This workaround (based on some ideas from sil2100 and slangasek for
iscsi / umount) is basically bringing the interfaces up and remounting /
so it can finally cleanup everything.

Things to note:

- It ONLY RUNS if there are iscsi leftovers
- It uses ifupdown only for networking (/etc/network/interfaces)
- It has to remount / to do networking and to run iscsid
- If it fails to bring network it will hang like before (kernel issue)
- It waits for iscsi to be logged in again (might take awhile to shutdown)
- If logout fails, it hangs again (unless the network is left configured, i 
could change it)

What is this script different then altering open-iscsi.service ?

It runs at the very end of systemd shutdown and it is very unlikely that
there are any services holding references to iscsi mounts, disallowing
them to be logged out.

Now I'll test Debian SID and check if this is found there, to open a bug in 
Debian project as well.
Before moving on into open-iscsi services - to create a cleanup unit file for 
the open-iscsi package, like this workaround - i'll dig into the kernel issue. 
I'm afraid no fix will be as good as making sure kernel let the queued I/O cmd 
go. I have also to make sure this workaround is changed to allow root iscsi to 
be logged out.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Brian Murray]

** Tags added: rls-aa-notfixing

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

** Changed in: open-iscsi (Ubuntu Xenial)
 Assignee: (unassigned) => Rafael David Tinoco (inaddy)

** Changed in: open-iscsi (Ubuntu Zesty)
 Assignee: (unassigned) => Rafael David Tinoco (inaddy)

** Changed in: open-iscsi (Ubuntu Artful)
 Assignee: (unassigned) => Rafael David Tinoco (inaddy)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

2 things for previous comment #75:

1)
I pasted wrong pastebin URL. I'll describe the fix: its a service unit file 
that runs at the end of shutdown, right before it, re-using /run (tmpfs). 
Service has a preparation script that copies binaries from initrd image - so 
there is a minimum execution environment available - and runs the second 
script. Second script initializes iscsid daemon and waits in a loop for all the 
portals to be ONLINE and existing sessions to be LOGGED. After that it does the 
logout on all existing sessions (this case this would be leftovers from my 
previous comment AND / root disks).

2)
I tested this approach and it works good for iscsi root disks because there is 
a network interface that stays connected all the time. In the case of leftovers 
from previous unsuccessful umount attempts, the network.target would be already 
gone (with networkd / ifupdown interfaces already shutdown). IF I keep my 
interfaces configured, then this approach works for root disks AND for 
leftovers. For this approach to be successful for this case, this "cleanup" 
service would have to raise existing interfaces up again for the iscsid to 
login and logout to work (OR the kernel hang be resolved, then leftovers would 
be just left there, without logouts).

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Unrelated to systemd. Related to open-iscsi systemd unit files and
kernel iscsi transport logout during shutdown.

** Changed in: systemd (Ubuntu Artful)
   Importance: High => Medium

** Changed in: systemd (Ubuntu Artful)
   Status: Confirmed => Triaged

** Changed in: systemd (Ubuntu Artful)
 Assignee: Nish Aravamudan (nacc) => (unassigned)

** No longer affects: systemd (Ubuntu Xenial)

** No longer affects: systemd (Ubuntu Zesty)

** No longer affects: systemd (Ubuntu Artful)

** Changed in: linux (Ubuntu Xenial)
 Assignee: (unassigned) => Rafael David Tinoco (inaddy)

** Changed in: linux (Ubuntu Zesty)
 Assignee: (unassigned) => Rafael David Tinoco (inaddy)

** Changed in: linux (Ubuntu Artful)
 Assignee: (unassigned) => Rafael David Tinoco (inaddy)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Unrelated to systemd. Related to open-iscsi systemd unit files and
kernel iscsi transport logout during shutdown.

** Changed in: open-iscsi (Ubuntu Xenial)
   Importance: Undecided => Medium

** Changed in: open-iscsi (Ubuntu Xenial)
   Status: New => In Progress

** Changed in: open-iscsi (Ubuntu Zesty)
   Importance: Undecided => Medium

** Changed in: open-iscsi (Ubuntu Zesty)
   Status: New => In Progress

** Changed in: open-iscsi (Ubuntu Artful)
   Importance: Undecided => Medium

** Changed in: open-iscsi (Ubuntu Artful)
   Status: New => In Progress

** Changed in: linux (Ubuntu Xenial)
   Importance: Undecided => Medium

** Changed in: linux (Ubuntu Xenial)
   Status: Incomplete => In Progress

** Changed in: linux (Ubuntu Zesty)
   Importance: Undecided => Medium

** Changed in: linux (Ubuntu Zesty)
   Status: Incomplete => In Progress

** Changed in: linux (Ubuntu Artful)
   Importance: Undecided => Medium

** Changed in: linux (Ubuntu Artful)
   Status: Incomplete => In Progress

** No longer affects: systemd (Ubuntu)

** Changed in: systemd (Ubuntu Xenial)
   Status: In Progress => Triaged

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Unrelated to systemd. Related to open-iscsi systemd unit files and
kernel iscsi transport logout during shutdown.

** Changed in: systemd (Ubuntu Zesty)
   Importance: Undecided => Medium

** Changed in: systemd (Ubuntu Zesty)
   Status: New => Triaged

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

## SUMMARY 1

Affects open-iscsi (NOT systemd):

* ISCSID NEEDS TO BE UP & RUNNING (AND LOGGED TO PORTAL) FOR THE LOGOUT
TO WORK

https://pastebin.canonical.com/197359/

Daemon shutdown documentation in iscsiadm command says (about stopping iscsid 
daemon with iscsiadm -k): 
""" ... This will immediately stop all iscsid operations and shutdown iscsid. 
It does not logout any sessions. Running this command is the same as doing 
"killall iscsid". Neither should normally be used, because if iscsid is doing 
error recovery or if there is an error while iscsid is not running, the system 
may not be able to recover. This command and iscsid's SIGTERM handling are 
experimental. """

Basically meaning you're not able to shutdown the daemon in a clean
manner if you have connected sessions. And basically, checking the
pastebin, you see the need for running iscsid daemon to logout through
iscsiadm command.

So from:
 
(systemctl edit --full open-iscsi.service)
...
Wants=network-online.target remote-fs-pre.target iscsid.service
After=network-online.target iscsid.service
Before=remote-fs-pre.target
...
ExecStartPre=/bin/systemctl --quiet is-active iscsid.service
ExecStart=/sbin/iscsiadm -m node --loginall=automatic
ExecStart=/lib/open-iscsi/activate-storage.sh
ExecStop=/lib/open-iscsi/umountiscsi.sh
ExecStop=/bin/sync
ExecStop=/lib/open-iscsi/logout-all.sh
...

and

(systemctl edit --full iscsid.service)
...
Wants=network-online.target remote-fs-pre.target
Before=remote-fs-pre.target
After=network.target network-online.target
...
ExecStartPre=/lib/open-iscsi/startup-checks.sh
ExecStart=/sbin/iscsid
ExecStop=/sbin/iscsiadm -k 0 2
...

If umountiscsi.sh (it doesn't kill applications using a mount point)
can't umount a mount point (from fstab generator or a mount unit) that
is still being used, the logout-all.sh will fail for logging out the
session of the disk being used. But this service unit would be
considered stopped, allowing "iscsid.service" unit to also be shutdown
(killing the iscsid daemon and leaving an opened iscsi session).

This would be problematic for iscsi root disks AND for the kernel issue
(comment #73), and this is being taken care by Francis Ginther
(fginther) in the next iscsi SRU: https://pastebin.canonical.com/196463/
by creating an iscsi-cleanup.service that will run a script that logs
out all remaining iscsi sessions (including /) right before kernel
shutdown is called by systemd-shutdown like showed in comment #73.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Eric Desrochers]

** Also affects: open-iscsi (Ubuntu Artful)
   Importance: Undecided
   Status: New

** Also affects: linux (Ubuntu Artful)
   Importance: Undecided
   Status: New

** Also affects: systemd (Ubuntu Artful)
   Importance: High
 Assignee: Nish Aravamudan (nacc)
   Status: Confirmed

** Also affects: open-iscsi (Ubuntu Zesty)
   Importance: Undecided
   Status: New

** Also affects: linux (Ubuntu Zesty)
   Importance: Undecided
   Status: New

** Also affects: systemd (Ubuntu Zesty)
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

** Also affects: open-iscsi (Ubuntu)
   Importance: Undecided
   Status: New

** Also affects: linux (Ubuntu)
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Okay, I removed almost everything out of equation:

- removed networking interfaces from systemd
- removed open-iscsi login/logout logic from systemd
- used a single network interface for the logins, on a single iscsi portal

was still able to reproduce the issue by:

- doing a simple login after configuring the network device: 
  ./net-start.sh ; sleep 1 ; sudo iscsiadm -m node --login

- shutting down the network device before any logout:
  ./net-stop.sh ; sudo shutdown -h now

There was no systemd service order in play (in between open-iscsi,
iscsid and networking / ifupdown scripts) and I was able to cause the
same issue 100%, same messages, same symptoms. This tells us that,
definitely turning down the network while still logged iscsi sessions is
causing the hangs (check #3 for why).

Summary of Causes

1)

network shuts down -> iscsi luns are logged out (attempt) -> iscsid daemon is 
shutdown
or
network shuts down -> iscsi daemon is shutdown -> iscsi luns are logged out

- logout would fail due to lack of network
- iscsi daemon would die and/or logout wouldn't work
- iscsi sessions would still be there
- system would hang (transport layer can't logout)

2)

iscsi daemon is shutdown -> iscsi luns are logged out -> network shuts
down

- this would cause the bug i'm mentioning: daemon dies, logout doesn't work.
- some iscsi sessions would still be there
- system would hang (transport layer can't logout)
- (to check: open-iscsi.service ExecStop= running in parallel ?!?)

3)

I used KVM watchdog virtual device + NMI from host to crash the guest
during the hang

http://pastebin.ubuntu.com/25394744/

And, finally, the hang is because kernel is hanged during its shutdown
logic.

> 0  0   0  81e11500  RU   0.0   0  0  [swapper/0]
> 0  0   1  8801a6a20e00  RU   0.0   0  0  [swapper/1]
> 0  0   2  8801a6a21c00  RU   0.0   0  0  [swapper/2]
> 0  0   3  8801a6a22a00  RU   0.0   0  0  [swapper/3]

ALL CPUs were idling during the hang.

crash> runq
CPU 0 RUNQUEUE: 8801ae016e00
  CURRENT: PID: 0  TASK: 81e11500  COMMAND: "swapper/0"
  RT PRIO_ARRAY: 8801ae016fb0
 [no tasks queued]
  CFS RB_ROOT: 8801ae016e98
 [no tasks queued]

CPU 1 RUNQUEUE: 8801ae096e00
  CURRENT: PID: 0  TASK: 8801a6a20e00  COMMAND: "swapper/1"
  RT PRIO_ARRAY: 8801ae096fb0
 [no tasks queued]
  CFS RB_ROOT: 8801ae096e98
 [no tasks queued]

CPU 2 RUNQUEUE: 8801ae116e00
  CURRENT: PID: 0  TASK: 8801a6a21c00  COMMAND: "swapper/2"
  RT PRIO_ARRAY: 8801ae116fb0
 [no tasks queued]
  CFS RB_ROOT: 8801ae116e98
 [no tasks queued]

CPU 3 RUNQUEUE: 8801ae196e00
  CURRENT: PID: 0  TASK: 8801a6a22a00  COMMAND: "swapper/3"
  RT PRIO_ARRAY: 8801ae196fb0
 [no tasks queued]
  CFS RB_ROOT: 8801ae196e98
 [no tasks queued]

NO task was scheduled to run.

crash> ps -u
   PIDPPID  CPU   TASKST  %MEM VSZRSS  COMM
  1  0   1  8801a69b8000  UN   0.0   15484   3204  systemd-shutdow

There was just ONE SINGLE user task running (systemd-shutdown)

crash> set 8801a69b8000
PID: 1
COMMAND: "systemd-shutdow"
   TASK: 8801a69b8000  [THREAD_INFO: 8801a69c]
CPU: 1
  STATE: TASK_UNINTERRUPTIBLE 
crash> bt
PID: 1  TASK: 8801a69b8000  CPU: 1   COMMAND: "systemd-shutdow"
 #0 [8801a69c3a30] __schedule at 8183e9ee
 #1 [8801a69c3a80] schedule at 8183f0d5
 #2 [8801a69c3a98] schedule_timeout at 81842199
 #3 [8801a69c3b40] io_schedule_timeout at 8183e604
 #4 [8801a69c3b70] wait_for_completion_io_timeout at 8183fc6c
 #5 [8801a69c3bd0] blk_execute_rq at 813cfe10
 #6 [8801a69c3c88] scsi_execute at 815c3fc7
 #7 [8801a69c3cc8] scsi_execute_req_flags at 815c60fe
 #8 [8801a69c3d30] sd_sync_cache at 815d37d7
 #9 [8801a69c3da8] sd_shutdown at 815d3c3c
#10 [8801a69c3dc0] device_shutdown at 8156112c
#11 [8801a69c3df0] kernel_power_off at 810a32f5
#12 [8801a69c3e00] SYSC_reboot at 810a34d3
#13 [8801a69c3f40] sys_reboot at 810a359e
#14 [8801a69c3f50] entry_SYSCALL_64_fastpath at 818431f2
RIP: 7face7d8f856  RSP: 7ffd4c271d08  RFLAGS: 0202
RAX: ffda  RBX: 7ffd4c271240  RCX: 7face7d8f856
RDX: 4321fedc  RSI: 28121969  RDI: fee1dead
RBP: 7ffd4c271230   R8: 1400   R9: 0005
R10: 7face88d36c0  R11: 0202  R12: 7ffd4c2713d0
R13: 00b7e6795c83  R14: 7ffd4c271c30  R15: 0001
ORIG_RAX: 00a9  CS: 0033  SS: 002b

And it called the "kernel_power_off" logic from a system call.

This logic iterates over all devices calling:

if (dev->bus && dev->bus->shutdown) {
dev->bus->shutdown(dev);
} else if (dev->driver && 

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

For now, the workaround for this bug is this:

inaddy@iscsihang:~$ sudo su -

# disconnect what iscsid has mapped
root@iscsihang:~$ iscsiadm -m node --logoutall=all

# disconnect all leftovers
root@iscsihang:~$ for file in `find /sys/devices -name *delete`; do echo 1 > 
$file; done

# shutdown
root@iscsihang:~$ shutdown -h now

(as long as you don't have iscsi root device)

Concentrating efforts in why systemd doesn't finish when iscsi
connections and paths are online. Will check why (even recent version)
open-iscsi looses track of connections if killed/restarted by a new
login.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Matt Schulte]

Excellent, thank you for the details.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

This is what I discovered:

This systemd issue is caused because there are iscsi connections still
logged in (not because of multipath, bonding, vlans, etc). One way of
reproducing this is by simply editing the open-iscsi.service unit and
disabling the logout script. It will, for me in all times, cause the
same hang as we are talking to.

inaddy@iscsihang:~$ sudo systemctl edit --full open-iscsi.service

...
ExecStop=/lib/open-iscsi/umountiscsi.sh
ExecStop=/bin/sync
#ExecStop=/lib/open-iscsi/logout-all.sh

But of course no one is doing this. So, reading logout-all you will
discover that it logs out all paths but it has some caveats:
ISCSI_ROOT_KEEP_ALL_SESSIONS_AT_SHUTDOWN (from /etc/default/open-iscsi)
and /run/open-iscsi/shutdown-keep-sessions would cause those iscsi
sessions not to be logged out.

Still, I wasn't using neither to reproduce so I found out something
else. Disabling both systemd services:

inaddy@iscsihang:~$ systemctl list-unit-files --full | grep iscsi
iscsi.service   disabled (alias for iscsid)
iscsid.service  disabled
open-iscsi.service  disabled

I was still able to reproduce the issue:

http://pastebin.ubuntu.com/25388348/

By issuing a logout when the iscsid daemon is gone. It looks like iscsid
is lost after it has died/been killed on the sessions that were
previously established. If you try to login again, it will be totally
lost when trying to logout, causing leftovers (opened iscsi sessions)
causing systemd to hang (even not having daemons running, in this case).

I'm investigating:

- Should systemd hang on those leftovers ? Since there are caveats that
explicitly allow you to leave opened sessions, for / for example, I
doubt it.

- Does iscsid daemon contain a bug that causes sessions to remain un-
mapped and opened if daemon is restarted without a logout of previously
opened sessions ?

Now that I'm able to reproduce this at will, I'll be able to answer
those questions soon.

Best,
Rafael

PS: Matt, I asked partnership program managers to reach out to
understand better your future needs.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Matt Schulte]

As I said I don't have one up right now, but Comment 3 above has my
interfaces for one of the recreates.

Nothing bothered me in any way.  I wanted to make sure that you knew
that I wasn't one guy who was looking to fix his one system, that's all.

You said "Usually those type of "partnership" product
enablements/fixes"...what do you mean by that?  Is there some avenue
that I could be pursuing to get my bugs addressed sooner?

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Might be unrelated, but I discovered that:

## root namespace

root@iscsihang:~$ mount | egrep -E "sd.*(xfs|ext4)"
/dev/sda1 on /ext4 type ext4 (rw,relatime,stripe=32,data=ordered,_netdev)
/dev/sdb1 on /xfs type xfs (rw,relatime,attr2,inode64,noquota,_netdev)

## specific mnt namespace

root@iscsihang:~$ ip netns exec testnamespace /bin/bash
root@iscsihang:~$ sudo umount /xfs
root@iscsihang:~$ sudo umount /ext4

## root namespace

root@iscsihang:~$ mount | egrep -E "sd.*(xfs|ext4)"
/dev/sda1 on /ext4 type ext4 (rw,relatime,stripe=32,data=ordered)
/dev/sdb1 on /xfs type xfs (rw,relatime,attr2,inode64,noquota)

When a mount namespace (mntns) umounts a --shared filesystem from the
root namespace, the _netdev flag is gone. That might stop init scripts
from correctly identifying when those filesystems should be umounted
(before the network is gone, for example) -> this might be "another"
bug.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Hypothesis,

Test (2) - The error is propagated to upper layers after X seconds.

In this case I'm testing:

$ sudo iscsiadm -m node -o show | grep node.session.timeo.replac
node.session.timeo.replacement_timeout = 60
node.session.timeo.replacement_timeout = 60

And same "locking" behavior is observed, but for a certain period of
time. Now, shutdown procedure stayed locked for exact 60 seconds before
allowing the machine to shutdown. This means that the systemd logic
NEEDS either a clear shutdown, for the _netdev filesystems, OR the error
to be propagated so the umount() can go on:

[   17.612020]  connection1:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294894197, last ping 4294895448, now 4294896700
[   17.644128]  connection2:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294894204, last ping 4294895456, now 4294896708

<60 seconds>

[   78.044264] sd 2:0:0:1: rejecting I/O to offline device
[   78.045297] blk_update_request: I/O error, dev sdb, sector 0
[   78.046315] sd 3:0:0:1: rejecting I/O to offline device
[   78.046357] sd 2:0:0:1: rejecting I/O to offline device
[   78.046363] blk_update_request: I/O error, dev sdb, sector 0
[   78.046487] sd 2:0:0:1: rejecting I/O to offline device
[   78.050117] blk_update_request: I/O error, dev sda, sector 20481074
[   78.051219] XFS (sda1): metadata I/O error: block 0x1387c32 ("xlog_iodone") 
error 5 numblks 64
[   78.052727] XFS (sda1): Log I/O Error Detected.  Shutting down filesystem
[   78.053901] XFS (sda1): Please umount the filesystem and rectify the 
problem(s)
[   78.060723] sd 3:0:0:1: rejecting I/O to offline device
[   78.061759] blk_update_request: I/O error, dev sda, sector 0




[  OK  ] Unmounted /ext4.
[  OK  ] Unmounted /xfs.
[  OK  ] Stopped File System Check on /dev/disk/by-label/XFS.
[  OK  ] Stopped File System Check on /dev/disk/by-label/EXT4.
[  OK  ] Removed slice system-systemd\x2dfsck.slice.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Hypothesis,

Test (1) - The error is NEVER propagated to upper layers:

# xfs and ext4 mounted automatically

inaddy@iscsihang:~$ mount | grep _netde
/dev/sda1 on /ext4 type ext4 (rw,relatime,stripe=32,data=ordered,_netdev)
/dev/sdb1 on /xfs type xfs (rw,relatime,attr2,inode64,noquota,_netdev)

# no error propagation

inaddy@iscsihang:~$ sudo iscsiadm -m node -o show | grep timeo.replace
node.session.timeo.replacement_timeout = -1
node.session.timeo.replacement_timeout = -1

# target server can't give any more packets to guest:

inaddy@machete:~$ sudo iptables -A INPUT -s 192.168.49.8 -p tcp
--destination-port 3260 -j DROP

# reboot can't succeed

inaddy@iscsihang:~$ sudo reboot

[   27.596135]  connection2:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294896692, last ping 4294897944, now 4294899196
[   27.628109]  connection1:0: ping timeout of 5 secs expired, recv timeout 5, 
last rx 4294896700, last ping 4294897952, now 4294899204

Systemd hangs forever:

[  OK  ] Stopped target Remote File Systems.
 Unmounting /ext4...
 Unmounting /xfs...

OBS: There is a tight relationship in between connection disappearing
before the umount service runs and the capability of systemd to shutdown
the machine entirely. I would say that, in case of no error propagation,
is even worse since kernel would be locked up forever:

[  240.132208] INFO: task systemd:1094 blocked for more than 120 seconds.
[  240.133499]   Not tainted 4.4.0-93-generic #116-Ubuntu
[  240.134544] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
message.
[  240.136092] INFO: task umount:1199 blocked for more than 120 seconds.
[  240.137262]   Not tainted 4.4.0-93-generic #116-Ubuntu
[  240.138302] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
message.
[  240.139742] INFO: task umount:1201 blocked for more than 120 seconds.
[  240.140898]   Not tainted 4.4.0-93-generic #116-Ubuntu
[  240.141953] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
message.

Systemd is still trying...

[  OK  ] Unmounted /ext4.
[  OK  ] Unmounted /xfs.
[  OK  ] Stopped File System Check on /dev/disk/by-label/XFS.
[  OK  ] Stopped File System Check on /dev/disk/by-label/EXT4.
[  OK  ] Removed slice system-systemd\x2dfsck.slice.
[  OK  ] Stopped target Remote File Systems (Pre).
 Stopping Login to default iSCSI targets...

[  360.140109] INFO: task systemd:1094 blocked for more than 120 seconds.
[  360.141219]   Not tainted 4.4.0-93-generic #116-Ubuntu
[  360.142100] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
message.
[  360.143377] INFO: task umount:1199 blocked for more than 120 seconds.
[  360.144451]   Not tainted 4.4.0-93-generic #116-Ubuntu
[  360.145333] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
message.
[  360.146576] INFO: task umount:1201 blocked for more than 120 seconds.
[  360.147586]   Not tainted 4.4.0-93-generic #116-Ubuntu
[  360.148472] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
message.

This will happen forever. I still have to find a way of causing systemd
to shutdown network and cause this hang because error, likely, is
propagated after the umount service gives up its logic (or something
like it) <-- theory.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Matt Schulte]

Please let me explain who I am and what we do to possibly shed some
light on this.

I am an Interoperability engineer at NetApp.  We test our storage
products with many popular operating systems, adapters, protocols and
switches, in varying combinations.

We test software initiator iscsi continuously (on other distros) with exactly 
the same configuration as I have described above and have never seen this 
behavior.  Indeed whatever the problem is seems fixed in 
Ubuntu upstream as I indicated above.  This is also new behavior introduced 
between 14.04 and 16.04 because it did not occur in older versions.

The purpose of me asking to get this issue resolved is not to fix MY
system, it is to fix it for my customers' systems.  We will not post
support for Ubuntu 16.04 (iscsi) without resolving this issue so that we
don't put our customers in the same position.

With all that said, I do not currently have a configuration available or
setup to reproduce this problem.  If it is required, I can add it to the
queue.  When it is up and running, it reproduces every single time.

Other folks on the thread may actually have it up and running right now
and be able to test the immediate fail before I get to it.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Hello Matt,

Good to read your feedback. When you have multipath AND iscsi, the error
propagation timeout can be given by one or both, that is why I explained
both. This issue is not an iscsi or multipath problem, it is a systemd
problem related to the fact that the network is removed before the disks
are unmounted. Good to know you don't have containers, that is something
that could hold the disk superblock reference as well (even not having
containers, filesystems mounted by _netdev can hold reference by
different mount namespaces started by systemd itself, not the case
here).

Could you do me a favor. Are you reproducing this issue in a constant
basis ? If you are, could you please reduce your replacement_timeout to
0 and see if you can reproduce it again ? Please be aware that, after
changing replacement_timeout = 0 in iscsid.conf, you will have to change
the parameter for the disks already discovered:

## example

Change iscsid.conf AND:

$ sudo iscsiadm -m node
192.168.48.1:3260,1 iqn.2017.tgtd:tid2.lun
192.168.48.1:3260,1 iqn.2017.tgtd:tid1.lun

$ sudo iscsiadm -m node -o show | grep node.session.timeo.replacement_timeout
node.session.timeo.replacement_timeout = 20
node.session.timeo.replacement_timeout = 20

$ sudo iscsiadm -m node -o update -n node.session.timeo.replacement_timeout -v 0
$ sudo iscsiadm -m node -o show | grep node.session.timeo.replacement_timeout
node.session.timeo.replacement_timeout = 0
node.session.timeo.replacement_timeout = 0

This will cause the I/O errors to be propagated right away - when
systemd disconnects the interface - and cause the later umount - done by
other systemd service - to succeed without waiting for pagecache timeout
+ iscsi timeout (20 for you). Hopefully the "0" will be fast enough to
make systemd to continue all the times.

Could you please let me know if it mitigates the issue ? If it does,
I'll work in the actual fix. This will just confirm hypothesis and help
in diagnose (and serve as a workaround if anyone cares).

Thank you
-Rafael

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Matt Schulte]

I have node.session.timeo.replacement_timeout = 20, which seems
completely reasonable in my book.

We have already established that this occurs without multipath even
enabled, thus it is not a multipath settings problem.

Also I have no containers.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

** Changed in: systemd (Ubuntu Xenial)
 Assignee: (unassigned) => Rafael David Tinoco (inaddy)

** Changed in: systemd (Ubuntu Xenial)
   Status: Confirmed => In Progress

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Dimitri John Ledkov]

** Changed in: systemd (Ubuntu Xenial)
 Assignee: Dimitri John Ledkov (xnox) => (unassigned)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Another thing. Please check for containers referencing the filesystems
in question. When you have _netdev filesystems, it is possible that the
"mount --shared" characteristics of containers (different mount
namescapes) are holding references to the filesystem in question.

A good test is to test the filesystem being mounted by /etc/fstab AND
manually. If mounted with _netdev, by /etc/fstab, then the containers
will have reference to the same filesystems as a shared mount. This will
cause a shutdown problem if you don't remove the container reference as
well.

Please do test this as well.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Anyone facing this..

Could you please make sure to propagate I/O errors properly to
filesystem in order for it to call its automatically shutdown procedure.
After that, the mount won't reference the super block in kernel and that
will allow the device mapper to be destroyed (this can be seen by
checking existence of device-mapper node in
/sys/fs/{xfs,ext3,ext4}/node, saying that the super block is still being
referenced).

I had a recent case where not propagating the error would cause the
filesystem to hang forever, causing lockups and not allowing the device-
mapper tables to be flushed by multipath (or even the OS to be
operational). In this case, if the network is turned off (or
interrupted) before the ISCSI is logged out, then the multipath (dm) and
iscsi (transport) layers HAVE TO propagate the I/O error to the
filesystem for it to shutdown BEFORE the device mapper flush logic is
attempted (by issuing ioctls to /dev/dm-X by multipath or lvm, example).

This is can be done by:

node.session.timeo.replacement_timeout = 0 (iscsid.conf)
(This parameter might be persistent on already discovered LUNs)

USAGE:
iscsiadm -m node -T $target_name -p $target_ip:$port -o update -n \
node.session.timeo.replacement_timeout -v $timeout_value 
on already logged/discovered LUNs. 

 AND 

dev_loss_tmo 10 (multipath.conf)

The fact that the network was removed BEFORE the ISCSI logout happened
is NOT an issue if ROOT device is NOT on ISCSI: You just have to allow
the block device <-> device mapper <-> transport layer SHUTDOWN logic to
WORK before flushing the device-mapper. (Hopefully the umount on _netdev
devices is done, causes an I/O error, filesystem shuts down, multipath
service closes multipath device-mapper devices).

Pay attention because if filesystem is umounted and ISCSI is gone, the
timeout will have to be waited before the error is PROPAGATED to other
layer. So if filesystem is umounted, ONE last superblock inode UPDATE
will be attempted (if no cache in pagecache). This update is to sync
superblock state and update metadata for the filesystem itself. This
update attempt will create an I/O error for the superblock inode (after
time timeout you configured). This will make the filesystem to be
shutdown (a half unmounted state). After shutdown, the logic trying to
umount (synchronously) the filesystem will continue (possibly in
systemd).

If the problem is because of you tried to flush the device mapper and
the filesystem was still referencing the device mapper super block, this
will fade away if the error is propagated and the filesystem is
shutdown, you can attempt doing the flush of device-mapper right after
filesystem shutdown happened.

Could you please try setting the timeouts properly for lower timings and
let me know if that mitigated the issue ?

Thank you

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Rafael David Tinoco]

Anyone facing this..

Could you please make sure to propagate I/O errors properly to
filesystem in order for it to call its automatically shutdown procedure.
After that, the mount won't reference the super block in kernel and that
will allow the device mapper to be closed/destroyed (this can be seen by
checking existence of device-mapper node in
/sys/fs/{xfs,ext3,ext4}/node, saying that the super block is still being
referenced).

I had a recent case where not propagating the error would cause the
filesystem to hang forever, causing lockups and not allowing the device-
mapper tables to be flushed by multipath. In this case, if the network
was turned off before the iscsi was logged, the multipath (dm) and iscsi
(transport) layers HAVE TO propagate the I/O error to the filesystem for
it to shutdown.

This is can be done by:

node.session.timeo.replacement_timeout = 0 (iscsid.conf)

USE: iscsiadm -m node -T $target_name -p $target_ip:$port  -o update -n \
node.session.timeo.replacement_timeout -v $timeout_value on already 
logged/discovered LUNs. This parameter might be persistent on already 
discovered LUNs.

AND:

dev_loss_tmo 10 (multipath.conf)


OBS: node.session.timeo.replacement_timeout has to be changed per 

The fact that the network was removed BEFORE the iscsi logout could
happen is NOT an issue if ROOT device is not ISCSI, you just have to
allow the block device <-> device mapper <-> transport layer logic
shutdown to work.

Pay attention because if filesystem is umounted and iscsi is gone, the
timeout will have to be waited before the error is propagated to other
layer. So if filesystem is umounted a last superblock inode update will
be attempted (if no cache in pagecache). This will make the I/O error
for the superblock inode to fail (after time timeout you configured).
This will make the filesystem to be shutdown (a half unmounted state).
After shutdown, the logic trying to umount (synchronously) the
filesystem will continue (possibly in systemd).

If the problem is because of you tried to flush the device mapper and
the filesystem was still referencing the device mapper super block, this
will fade away if the error is propagated and the filesystem is
shutdown, you can attempt doing the flush of device-mapper right after
filesystem shutdown happened.

Could you please try setting the timeouts properly for lower timings and
let me know if that mitigated the issue ?

Thank you

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Thiago Martins]

Is it possible to reproduce this problem with upstart or sysvinit-core?

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Matt Schulte]

I can confirm that the daily 17.10 build does not exhibit the problem.

Interestingly, the first time we tried we forgot to stop running IO to
our iSCSI volumes and it _looked_ like we reproduced the problem, then
the IOs timed out and our application stopped and we were able to
complete the reboot.

The next time we retried and correctly stopped our application before
shutting down and had no issue.  Whatever it is, is fixed in 17.10.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Launchpad Bug Tracker]

Status changed to 'Confirmed' because the bug affects multiple users.

** Changed in: systemd (Ubuntu Xenial)
   Status: New => Confirmed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Kai Holthaus]

@nacc: Yes.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Nish Aravamudan]

@kai-holthaus: after editing the service file, did you issue a `sudo
systemctl daemon-reload` ?

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Kai Holthaus]

@nacc: The system was originally installed with Ubuntu Server 14.04. It
has been upgraded to 16.04.

apt policy open-iscsi:
open-iscsi:
  Installed: 2.0.873+git0.3b4b4500-14ubuntu3.3
  Candidate: 2.0.873+git0.3b4b4500-14ubuntu3.3
  Version table:
 *** 2.0.873+git0.3b4b4500-14ubuntu3.3 500
500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 
Packages
100 /var/lib/dpkg/status
 2.0.873+git0.3b4b4500-14ubuntu3 500
500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
 2.0.873-3ubuntu13 500
500 http://us.archive.ubuntu.com/ubuntu wily/main amd64 Packages

Modifying those two lines in /lib/systemd/system/open-iscsi.service does not 
have any effect - I'm still running into the timeout.
I have attached a new journal.txt (per instructions in #8). This time, the 
error message is
logout-all.sh[2206]: iscsiadm: No matching sessions found

** Attachment added: "journal.txt"
   
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+attachment/4894804/+files/journal.txt

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Nish Aravamudan]

@Kai: Hrm, that is odd -- was your 16.04 system the fresh install or an
upgraded one?

The lines I pasted in #49 are supposed to be present in 16.04 (afaict).

If they don't, can you provide `apt policy open-iscsi` in a pastebin?
And can you test if changing those lines to match fixes the issue there?

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Kai Holthaus]

@nacc: No, my lines do not like that. On my xenial system, I have:

Wants=network-online.target remote-fs-pre.target
After=network-online.target

Which is probably why on my system the iscsid daemon shuts down before
the logout.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Kai Holthaus]

@nacc: Add-on: On my Debian systems (Debian jessie and stretch) that
also experience this issue, however, the lines do look like what you
described - and still the system hangs at reboot / shutdown.

In a previous message it sounded like you had identified an issue and a
fix - did I misinterpret your response? If so, was it a fix I can apply
to my xenial system and test?

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Nish Aravamudan]

@Kai: In xenial, open-iscsi.service should have the following lines
alreadY

Wants=network-online.target remote-fs-pre.target iscsid.service
After=network-online.target iscsid.service

Can you confirm that is the case on your system?

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Kai Holthaus]

I've installed a system with the daily artful server image, and the
error is not present for me. However, I'm not sure how useful that test
is, given that I didn't experience the error during a fresh install of
xenial.

I've attached the journal from the artful install, and it looks to me
like the steps are now in the right order. Was adding "iscsid.service"
to the "after" and "wants" lines in /lib/systemd/system/open-
iscsi.service the only change that was required? If so, I'd be happy to
test that on my server that's experiencing the issue...

** Attachment added: "journal.txt"
   
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+attachment/4892287/+files/journal.txt

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Nish Aravamudan]

@kai-holthause or @gimpeystrada: would either of you be able to test
with the daily artful (17.10) ISO?

@kai-holthause: I think you did find a real ordering issue, nicely done
:)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Dimitri John Ledkov]

** Also affects: systemd (Ubuntu Xenial)
   Importance: Undecided
   Status: New

** Changed in: systemd (Ubuntu Xenial)
 Assignee: (unassigned) => Dimitri John Ledkov (xnox)

** Changed in: systemd (Ubuntu Xenial)
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Kai Holthaus]

Thanks, @nacc.
So, here's what I've done in the meantime:
- I've built another server with 14.04. Clean install, only OpenSSH server. 
Connected to an iscsi target. Reboot works.
- Upgraded the server to 16.04.02. Reboot still works. Guess my hunch was 
wrong. :(
- I've attached the journal as requested. You'll see that the iscsid daemon 
shuts down before the "logout-all.sh" script runs. Since iscsid has shut down, 
that script does not run successfully.

My machines that exhibit the problem use an iSCSI drive for:
- mythbackend (storage for recorded videos - that's the Ubuntu journal attached)
- postgres and mysql storage (Debian server)
- nginx storage (also on Debian)

Thanks for your help.

** Attachment added: "Requested journal file"
   
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+attachment/4891508/+files/journal.txt

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Nish Aravamudan]

@kai-holthaus: Thank you for the excellent data point, again, and for
your willingness to provide the data!

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Nish Aravamudan]

@kai-holthaus: also, afaict, that might mean you are hitting a
*different* bug (yay for so many bugs) in open-iscsi relative to
@gimpeystrada, as @gimpeystrada and others in this channel have said
they experience this issue with a fresh install of 16.04 (16.10 and
17.04 as well). That's ok, we can look at the journal and figure it out
first, but the end result might be opening another bug.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Kai Holthaus]

@nacc: I can certainly do that - but it won't be quite as fast as my response 
yesterday, as the machine that exhibits this error is in use during the day, 
and I need to schedule the downtime.
In the meantime, I have installed a test machine from scratch, using the 
standard Ubuntu LTS server ISO. All I did on that machine was to set a static 
IP address, upgraded to the latest packages, and configured an open-iscsi mount.

Lo and behold, the test machine does not exhibit the error. My machine
that does exhibit the error was upgraded using 'do-release-upgrade' from
the previous LTS version (14.04). My Debian machines similarly were
upgraded from wheezy to jessie (which introduced the issue with the
switch to systemd).

So, for now, I would say there's something different between upgrade and
fresh install that causes this. I will find some time today for the
downtime of my other machine and attach the persistent journal info as
requested.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Nish Aravamudan]

@kai-holthaus: thank you for responding so quickly. Can you set up a
persistent journal as described in c#8 and provide the journal as
mentioned in that comment for the failed shutdown?

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Kai Holthaus]

I checked my system (16.04.02) which is also exhibiting this issue (my Debian 
systems do as well). On both OS, the open-iscsi service is in "active (exited)" 
state after booting (and not in failed state):
● open-iscsi.service - Login to default iSCSI targets
   Loaded: loaded (/lib/systemd/system/open-iscsi.service; enabled; vendor 
preset: enabled)
   Active: active (exited) since Sat 2017-06-03 06:00:19 PDT; 3 days ago
 Docs: man:iscsiadm(8)
   man:iscsid(8)
  Process: 978 ExecStart=/lib/open-iscsi/activate-storage.sh (code=exited, 
status=0/SUCCESS)
  Process: 918 ExecStart=/sbin/iscsiadm -m node --loginall=automatic 
(code=exited, status=0/SUCCESS)
  Process: 905 ExecStartPre=/bin/systemctl --quiet is-active iscsid.service 
(code=exited, status=0/SUCCESS)
 Main PID: 978 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/open-iscsi.service

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Matt Schulte]

My test configuration has long since been re-tasked.  I will eventually
be able to come back and test as you ask, but it may take a few weeks.
If anyone else on the bug has their systems up and available, please
test and reply.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Nish Aravamudan]

@gimpeystrada: Thank you for providing your journal logs (in the future,
providing the raw journal file is better, as then we can run
`journalctl` on it locally). From the logs:

Apr 26 08:52:13 ICTM1612S02H1 iscsiadm[3234]: iscsiadm: initiator reported 
error (8 - connection timed out)
Apr 26 08:52:13 ICTM1612S02H1 iscsiadm[3234]: iscsiadm: Could not log into all 
portals
Apr 26 08:52:13 ICTM1612S02H1 iscsiadm[3234]: :0113,3260] successful.
Apr 26 08:52:13 ICTM1612S02H1 systemd[1]: open-iscsi.service: Child 3234 
belongs to open-iscsi.service
Apr 26 08:52:13 ICTM1612S02H1 systemd[1]: open-iscsi.service: Main process 
exited, code=exited, status=8/n/a
Apr 26 08:52:13 ICTM1612S02H1 systemd[1]: open-iscsi.service: Changed start -> 
failed
Apr 26 08:52:13 ICTM1612S02H1 systemd[1]: Sent message type=signal sender=n/a 
destination=n/a object=/org/freedesktop/systemd1 
interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=936 
reply_cookie=0 error=n/a
Apr 26 08:52:13 ICTM1612S02H1 systemd-logind[2801]: Got message type=signal 
sender=:1.2 destination=n/a object=/org/freedesktop/systemd1 
interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=936 
reply_cookie=0 error=n/a
Apr 26 08:52:13 ICTM1612S02H1 systemd[1]: open-iscsi.service: Job 
open-iscsi.service/start finished, result=failed
Apr 26 08:52:13 ICTM1612S02H1 systemd[1]: Failed to start Login to default 
iSCSI targets.
Apr 26 08:52:13 ICTM1612S02H1 systemd[1]: Sent message type=signal sender=n/a 
destination=n/a object=/org/freedesktop/systemd1 
interface=org.freedesktop.systemd1.Manager member=JobRemoved cookie=937 
reply_cookie=0 error=n/a
Apr 26 08:52:13 ICTM1612S02H1 systemd[1]: open-iscsi.service: Unit entered 
failed state.
Apr 26 08:52:13 ICTM1612S02H1 systemd[1]: open-iscsi.service: Failed with 
result 'exit-code'.
Apr 26 08:52:13 ICTM1612S02H1 systemd[1]: open-iscsi.service: cgroup is empty

I *think* the issue in this case is that open-iscsi.service is actually
in a failed state before shutdown, which I think means it does not run
the ExecStop commands? Can you verify that is the case (boot your
system, `systemctl is-system-running` should indicate "degraded",
`systemctl status open-iscsi.service` should indicate Failed). The
problem is that the open-iscsi.service has:

ExecStartPre=/bin/systemctl --quiet is-active iscsid.service
ExecStart=/sbin/iscsiadm -m node --loginall=automatic
ExecStart=/lib/open-iscsi/activate-storage.sh
ExecStop=/lib/open-iscsi/umountiscsi.sh
ExecStop=/bin/sync
ExecStop=/lib/open-iscsi/logout-all.sh

This indicates it, on 'start' of the service, it will try to login to
and active all configured iSCSI targets (the two ExecStart lines).
However, if either of those fail (as they did in the journal in this
case), the ExecStop lines *do not* run. From `man systemd.service`:

   Note that if any of the commands specified in ExecStartPre=,
   ExecStart=, or ExecStartPost= fail (and are not prefixed with "-",
   see above) or time out before the service is fully up, execution
   continues with commands specified in ExecStopPost=, the commands in
   ExecStop= are skipped.

So, I think that we should be using ExecStopPost instead.

@gimpeystrada, can you test locally if your system works correctly by
editing /lib/systemd/system/open-iscsi.service to use ExecStopPost
rather than ExecStop? You will need to run `systemctl daemon-reload`
after editing the service file, I believe.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Re: [Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Nish Aravamudan]

On Wed, May 24, 2017 at 1:45 PM, Scott <1569...@bugs.launchpad.net> wrote:
> Fresh install 16.04 download ISO from ubuntu
> Can confirm this issue as well.
> If I manually log out of the iscsi session reboot no issue

Interesting data point.

> If I am logged into iscsi session reboot hangs. Need to physically
power cycle server

I think the open-iscsi or iscsid services have a killall sessions
portion when the service shuts down. I wonder if that's either a) not
running or b) somehow incorrect. I could imagine for iSCSI root it'd
be an issue, but for non-root, I'm not sure why that's not working.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Scott]

Fresh install 16.04 download ISO from ubuntu 
Can confirm this issue as well.
If I manually log out of the iscsi session reboot no issue
If I am logged into iscsi session reboot hangs. Need to physically power cycle 
server

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Matt Schulte]

Didn't take as long as I thought.  Installed the most recent build of
Zesty (zesty-server-amd64.iso 2017-02-22 06:54676M).

I was able to:

1. Confirm the issue still exists.
2. Confirm that the issue is not caused by multipath (i.e. it still occurs 
during reboot when multipathing is disabled).

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Nish Aravamudan]

No problem -- also, you can just test in 16.10 without multipath. I can
spend time reproducing it, etc. later to verify whatever fix we come up
with is correct.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Matt Schulte]

It is possible to test without multipath, this will take some time as I
no longer have anything with 16.04 installed.  For fun I may go ahead
and try with 17.04 so we can figure out if it still exists then we'll
kind of kill two birds with one stone.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Nish Aravamudan]

@gimpeystrada: thank you for the prompt response!

Err, right on multipath. Is it at all possible to test without multipath
in use? To help narrow down if it's really in iSCSI or if it's in the
multipath layer?

Note additional bugs aren't required (I understand at some point this
one may have expired, but we'll end up creating tasks here for all
affected releases).

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Matt Schulte]

iSCSI disks are not the root/boot disks.

Multipath IS in use.  Hence the /dev/mapper at the beginning of the disk
names.

Yes I have tried 16.10, it is still present and I have another bug for
that release.  https://bugs.launchpad.net/bugs/1636862

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Nish Aravamudan]

So let's separate out two issues in this bug.

@gimpeystrada, I believe your issue was always with iSCSI disks that
were not in-use as the root/boot disk, correct? Looking at your fstab, I
think also you don't have multipath over iSCSI, correct?

For anyone else indicating this bug is also affecting them, if
@gimpeystrada replies in the affirmative to the above (Not using iSCSI
for root disk, not using multipath), then I would like to use separate
bugs for those issues. They may all have the same root-cause, but it
becomes rather noisy, otherwise.

@gimpestrada, have you tried 16.10 or 17.04 to see if it has been fixed
already? Just curious.

** Changed in: systemd (Ubuntu)
 Assignee: (unassigned) => Nish Aravamudan (nacc)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Matt Schulte]

Any idea on this one folks?

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Chris Osgood]

Seeing a similar issue here with so called "diskless" clients. That is,
all of their filesystems are network mounts including the root
filesystem. These systems use iscsi targets for the root filesystem and
during shutdown the network is turned off before the file I/O operations
are completed which causes the system to hang with filesystem IO errors.

I'm currently looking to see if there is some way to make systemd keep
the network up no matter what, all the way until the machine physically
reboots or shuts down (ie. disable the network shutdown completely).

This worked in 14.04 and got broken in 16.04 apparently due to the
switch to systemd as far as I can tell (great tool, isn't it?).

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Matt Schulte]

New bug for 16.10:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1636862

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Alberto Salvia Novella]

** Changed in: systemd (Ubuntu)
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Alberto Salvia Novella]

** Attachment added: "Screenshot.jpg"
   
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+attachment/4769386/+files/Screenshot.jpg

** Description changed:

  I have 4 servers running the latest 16.04 updates from the development
  branch (as of right now).
  
  Each server is connected to NetApp storage using iscsi software
  initiator.  There are a total of 56 volumes spread across two NetApp
  arrays.  Each volume has 4 paths available to it which are being managed
  by device mapper.
  
  While logged into the iscsi sessions all I have to do is reboot the
  server and I get a hang.
  
  I see a message that says:
  
-   "Reached target Shutdown"
+   "Reached target Shutdown"
  
  followed by
  
-   "systemd-shutdown[1]: Failed to finalize DM devices, ignoring"
+   "systemd-shutdown[1]: Failed to finalize DM devices, ignoring"
  
  and then I see 8 lines that say:
  
-   "connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
-   "connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
-   "connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
-   "connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
-   "connection5:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
-   "connection6:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
-   "connection7:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
-   "connection8:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
-   NOTE: the actual values of the *'s differ for each line above.
+   "connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
+   "connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
+   "connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
+   "connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
+   "connection5:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
+   "connection6:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
+   "connection7:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
+   "connection8:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
+   NOTE: the actual values of the *'s differ for each line above.
  
  This seems like a bug somewhere but I am unaware of any additional
  logging that I could turn on to pinpoint the problem.
  
  Note I also have similar setups that are not doing iscsi and they don't
  have this problem.
  
  Here is a screenshot of what I see on the shell when I try to reboot:
  
- https://www.dropbox.com/s/93mv7cj8asspcpa/ShutdownCapture.JPG?dl=0
+ (https://launchpadlibrarian.net/291303059/Screenshot.jpg)
  
  This is being tracked in NetApp bug tracker CQ number 860251.
  
  If I log out of all iscsi sessions before rebooting then I do not
  experience the hang:
  
  iscsiadm -m node -U all
  
  We are wondering if this could be some kind of shutdown ordering
  problem.  Like the network devices have already disappeared and then
  iscsi tries to perform some operation (hence the ping timeouts).

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

[Matt Schulte]

** Changed in: systemd (Ubuntu)
   Status: Expired => Confirmed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs