[Touch-packages] [Bug 1925211] Comment bridged from LTC Bugzilla
--- Comment From heinz-werner_se...@de.ibm.com 2021-06-14 08:34 EDT--- IBM bugzilla status->closed, Fix Released by all requested distros. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1925211 Title: Hot-unplug of disks leaves broken block devices around in Hirsute on s390x Status in Ubuntu on IBM z Systems: Fix Released Status in linux package in Ubuntu: Fix Released Status in systemd package in Ubuntu: Invalid Status in udev package in Ubuntu: Invalid Status in linux source package in Hirsute: Fix Released Status in systemd source package in Hirsute: Invalid Status in udev source package in Hirsute: Invalid Bug description: SRU Justification [Impact] Hot removal of disks under kvm on s390 does not result in the kernel removing the block device, which can lead to hung tasks and other issues. [Test Plan] See steps to reproduce the bug in the original description below. To test, execute these steps and confirm that the block device gets removed as expected. [Where problems could occur] The fix is a revert of the changes which introduced this regression. The original commit was a removal of supposedly unused code, but it seems a mistake was made in the logic around unregistering of disks. Reverting the changes could have potential to introduce bugs related to other virt devices, especially if it interacts badly with subsequent driver changes. However, the patch reverted cleanly, and reverting restores the code to the state which has been working well in previous kernels and seems like the lowest risk option until a proper fix is available upstream. --- Repro: #1 Get a guest $ uvt-kvm create --disk 5 --password=ubuntu h release=hirsute arch=s390x label=daily $ uvt-kvm wait h release=hirsute arch=s390x label=daily #2 Attach and Detach disk $ sudo qemu-img create -f qcow2 /var/lib/libvirt/images/test.qcow2 10M $ virsh attach-disk h /var/lib/libvirt/images/test.qcow2 vdc $ virsh detach-disk h vdc From libvirts POV it is gone at this point $ virsh domblklist h Target Source -- vda /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs.qcow vdb /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs-ds.qcow But the guest thinks still it is present $ uvt-kvm ssh --insecure hirsute-2nd-zfs lsblk ... vdc252:32 0 20M 0 disk This even remains a while after (not a race). Any access to it in the guest will hang (as you'd expect of a non-existing blockdev) 4 017581739 20 0 12140 4800 - S+ pts/0 0:00 | \_ sudo mkfs.ext4 /dev/vdc 4 017591758 20 0 6924 1044 - D+ pts/0 0:00 | \_ mkfs.ext4 /dev/vdc The result above was originally found with hirsute-guest@hirsute-host on s390x I do NOT see the same with groovy-guest@hirsute-host on s390x I DO see the same with hirsute-guest@groovy-host on s390x => Guest version dependent not Host/Hipervisor dependent I DO see the same with ZFS disks AND LVM disks being added => not type dependent I do NOT see the same on x86. => Arch dependent ?? ... the evidence slowly points towards an issue in the guest, damn we are so close to release - but non-fully detaching disks are critical in my POV :-/ Filing this as-is for awareness, but certainly this will need more debugging. Unsure where this is going to eventually I'll now file it for kernel/udev/systemd. If there are any known issues/components that are related let me know please! --- ProblemType: Bug AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.11-0ubuntu65 Architecture: s390x ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' CRDA: N/A CasperMD5CheckResult: unknown DistroRelease: Ubuntu 21.04 IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lspci-vt: -[:00]- Lsusb: Error: command ['lsusb'] failed with exit code 1: Lsusb-t: Error: command ['lsusb', '-t'] failed with exit code 1: /sys/bus/usb/devices: No such file or directory Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1: Package: udev PackageArchitecture: s390x PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ProcVersionSignature: User Name 5.11.0-14.15-generic 5.11.12 RelatedPackageVersions: linux-restricted-modules-5.11.0-14-generic N/A linux-backports-modules-5.11.0-14-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' Tags: hirsute uec-images Uname:
[Touch-packages] [Bug 1925211] Comment bridged from LTC Bugzilla
--- Comment From heinz-werner_se...@de.ibm.com 2021-05-19 07:18 EDT--- IBM Response: Yes. this makes sense. Previously Canonical reverted the "s390/cio: remove pm support from ccw bus driver" patch because of the issue. Right solution is to unrevert the revert-patch and add "s390/cio: remove invalid condition on IO_SCH_UNREG" patch. Which makes both available in the kernel. So, we agree on this. In stable 5.13-rc1, we have both the patches. Which is same as 5.11.20 for hirsute. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1925211 Title: Hot-unplug of disks leaves broken block devices around in Hirsute on s390x Status in Ubuntu on IBM z Systems: Fix Committed Status in linux package in Ubuntu: Fix Committed Status in systemd package in Ubuntu: Invalid Status in udev package in Ubuntu: Invalid Status in linux source package in Hirsute: Fix Committed Status in systemd source package in Hirsute: Invalid Status in udev source package in Hirsute: Invalid Bug description: SRU Justification [Impact] Hot removal of disks under kvm on s390 does not result in the kernel removing the block device, which can lead to hung tasks and other issues. [Test Plan] See steps to reproduce the bug in the original description below. To test, execute these steps and confirm that the block device gets removed as expected. [Where problems could occur] The fix is a revert of the changes which introduced this regression. The original commit was a removal of supposedly unused code, but it seems a mistake was made in the logic around unregistering of disks. Reverting the changes could have potential to introduce bugs related to other virt devices, especially if it interacts badly with subsequent driver changes. However, the patch reverted cleanly, and reverting restores the code to the state which has been working well in previous kernels and seems like the lowest risk option until a proper fix is available upstream. --- Repro: #1 Get a guest $ uvt-kvm create --disk 5 --password=ubuntu h release=hirsute arch=s390x label=daily $ uvt-kvm wait h release=hirsute arch=s390x label=daily #2 Attach and Detach disk $ sudo qemu-img create -f qcow2 /var/lib/libvirt/images/test.qcow2 10M $ virsh attach-disk h /var/lib/libvirt/images/test.qcow2 vdc $ virsh detach-disk h vdc From libvirts POV it is gone at this point $ virsh domblklist h Target Source -- vda /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs.qcow vdb /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs-ds.qcow But the guest thinks still it is present $ uvt-kvm ssh --insecure hirsute-2nd-zfs lsblk ... vdc252:32 0 20M 0 disk This even remains a while after (not a race). Any access to it in the guest will hang (as you'd expect of a non-existing blockdev) 4 017581739 20 0 12140 4800 - S+ pts/0 0:00 | \_ sudo mkfs.ext4 /dev/vdc 4 017591758 20 0 6924 1044 - D+ pts/0 0:00 | \_ mkfs.ext4 /dev/vdc The result above was originally found with hirsute-guest@hirsute-host on s390x I do NOT see the same with groovy-guest@hirsute-host on s390x I DO see the same with hirsute-guest@groovy-host on s390x => Guest version dependent not Host/Hipervisor dependent I DO see the same with ZFS disks AND LVM disks being added => not type dependent I do NOT see the same on x86. => Arch dependent ?? ... the evidence slowly points towards an issue in the guest, damn we are so close to release - but non-fully detaching disks are critical in my POV :-/ Filing this as-is for awareness, but certainly this will need more debugging. Unsure where this is going to eventually I'll now file it for kernel/udev/systemd. If there are any known issues/components that are related let me know please! --- ProblemType: Bug AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.11-0ubuntu65 Architecture: s390x ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' CRDA: N/A CasperMD5CheckResult: unknown DistroRelease: Ubuntu 21.04 IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lspci-vt: -[:00]- Lsusb: Error: command ['lsusb'] failed with exit code 1: Lsusb-t: Error: command ['lsusb', '-t'] failed with exit code 1: /sys/bus/usb/devices: No such file or directory Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1: Package: udev PackageArchitecture: s390x PciMultimedia: ProcFB: ProcKernelCmdLine:
[Touch-packages] [Bug 1925211] Comment bridged from LTC Bugzilla
--- Comment From vineeth.vija...@ibm.com 2021-04-23 01:50 EDT---
---snip---
--- a/drivers/s390/cio/device.c
+++ b/drivers/s390/cio/device.c
@@ -1525,8 +1525,7 @@ static int io_subchannel_sch_event(struct subchannel
*sch, int process)
switch (action) {
case IO_SCH_ORPH_UNREG:
case IO_SCH_UNREG:
- if (!cdev)
- css_sch_device_unregister(sch);
+ css_sch_device_unregister(sch);
break;
case IO_SCH_ORPH_ATTACH:
case IO_SCH_UNREG_ATTACH:
I think we'd want an answer from the IBM devs which solution (full revert,
kaihenfeng patch, cpaelzer patch, another approach) they would prefer - then we
can submit it upstream for them to include officially and we can carry it as
delta until we rebase onto a version that has it applied anyway.
---snip---
Thank you very much for reporting this. Yes. This is a leftover from the
pm-remove patch and the right solution is as mentioned above here. We
shall prepare the patch and share it to the external mailing-list.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1925211
Title:
Hot-unplug of disks leaves broken block devices around in Hirsute on
s390x
Status in Ubuntu on IBM z Systems:
Triaged
Status in linux package in Ubuntu:
Triaged
Status in systemd package in Ubuntu:
Invalid
Status in udev package in Ubuntu:
Invalid
Status in linux source package in Hirsute:
Triaged
Status in systemd source package in Hirsute:
Invalid
Status in udev source package in Hirsute:
Invalid
Bug description:
SRU Justification
[Impact]
Hot removal of disks under kvm on s390 does not result in the kernel
removing the block device, which can lead to hung tasks and other
issues.
[Test Plan]
See steps to reproduce the bug in the original description below. To
test, execute these steps and confirm that the block device gets
removed as expected.
[Where problems could occur]
The fix is a revert of the changes which introduced this regression.
The original commit was a removal of supposedly unused code, but it
seems a mistake was made in the logic around unregistering of disks.
Reverting the changes could have potential to introduce bugs related
to other virt devices, especially if it interacts badly with
subsequent driver changes. However, the patch reverted cleanly, and
reverting restores the code to the state which has been working well
in previous kernels and seems like the lowest risk option until a
proper fix is available upstream.
---
Repro:
#1 Get a guest
$ uvt-kvm create --disk 5 --password=ubuntu h release=hirsute arch=s390x
label=daily
$ uvt-kvm wait h release=hirsute arch=s390x label=daily
#2 Attach and Detach disk
$ sudo qemu-img create -f qcow2 /var/lib/libvirt/images/test.qcow2 10M
$ virsh attach-disk h /var/lib/libvirt/images/test.qcow2 vdc
$ virsh detach-disk h vdc
From libvirts POV it is gone at this point
$ virsh domblklist h
Target Source
--
vda /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs.qcow
vdb /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs-ds.qcow
But the guest thinks still it is present
$ uvt-kvm ssh --insecure hirsute-2nd-zfs lsblk
...
vdc252:32 0 20M 0 disk
This even remains a while after (not a race).
Any access to it in the guest will hang (as you'd expect of a non-existing
blockdev)
4 017581739 20 0 12140 4800 - S+ pts/0 0:00 |
\_ sudo mkfs.ext4 /dev/vdc
4 017591758 20 0 6924 1044 - D+ pts/0 0:00 |
\_ mkfs.ext4 /dev/vdc
The result above was originally found with hirsute-guest@hirsute-host
on s390x
I do NOT see the same with groovy-guest@hirsute-host on s390x
I DO see the same with hirsute-guest@groovy-host on s390x
=> Guest version dependent not Host/Hipervisor dependent
I DO see the same with ZFS disks AND LVM disks being added
=> not type dependent
I do NOT see the same on x86.
=> Arch dependent ??
... the evidence slowly points towards an issue in the guest, damn we are so
close to release - but non-fully detaching disks are critical in my POV :-/
Filing this as-is for awareness, but certainly this will need more debugging.
Unsure where this is going to eventually I'll now file it for
kernel/udev/systemd.
If there are any known issues/components that are related let me know please!
---
ProblemType: Bug
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code
2: ls: cannot access '/dev/snd/': No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu65
Architecture: s390x
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
CRDA: N/A
