[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
** Tags added: cscc -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
Since it's also in disco master-next, I set the overall ticket to Fix Released. ** Changed in: linux (Ubuntu) Status: Fix Committed => Fix Released ** Changed in: ubuntu-z-systems Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
This bug was fixed in the package linux - 4.15.0-42.45 --- linux (4.15.0-42.45) bionic; urgency=medium * linux: 4.15.0-42.45 -proposed tracker (LP: #1803592) * [FEAT] Guest-dedicated Crypto Adapters (LP: #1787405) - KVM: s390: reset crypto attributes for all vcpus - KVM: s390: vsie: simulate VCPU SIE entry/exit - KVM: s390: introduce and use KVM_REQ_VSIE_RESTART - KVM: s390: refactor crypto initialization - s390: vfio-ap: base implementation of VFIO AP device driver - s390: vfio-ap: register matrix device with VFIO mdev framework - s390: vfio-ap: sysfs interfaces to configure adapters - s390: vfio-ap: sysfs interfaces to configure domains - s390: vfio-ap: sysfs interfaces to configure control domains - s390: vfio-ap: sysfs interface to view matrix mdev matrix - KVM: s390: interface to clear CRYCB masks - s390: vfio-ap: implement mediated device open callback - s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl - s390: vfio-ap: zeroize the AP queues - s390: vfio-ap: implement VFIO_DEVICE_RESET ioctl - KVM: s390: Clear Crypto Control Block when using vSIE - KVM: s390: vsie: Do the CRYCB validation first - KVM: s390: vsie: Make use of CRYCB FORMAT2 clear - KVM: s390: vsie: Allow CRYCB FORMAT-2 - KVM: s390: vsie: allow CRYCB FORMAT-1 - KVM: s390: vsie: allow CRYCB FORMAT-0 - KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-1 - KVM: s390: vsie: allow guest FORMAT-1 CRYCB on host FORMAT-2 - KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-2 - KVM: s390: device attrs to enable/disable AP interpretation - KVM: s390: CPU model support for AP virtualization - s390: doc: detailed specifications for AP virtualization - KVM: s390: fix locking for crypto setting error path - KVM: s390: Tracing APCB changes - s390: vfio-ap: setup APCB mask using KVM dedicated function - s390/zcrypt: Add ZAPQ inline function. - s390/zcrypt: Review inline assembler constraints. - s390/zcrypt: Integrate ap_asm.h into include/asm/ap.h. - s390/zcrypt: fix ap_instructions_available() returncodes - s390/zcrypt: remove VLA usage from the AP bus - s390/zcrypt: Remove deprecated ioctls. - s390/zcrypt: Remove deprecated zcrypt proc interface. - s390/zcrypt: Support up to 256 crypto adapters. - [Config:] Enable CONFIG_S390_AP_IOMMU and set CONFIG_VFIO_AP to module. * Bypass of mount visibility through userns + mount propagation (LP: #1789161) - mount: Retest MNT_LOCKED in do_umount - mount: Don't allow copying MNT_UNBINDABLE|MNT_LOCKED mounts * CVE-2018-18955: nested user namespaces with more than five extents incorrectly grant privileges over inode (LP: #1801924) // CVE-2018-18955 - userns: also map extents in the reverse map to kernel IDs * kdump fail due to an IRQ storm (LP: #1797990) - SAUCE: x86/PCI: Export find_cap() to be used in early PCI code - SAUCE: x86/quirks: Add parameter to clear MSIs early on boot - SAUCE: x86/quirks: Scan all busses for early PCI quirks -- Thadeu Lima de Souza Cascardo Thu, 15 Nov 2018 17:01:46 -0200 ** Changed in: linux (Ubuntu Bionic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
This bug was fixed in the package linux - 4.18.0-12.13 --- linux (4.18.0-12.13) cosmic; urgency=medium * linux: 4.18.0-12.13 -proposed tracker (LP: #1802743) * [FEAT] Guest-dedicated Crypto Adapters (LP: #1787405) - s390/zcrypt: Add ZAPQ inline function. - s390/zcrypt: Review inline assembler constraints. - s390/zcrypt: Integrate ap_asm.h into include/asm/ap.h. - s390/zcrypt: fix ap_instructions_available() returncodes - KVM: s390: vsie: simulate VCPU SIE entry/exit - KVM: s390: introduce and use KVM_REQ_VSIE_RESTART - KVM: s390: refactor crypto initialization - s390: vfio-ap: base implementation of VFIO AP device driver - s390: vfio-ap: register matrix device with VFIO mdev framework - s390: vfio-ap: sysfs interfaces to configure adapters - s390: vfio-ap: sysfs interfaces to configure domains - s390: vfio-ap: sysfs interfaces to configure control domains - s390: vfio-ap: sysfs interface to view matrix mdev matrix - KVM: s390: interface to clear CRYCB masks - s390: vfio-ap: implement mediated device open callback - s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl - s390: vfio-ap: zeroize the AP queues - s390: vfio-ap: implement VFIO_DEVICE_RESET ioctl - KVM: s390: Clear Crypto Control Block when using vSIE - KVM: s390: vsie: Do the CRYCB validation first - KVM: s390: vsie: Make use of CRYCB FORMAT2 clear - KVM: s390: vsie: Allow CRYCB FORMAT-2 - KVM: s390: vsie: allow CRYCB FORMAT-1 - KVM: s390: vsie: allow CRYCB FORMAT-0 - KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-1 - KVM: s390: vsie: allow guest FORMAT-1 CRYCB on host FORMAT-2 - KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-2 - KVM: s390: device attrs to enable/disable AP interpretation - KVM: s390: CPU model support for AP virtualization - s390: doc: detailed specifications for AP virtualization - KVM: s390: fix locking for crypto setting error path - KVM: s390: Tracing APCB changes - s390: vfio-ap: setup APCB mask using KVM dedicated function - [Config:] Enable CONFIG_S390_AP_IOMMU and set CONFIG_VFIO_AP to module. * Bypass of mount visibility through userns + mount propagation (LP: #1789161) - mount: Retest MNT_LOCKED in do_umount - mount: Don't allow copying MNT_UNBINDABLE|MNT_LOCKED mounts * CVE-2018-18955: nested user namespaces with more than five extents incorrectly grant privileges over inode (LP: #1801924) // CVE-2018-18955 - userns: also map extents in the reverse map to kernel IDs * kdump fail due to an IRQ storm (LP: #1797990) - SAUCE: x86/PCI: Export find_cap() to be used in early PCI code - SAUCE: x86/quirks: Add parameter to clear MSIs early on boot - SAUCE: x86/quirks: Scan all busses for early PCI quirks * crash in ENA driver on removing an interface (LP: #1802341) - SAUCE: net: ena: fix crash during ena_remove() * Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding (LP: #1797367) - s390/qeth: reduce hard-coded access to ccw channels - s390/qeth: sanitize strings in debug messages * Add checksum offload and TSO support for HiNIC adapters (LP: #1800664) - net-next/hinic: add checksum offload and TSO support * smartpqi updates for ubuntu 18.04.2 (LP: #1798208) - scsi: smartpqi: improve handling for sync requests - scsi: smartpqi: improve error checking for sync requests - scsi: smartpqi: add inspur advantech ids - scsi: smartpqi: fix critical ARM issue reading PQI index registers - scsi: smartpqi: bump driver version to 1.1.4-130 * [GLK/CLX] Enhanced IBRS (LP: #1786139) - x86/speculation: Remove SPECTRE_V2_IBRS in enum spectre_v2_mitigation - x86/speculation: Support Enhanced IBRS on future CPUs * Enable keyboard wakeup for S2Idle laptops (LP: #1798552) - Input: i8042 - enable keyboard wakeups by default when s2idle is used * Overlayfs in user namespace leaks directory content of inaccessible directories (LP: #1793458) // CVE-2018-6559 - SAUCE: overlayfs: ensure mounter privileges when reading directories * Update ENA driver to version 2.0.1K (LP: #1798182) - net: ena: remove ndo_poll_controller - net: ena: fix auto casting to boolean - net: ena: minor performance improvement - net: ena: complete host info to match latest ENA spec - net: ena: introduce Low Latency Queues data structures according to ENA spec - net: ena: add functions for handling Low Latency Queues in ena_com - net: ena: add functions for handling Low Latency Queues in ena_netdev - net: ena: use CSUM_CHECKED device indication to report skb's checksum status - net: ena: explicit casting and initialization, and clearer error handling - net: ena: limit refill Rx threshold to 256 to avoid latency issues - net: ena: change rx copybreak default to reduce kernel memory pressure - net: ena: remove redundant
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
** Tags removed: verification-needed-cosmic ** Tags added: verification-done-cosmic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
** Tags removed: kernel-key ** Tags added: kernel-da-key -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
--- Comment From kieni...@de.ibm.com 2018-11-16 07:45 EDT--- Fix has been successfully verified by the SSC team. With kernel 4.15.0-40.43 from proposed our test procedure no longer raises a kernel panic or fails in any other way. Thanks much for fixing ** Tags removed: verification-needed-bionic ** Tags added: verification-done-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
** Changed in: linux (Ubuntu) Importance: High => Critical ** Changed in: linux (Ubuntu Bionic) Importance: High => Critical ** Changed in: linux (Ubuntu Cosmic) Importance: High => Critical -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed- bionic'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: verification-needed-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- cosmic' to 'verification-done-cosmic'. If the problem still exists, change the tag 'verification-needed-cosmic' to 'verification-failed- cosmic'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: verification-needed-cosmic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
** Changed in: linux (Ubuntu) Status: In Progress => Fix Committed ** Changed in: ubuntu-z-systems Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
** Changed in: linux (Ubuntu Bionic) Status: In Progress => Fix Committed ** Changed in: linux (Ubuntu Cosmic) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
** Changed in: linux (Ubuntu Cosmic) Importance: Undecided => High ** Changed in: linux (Ubuntu Cosmic) Status: New => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
** Also affects: linux (Ubuntu Cosmic) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
SRU submitted: [SRU][Cosmic][Bionic][PATCH 0/6] Fixes for LP1797367 https://lists.ubuntu.com/archives/kernel-team/2018-November/096556.html -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
** Description changed:
- == Comment: #0 - Athira Rajeev
+ == SRU Justification ==
+
+ While running a series of stress tests for network on a bond device on Ubuntu
18.04.1 with kernel 4.15.0-36.39,
+ kernel panic is observed (btw. also on non-bond devices).
+ This looks like a race between disabling a qeth device and accessing debugfs.
+ This is critical and leads repeatedly to a crash (sooner or later).
+
+ == Fix ==
+
+ e19e5be8b4ca ("s390/qeth: sanitize strings in debug messages")
+
+ pre-reqs:
+ 750b162 ("s390/qeth: reduce hard-coded access to ccw channels")
+ d857e11 ("s390/qeth: remove outdated portname debug msg")
+ 9d0a58f ("s390/qeth: avoid using is_multicast_ether_addr_64bits on (u8 *)[6]")
+ 8174aa8 ("s390/qeth: consolidate qeth MAC address helpers")
+ 4641b02 ("s390/qeth: don't keep track of MAC address's cast type")
+
+ == Regression Potential ==
+
+ Low, because:
+
+ - limited to s390x
+ - and again limited to qeth driver
+ - patches a problem identified during testing
+ - fix was tested by IBM before submitted
+
+ == Test Case ==
+
+ run:
+#!/bin/bash
+var=0
+while :
+do
+ var=$((var + 1))
+ echo "DBG count is $var"
+ mkdir /tmp/DBGINFO
+ dbginfo.sh -d /tmp/DBGINFO
+ rm -rf /tmp/DBGINFO*
+ echo "chzdev now is $var"
+ chzdev -e
+ chzdev -d
+done
+ and in avg. in less than 20 cycles a crash happens (usually < 10).
+
+ __
+
+ == Comment: #0 - Athira Rajeev
---Problem Description---
While running a series of stress tests for network bonding on UBUNTU 18.04.1
with kernel 4.15.0-36.39, kernel panic is observed.
There are two instance of panic experienced with the same test procedures one
of which indicates to be a kernel BUG.
-
- Contact Information = Athira Rajeev , Waiki Wright <
wa...@us.ibm.com >
-
+
+ Contact Information = Athira Rajeev , Waiki Wright
+ < wa...@us.ibm.com >
+
---uname output---
#39-Ubuntu SMP Mon Sep 24 16:13:24 UTC 2018 4.15.0-36.39
-
- Machine Type = This issue is observed on z13 system
- ---Debugger---
- A debugger was configured,
-
+
+ Machine Type = This issue is observed on z13 system
+ ---Debugger---
+ A debugger was configured,
+
---Steps to Reproduce---
- This happens while running stress tests for network bonding. kernel memory
exposure attempt is detected and the BUG() is called from the code snippet:
mm/usercopy.c:72
- dump was configured and crash dump is available.
- Results of few crash commands like bt, log are added in Attachment
+ This happens while running stress tests for network bonding. kernel memory
exposure attempt is detected and the BUG() is called from the code snippet:
mm/usercopy.c:72
+ dump was configured and crash dump is available.
+ Results of few crash commands like bt, log are added in Attachment
Relevant part of dmesg pointing to kernel BUG
<<>>
[14746.977364] kernel BUG at
/build/linux-PABIrW/linux-4.15.0/mm/usercopy.c:72!
[14746.977377] illegal operation: 0001 ilc:1 [#1] SMP
[14746.977378] Modules linked in: macsec vsock_diag vsock sctp_diag sctp
dccp_diag dccp tcp_diag udp_diag raw_diag inet_diag unix_diag af_packet_diag
netlink_diag bonding binfmt_misc qeth_l3 8021q garp mrp stp llc xt_tcpudp
qeth_l2 nf_conntrack_ipv6 nf_defrag_ipv6 scsi_dh_rdac scsi_dh_emc scsi_dh_alua
s390_trng ghash_s390 prng sha512_s390 sha256_s390 sha1_s390 sha_common chsc_sch
eadm_sch nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c
crc32_vx_s390 qeth ccwgroup ip6table_filter ip6_tables vfio_ccw vfio_mdev mdev
vfio_iommu_type1 vfio iptable_filter sch_fq_codel ip_tables x_tables aes_s390
des_s390 des_generic dm_crypt dm_service_time dm_multipath zfcp
scsi_transport_fc qdio dasd_eckd_mod dasd_mod btrfs xor zstd_compress raid6_pq
zlib_deflate
[14746.977401] CPU: 1 PID: 20905 Comm: dump2tar Tainted: G OE
4.15.0-36-generic #39-Ubuntu
[14746.977403] Hardware name: IBM 3906 M02 757 (LPAR)
[14746.977404] Krnl PSW : 0f2d230d 6abe14d5
(__check_object_size+0x15a/0x1e0)
[14746.977408]R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0
RI:0 EA:3
[14746.977410] Krnl GPRS: 0002 00e95334 0064
0001e6518828
[14746.977412]0037cc8e 00a9577c
[14746.977413]647b 0001d8c120a8 0001
8088
[14746.977433]0001d8c0a020 0090da38 0037cc8e
00016fdfbcd0
[14746.977440] Krnl Code: 0037cc82: c0200038ef69larl
%r2,a9ab54
- 0037cc88: c0e5fff32838brasl
%r14,1e1cf8
- #0037cc8e: a7f40001brc
15,37cc90
- >0037cc92: e330d0080004lg
%r3,8(%r13)
- 0037cc98: e320d004lg
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
The dump from the 'IBM Box' file share has a different md5 hash (and a slightly different size) - compared to the one from the ftp server. But this time I could successfully 'rearrange' (-R) the dump file. Hence I replaced the the old file (PS27-JOHN-zSWTestAppliance-2018-10-16T15_07_56-AZIZ0104E-8196323d-3fcf-44d8-9b1a-bd1e6a694bad.bin) on mombin for internal access (and double checked the hash again). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
** Changed in: linux (Ubuntu) Status: Triaged => In Progress ** Changed in: ubuntu-z-systems Status: Confirmed => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
With more time spent on this, it feels like some question to be passed along to my former colleagues. Unfortunately the dump does not contain any data that is simple to track because the initial BUG() trigger already ended by the time the final panic is done. However debug_output() is rather unique for s390x and related to the fast dmesg like debug feature. Toggling the ccw device will attach/detach it to/from the qeth driver which registers/deregisters s390dbf views. De-registration is the problematic part. While that call is taking a mutex and then decrements a ref-count to the debug info area and that in theory should be protected by debug_open() incrementing that reference. But then there is also a snapshot copy copy included in the file info structure and I am not really sure how that exactly relates to the original structure and whether debug_output() would need additional ref-counting. Alternatively I also notice that qeth_core_exit() does not exactly undoes qeth_core_init(). It destroys the workqueue first while that was the first thing to create and the error path of qeth_core_init() would destroy the workqueue as the last thing, after having unregistered the debug views. Whether that really matters in this context, I am not sure. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
** Changed in: linux (Ubuntu Bionic) Status: Triaged => In Progress ** Changed in: ubuntu-z-systems Status: Triaged => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
So with the script we are able to re-create the panic on one of our systems. Which is good because I was not yet able to extract anything from the uploaded dump (using makedumpfile to convert from kdump_flat format seems to fail finding an end marker). Generally from the data seen so far all point to a race between disabling a qeth device and accessing debugfs. Since tear down of network device structures is handled asynchronously there is a chance that this might be racing with the access to debug data. This is something that is generally known to upstream but did not receive that much attention either. Access to debugfs should be rather limited to, as its name indicates, debug use. In that light, the question is whether this really should be treated release critical. Trying to get this race free might turn into a bigger task and at the same time should not be something any customer should run into during normal operation. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
** Changed in: ubuntu-z-systems Importance: High => Critical -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
--- Comment From heinz-werner_se...@de.ibm.com 2018-10-18 08:27 EDT--- Increased priority for getting thsi fixed very urgent. Ship issueMany thx in advance. ** Tags removed: severity-high targetmilestone-inin--- ** Tags added: severity-critical targetmilestone-inin1804 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
The dump using 4.15.0-38 from proposed is now accessible (Canonical internal) via my home (fheimes) at 'mombin': $ ls PS27-JOHN-*.bin PS27-JOHN-zSWTestAppliance-2018-10-16T15_07_56-AZIZ0104E-8196323d-3fcf-44d8-9b1a-bd1e6a694bad.bin -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
Thanks for testing on 4.15.0-38 from proposed. Requested the retrieval of the dump file ... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
We would like to know the results from kernel 4.15.0-38 from proposed, since that comes with several patches on top. If dumps from that kernel are available, please upload to the ftp server (#5) and I make them internally available, too. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
Any update? We do have more dumps if needed. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
I opened a RT ticket to retrieve the files. They are now accessible (Canonical internal) via my home (fheimes) at 'mombin': 94293893 Oct 12 11:22 PM96-TESTREST-zSWTestAppliance-2018-10-08T06_50_42-AZIZ0104E-39181eaf-7093-455c-975e-f454936e1c92.bin 129272177 Oct 12 11:26 PS27-STEVE-zSWTestAppliance-2018-10-03T23_01_58-AZIZ0104E-1763fc60-2b01-415b-abd7-07c7502b3045.bin -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
You may use apport (apport-cli / apport-bug) and upload any data to this launchpad bug. https://help.ubuntu.com/lts/serverguide/reporting-bugs.html.en Alternatively you can do an (anonymous) ftp upload of very huge files here: archive.admin.canonical.com -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
The stack trace looks to indicate a debugfs read triggering an out of bounds read against a kernel object backed by the slab. That is that the read is offset into the object such that the attempted read would have traversed onto the following object exposing kernel memory for an unrelated item. Do we know what the userspace application triggering this is actually doing when the fault triggers? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
Has this been observed only with this kernel version (or not tried), Or in other words did any 4.15 kernel not cause these issues. Also we currently have the next scheduled kernel being in testing (-proposed). It contains a bunch of accumulated stable updates, so it would be good to see whether it still fails or maybe solves the issue already (https://launchpad.net/ubuntu/+source/linux/4.15.0-38.41). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
** Changed in: ubuntu-z-systems Status: New => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
** Changed in: linux (Ubuntu) Importance: Undecided => Critical ** Tags added: kernel-key ** Changed in: linux (Ubuntu) Importance: Critical => High ** Also affects: linux (Ubuntu Bionic) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Bionic) Importance: Undecided => High ** Changed in: linux (Ubuntu Bionic) Status: New => Triaged ** Changed in: linux (Ubuntu) Status: New => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1797367] Re: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
** Also affects: ubuntu-z-systems Importance: Undecided Status: New ** Changed in: ubuntu-z-systems Importance: Undecided => High ** Changed in: ubuntu-z-systems Assignee: (unassigned) => Canonical Kernel Team (canonical-kernel-team) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1797367 Title: Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1797367/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
