[Kernel-packages] [Bug 1732238] Re: arm64: Unfair rwlock can stall the system
** Changed in: linux (Ubuntu) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1732238 Title: arm64: Unfair rwlock can stall the system Status in linux package in Ubuntu: Fix Released Status in linux source package in Artful: Fix Released Bug description: [Impact] There is a long-standing upstream bug with the ARM64 specific implementation of RW locks. The implementation can starve writers under lock contention leading to RCU stalls, driver timeouts and general system instability. [Test Case] $ stress-ng --kill 0 -t 300 -v You'll see the console fill with messages like: [ 2534.423119] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2534.428606]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=6770 [ 2534.437029](detected by 0, t=15005 jiffies, g=1479, c=1478, q=473) [ 2714.623691] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2714.629181]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=12819 [ 2714.637692](detected by 116, t=60058 jiffies, g=1479, c=1478, q=1736) [ 2747.216955] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:5:1464] [ 2775.399061] watchdog: BUG: soft lockup - CPU#13 stuck for 123s! [systemd-network:2936] [Regression Risk] The proposed fix for this comprises clean cherry-picks from the v4.15 merge window. The code modified for this fix is restricted to x86 & arm64, as they are the only Ubuntu architectures that define ARCH_USE_QUEUED_LOCKS. Fix verified on a 228 CPU arm64 (ThunderX2) server and regression tested on a 128-cpu x86 system using stress-ng and locktorture. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1732238/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1732238] Re: arm64: Unfair rwlock can stall the system
This bug was fixed in the package linux - 4.13.0-36.40 --- linux (4.13.0-36.40) artful; urgency=medium * linux: 4.13.0-36.40 -proposed tracker (LP: #1750010) * Rebuild without "CVE-2017-5754 ARM64 KPTI fixes" patch set linux (4.13.0-35.39) artful; urgency=medium * linux: 4.13.0-35.39 -proposed tracker (LP: #1748743) * CVE-2017-5715 (Spectre v2 Intel) - Revert "UBUNTU: SAUCE: turn off IBPB when full retpoline is present" - SAUCE: turn off IBRS when full retpoline is present - [Packaging] retpoline files must be sorted - [Packaging] pull in retpoline files linux (4.13.0-34.37) artful; urgency=medium * linux: 4.13.0-34.37 -proposed tracker (LP: #1748475) * libata: apply MAX_SEC_1024 to all LITEON EP1 series devices (LP: #1743053) - libata: apply MAX_SEC_1024 to all LITEON EP1 series devices * KVM patches for s390x to provide facility bits 81 (ppa15) and 82 (bpb) (LP: #1747090) - KVM: s390: wire up bpb feature * artful 4.13 i386 kernels crash after memory hotplug remove (LP: #1747069) - Revert "mm, memory_hotplug: do not associate hotadded memory to zones until online" * CVE-2017-5715 (Spectre v2 Intel) - x86/feature: Enable the x86 feature to control Speculation - x86/feature: Report presence of IBPB and IBRS control - x86/enter: MACROS to set/clear IBRS and set IBPB - x86/enter: Use IBRS on syscall and interrupts - x86/idle: Disable IBRS entering idle and enable it on wakeup - x86/idle: Disable IBRS when offlining cpu and re-enable on wakeup - x86/mm: Set IBPB upon context switch - x86/mm: Only set IBPB when the new thread cannot ptrace current thread - x86/entry: Stuff RSB for entry to kernel for non-SMEP platform - x86/kvm: add MSR_IA32_SPEC_CTRL and MSR_IA32_PRED_CMD to kvm - x86/kvm: Set IBPB when switching VM - x86/kvm: Toggle IBRS on VM entry and exit - x86/spec_ctrl: Add sysctl knobs to enable/disable SPEC_CTRL feature - x86/spec_ctrl: Add lock to serialize changes to ibrs and ibpb control - x86/cpu/AMD: Add speculative control support for AMD - x86/microcode: Extend post microcode reload to support IBPB feature - KVM: SVM: Do not intercept new speculative control MSRs - x86/svm: Set IBRS value on VM entry and exit - x86/svm: Set IBPB when running a different VCPU - KVM: x86: Add speculative control CPUID support for guests - SAUCE: turn off IBPB when full retpoline is present * Artful 4.13 fixes for tun (LP: #1748846) - tun: call dev_get_valid_name() before register_netdevice() - tun: allow positive return values on dev_get_valid_name() call - tun/tap: sanitize TUNSETSNDBUF input * boot failure on AMD Raven + WestonXT (LP: #1742759) - SAUCE: drm/amdgpu: add atpx quirk handling (v2) linux (4.13.0-33.36) artful; urgency=low * linux: 4.13.0-33.36 -proposed tracker (LP: #1746903) [ Stefan Bader ] * starting VMs causing retpoline4 to reboot (LP: #1747507) // CVE-2017-5715 (Spectre v2 retpoline) - x86/retpoline: Fill RSB on context switch for affected CPUs - x86/retpoline: Add LFENCE to the retpoline/RSB filling RSB macros - x86/retpoline: Optimize inline assembler for vmexit_fill_RSB - x86/retpoline: Remove the esp/rsp thunk - x86/retpoline: Simplify vmexit_fill_RSB() * Missing install-time driver for QLogic QED 25/40/100Gb Ethernet NIC (LP: #1743638) - [d-i] Add qede to nic-modules udeb * hisi_sas: driver robustness fixes (LP: #1739807) - scsi: hisi_sas: fix reset and port ID refresh issues - scsi: hisi_sas: avoid potential v2 hw interrupt issue - scsi: hisi_sas: fix v2 hw underflow residual value - scsi: hisi_sas: add v2 hw DFX feature - scsi: hisi_sas: add irq and tasklet cleanup in v2 hw - scsi: hisi_sas: service interrupt ITCT_CLR interrupt in v2 hw - scsi: hisi_sas: fix internal abort slot timeout bug - scsi: hisi_sas: us start_phy in PHY_FUNC_LINK_RESET - scsi: hisi_sas: fix NULL check in SMP abort task path - scsi: hisi_sas: fix the risk of freeing slot twice - scsi: hisi_sas: kill tasklet when destroying irq in v3 hw - scsi: hisi_sas: complete all tasklets prior to host reset * [Artful/Zesty] ACPI APEI error handling bug fixes (LP: #1732990) - ACPI: APEI: fix the wrong iteration of generic error status block - ACPI / APEI: clear error status before acknowledging the error * [Zesty/Artful] On ARM64 PCIE physical function passthrough guest fails to boot (LP: #1732804) - vfio/pci: Virtualize Maximum Payload Size - vfio/pci: Virtualize Maximum Read Request Size * hisi_sas: Add ATA command support for SMR disks (LP: #1739891) - scsi: hisi_sas: support zone management commands * thunderx2: i2c driver PEC and ACPI clock fixes (LP: #1738073) - ACPI / APD: Add clock frequency for ThunderX2 I2C controller - i2c: xlp9xx: Get clock frequency with clk API - i2c: xlp9xx: Handle
[Kernel-packages] [Bug 1732238] Re: arm64: Unfair rwlock can stall the system
Artful verification: I was able to successfully run the above stress-ng commmand w/o any errors on the console. ** Tags removed: verification-needed-artful ** Tags added: verification-done-artful -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1732238 Title: arm64: Unfair rwlock can stall the system Status in linux package in Ubuntu: Fix Committed Status in linux source package in Artful: Fix Committed Bug description: [Impact] There is a long-standing upstream bug with the ARM64 specific implementation of RW locks. The implementation can starve writers under lock contention leading to RCU stalls, driver timeouts and general system instability. [Test Case] $ stress-ng --kill 0 -t 300 -v You'll see the console fill with messages like: [ 2534.423119] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2534.428606]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=6770 [ 2534.437029](detected by 0, t=15005 jiffies, g=1479, c=1478, q=473) [ 2714.623691] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2714.629181]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=12819 [ 2714.637692](detected by 116, t=60058 jiffies, g=1479, c=1478, q=1736) [ 2747.216955] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:5:1464] [ 2775.399061] watchdog: BUG: soft lockup - CPU#13 stuck for 123s! [systemd-network:2936] [Regression Risk] The proposed fix for this comprises clean cherry-picks from the v4.15 merge window. The code modified for this fix is restricted to x86 & arm64, as they are the only Ubuntu architectures that define ARCH_USE_QUEUED_LOCKS. Fix verified on a 228 CPU arm64 (ThunderX2) server and regression tested on a 128-cpu x86 system using stress-ng and locktorture. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1732238/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1732238] Re: arm64: Unfair rwlock can stall the system
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- artful' to 'verification-done-artful'. If the problem still exists, change the tag 'verification-needed-artful' to 'verification-failed- artful'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: verification-needed-artful -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1732238 Title: arm64: Unfair rwlock can stall the system Status in linux package in Ubuntu: Fix Committed Status in linux source package in Artful: Fix Committed Bug description: [Impact] There is a long-standing upstream bug with the ARM64 specific implementation of RW locks. The implementation can starve writers under lock contention leading to RCU stalls, driver timeouts and general system instability. [Test Case] $ stress-ng --kill 0 -t 300 -v You'll see the console fill with messages like: [ 2534.423119] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2534.428606]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=6770 [ 2534.437029](detected by 0, t=15005 jiffies, g=1479, c=1478, q=473) [ 2714.623691] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2714.629181]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=12819 [ 2714.637692](detected by 116, t=60058 jiffies, g=1479, c=1478, q=1736) [ 2747.216955] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:5:1464] [ 2775.399061] watchdog: BUG: soft lockup - CPU#13 stuck for 123s! [systemd-network:2936] [Regression Risk] The proposed fix for this comprises clean cherry-picks from the v4.15 merge window. The code modified for this fix is restricted to x86 & arm64, as they are the only Ubuntu architectures that define ARCH_USE_QUEUED_LOCKS. Fix verified on a 228 CPU arm64 (ThunderX2) server and regression tested on a 128-cpu x86 system using stress-ng and locktorture. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1732238/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1732238] Re: arm64: Unfair rwlock can stall the system
** Changed in: linux (Ubuntu Artful) Status: Confirmed => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1732238 Title: arm64: Unfair rwlock can stall the system Status in linux package in Ubuntu: Fix Committed Status in linux source package in Artful: Fix Committed Bug description: [Impact] There is a long-standing upstream bug with the ARM64 specific implementation of RW locks. The implementation can starve writers under lock contention leading to RCU stalls, driver timeouts and general system instability. [Test Case] $ stress-ng --kill 0 -t 300 -v You'll see the console fill with messages like: [ 2534.423119] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2534.428606]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=6770 [ 2534.437029](detected by 0, t=15005 jiffies, g=1479, c=1478, q=473) [ 2714.623691] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2714.629181]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=12819 [ 2714.637692](detected by 116, t=60058 jiffies, g=1479, c=1478, q=1736) [ 2747.216955] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:5:1464] [ 2775.399061] watchdog: BUG: soft lockup - CPU#13 stuck for 123s! [systemd-network:2936] [Regression Risk] The proposed fix for this comprises clean cherry-picks from the v4.15 merge window. The code modified for this fix is restricted to x86 & arm64, as they are the only Ubuntu architectures that define ARCH_USE_QUEUED_LOCKS. Fix verified on a 228 CPU arm64 (ThunderX2) server and regression tested on a 128-cpu x86 system using stress-ng and locktorture. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1732238/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1732238] Re: arm64: Unfair rwlock can stall the system
** Changed in: linux (Ubuntu) Status: Confirmed => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1732238 Title: arm64: Unfair rwlock can stall the system Status in linux package in Ubuntu: Fix Committed Status in linux source package in Artful: Confirmed Bug description: [Impact] There is a long-standing upstream bug with the ARM64 specific implementation of RW locks. The implementation can starve writers under lock contention leading to RCU stalls, driver timeouts and general system instability. [Test Case] $ stress-ng --kill 0 -t 300 -v You'll see the console fill with messages like: [ 2534.423119] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2534.428606]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=6770 [ 2534.437029](detected by 0, t=15005 jiffies, g=1479, c=1478, q=473) [ 2714.623691] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2714.629181]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=12819 [ 2714.637692](detected by 116, t=60058 jiffies, g=1479, c=1478, q=1736) [ 2747.216955] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:5:1464] [ 2775.399061] watchdog: BUG: soft lockup - CPU#13 stuck for 123s! [systemd-network:2936] [Regression Risk] The proposed fix for this comprises clean cherry-picks from the v4.15 merge window. The code modified for this fix is restricted to x86 & arm64, as they are the only Ubuntu architectures that define ARCH_USE_QUEUED_LOCKS. Fix verified on a 228 CPU arm64 (ThunderX2) server and regression tested on a 128-cpu x86 system using stress-ng and locktorture. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1732238/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1732238] Re: arm64: Unfair rwlock can stall the system
** Description changed: [Impact] There is a long-standing upstream bug with the ARM64 specific implementation of RW locks. The implementation can starve writers under lock contention leading to RCU stalls, driver timeouts and general system instability. [Test Case] $ stress-ng --kill 0 -t 300 -v You'll see the console fill with messages like: [ 2534.423119] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2534.428606]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=6770 [ 2534.437029](detected by 0, t=15005 jiffies, g=1479, c=1478, q=473) [ 2714.623691] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2714.629181]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=12819 [ 2714.637692](detected by 116, t=60058 jiffies, g=1479, c=1478, q=1736) [ 2747.216955] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:5:1464] [ 2775.399061] watchdog: BUG: soft lockup - CPU#13 stuck for 123s! [systemd-network:2936] [Regression Risk] - TBD + The proposed fix for this comprises clean cherry-picks from the v4.15 merge window. The code modified for this fix is restricted to x86 & arm64, as they are the only Ubuntu architectures that define ARCH_USE_QUEUED_LOCKS. Fix verified on a 228 CPU arm64 (ThunderX2) server and regression tested on a 128-cpu x86 system using stress-ng and locktorture. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1732238 Title: arm64: Unfair rwlock can stall the system Status in linux package in Ubuntu: Confirmed Status in linux source package in Artful: Confirmed Bug description: [Impact] There is a long-standing upstream bug with the ARM64 specific implementation of RW locks. The implementation can starve writers under lock contention leading to RCU stalls, driver timeouts and general system instability. [Test Case] $ stress-ng --kill 0 -t 300 -v You'll see the console fill with messages like: [ 2534.423119] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2534.428606]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=6770 [ 2534.437029](detected by 0, t=15005 jiffies, g=1479, c=1478, q=473) [ 2714.623691] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2714.629181]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=12819 [ 2714.637692](detected by 116, t=60058 jiffies, g=1479, c=1478, q=1736) [ 2747.216955] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:5:1464] [ 2775.399061] watchdog: BUG: soft lockup - CPU#13 stuck for 123s! [systemd-network:2936] [Regression Risk] The proposed fix for this comprises clean cherry-picks from the v4.15 merge window. The code modified for this fix is restricted to x86 & arm64, as they are the only Ubuntu architectures that define ARCH_USE_QUEUED_LOCKS. Fix verified on a 228 CPU arm64 (ThunderX2) server and regression tested on a 128-cpu x86 system using stress-ng and locktorture. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1732238/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1732238] Re: arm64: Unfair rwlock can stall the system
Example showing driver timeouts: ubuntu@boomer:~$ stress-ng --kill 0 -t 300 -v stress-ng: debug: [3344] 224 processors online, 224 processors configured stress-ng: info: [3344] dispatching hogs: 224 kill stress-ng: debug: [3344] /sys/devices/system/cpu/cpu0/cache does not exist stress-ng: info: [3344] cache allocate: using built-in defaults as unable to determine cache details stress-ng: info: [3344] cache allocate: default cache size: 2048K stress-ng: debug: [3344] starting stressors stress-ng: debug: [3345] stress-ng-kill: started [3345] (instance 0) stress-ng: debug: [3346] stress-ng-kill: started [3346] (instance 1) stress-ng: debug: [3347] stress-ng-kill: started [3347] (instance 2) stress-ng: debug: [3348] stress-ng-kill: started [3348] (instance 3) stress-ng: debug: [3349] stress-ng-kill: started [3349] (instance 4) [ 1447.474535] INFO: rcu_sched detected stalls on CPUs/tasks: [ 1447.480020] 27-...: (66 GPs behind) idle=1ba/140/0 softirq=3878/3878 fqs=7264 [ 1447.488363] 136-...: (93 GPs behind) idle=972/140/0 softirq=2760/2760 fqs=7265 [ 1447.496788] (detected by 161, t=15007 jiffies, g=1128, c=1127, q=790) [ 1451.646152] xhci_hcd :01:04.1: xHCI host controller not responding, assume dead [ 1451.653819] xhci_hcd :01:04.1: HC died; cleaning up [ 1451.653829] usb 3-1-port1: cannot reset (err = -22) [ 1451.653832] usb 3-1-port1: cannot reset (err = -22) [ 1451.653833] usb 3-1-port1: cannot reset (err = -22) [ 1451.653834] usb 3-1-port1: cannot reset (err = -22) [ 1451.653835] usb 3-1-port1: cannot reset (err = -22) [ 1451.653837] usb 3-1-port1: Cannot enable. Maybe the USB cable is bad? [ 1451.653839] usb 3-1-port1: cannot disable (err = -22) [ 1451.653848] usb 3-1-port2: cannot reset (err = -22) [ 1451.653851] usb 3-1-port2: cannot reset (err = -22) [ 1451.653852] usb 3-1-port2: cannot reset (err = -22) [ 1451.653854] usb 3-1-port2: cannot reset (err = -22) [ 1451.653855] usb 3-1-port2: cannot reset (err = -22) [ 1451.653856] usb 3-1-port2: Cannot enable. Maybe the USB cable is bad? [ 1451.653858] usb 3-1-port2: cannot disable (err = -22) [ 1451.653860] usb 3-1-port1: cannot reset (err = -22) [ 1451.653861] usb 3-1-port1: cannot reset (err = -22) [ 1451.653862] usb 3-1-port1: cannot reset (err = -22) [ 1451.653863] usb 3-1-port1: cannot reset (err = -22) [ 1451.653864] usb 3-1-port1: cannot reset (err = -22) [ 1451.653865] usb 3-1-port1: Cannot enable. Maybe the USB cable is bad? [ 1451.653866] usb 3-1-port1: cannot disable (err = -22) [ 1451.653868] usb 3-1-port2: cannot reset (err = -22) [ 1451.653870] usb 3-1-port2: cannot reset (err = -22) [ 1451.653871] usb 3-1-port2: cannot reset (err = -22) [ 1451.653872] usb 3-1-port2: cannot reset (err = -22) [ 1451.653873] usb 3-1-port2: cannot reset (err = -22) [ 1451.653873] usb 3-1-port2: Cannot enable. Maybe the USB cable is bad? [ 1451.653875] usb 3-1-port2: cannot disable (err = -22) [ 1451.653876] usb 3-1-port1: cannot reset (err = -22) [ 1451.653878] usb 3-1-port1: cannot reset (err = -22) [ 1451.653879] usb 3-1-port1: cannot reset (err = -22) [ 1451.653880] usb 3-1-port1: cannot reset (err = -22) [ 1451.653881] usb 3-1-port1: cannot reset (err = -22) [ 1451.653882] usb 3-1-port1: Cannot enable. Maybe the USB cable is bad? [ 1451.653883] usb 3-1-port1: cannot disable (err = -22) [ 1451.653885] usb 3-1-port2: cannot reset (err = -22) [ 1451.653886] usb 3-1-port2: cannot reset (err = -22) [ 1451.653887] usb 3-1-port2: cannot reset (err = -22) [ 1451.653888] usb 3-1-port2: cannot reset (err = -22) [ 1451.653889] usb 3-1-port2: cannot reset (err = -22) [ 1451.653890] usb 3-1-port2: Cannot enable. Maybe the USB cable is bad? [ 1451.653892] usb 3-1-port2: cannot disable (err = -22) [ 1451.653893] usb 3-1-port1: cannot reset (err = -22) [ 1451.653895] usb 3-1-port1: cannot reset (err = -22) [ 1451.653896] usb 3-1-port1: cannot reset (err = -22) [ 1451.653897] usb 3-1-port1: cannot reset (err = -22) [ 1451.653898] usb 3-1-port1: cannot reset (err = -22) [ 1451.653899] usb 3-1-port1: Cannot enable. Maybe the USB cable is bad? [ 1451.653900] usb 3-1-port1: cannot disable (err = -22) [ 1451.653902] usb 3-1-port2: cannot reset (err = -22) [ 1451.653903] usb 3-1-port1: cannot disable (err = -22) [ 1451.653903] usb 3-1-port2: cannot reset (err = -22) [ 1451.653905] usb 3-1-port2: cannot reset (err = -22) [ 1451.653906] usb 3-1-port2: cannot reset (err = -22) [ 1451.653907] usb 3-1-port2: cannot reset (err = -22) [ 1451.653908] usb 3-1-port2: Cannot enable. Maybe the USB cable is bad? [ 1451.653909] usb 3-1-port2: cannot disable (err = -22) [ 1451.653910] usb 3-1-port2: cannot disable (err = -22) [ 1451.653920] hub 3-1:1.0: hub_ext_port_status failed (err = -22) [ 1451.653923] hub 3-1:1.0: hub_ext_port_status failed (err = -22) [ 1626.734665] INFO: task kworker/u448:0:6 blocked for more than 120 seconds. [ 1626.741536] Not tainted 4.13.0-20-generic #23-Ubuntu [ 1626.747047] "echo 0 >
[Kernel-packages] [Bug 1732238] Re: arm64: Unfair rwlock can stall the system
** Description changed: [Impact] - There is a long-standing upstream bug with the ARM64 specific implementation of RW locks. The implementation can starve writers under lock contention leading to RCU stalls and general system instability. + There is a long-standing upstream bug with the ARM64 specific implementation of RW locks. The implementation can starve writers under lock contention leading to RCU stalls, driver timeouts and general system instability. [Test Case] $ stress-ng --kill 0 -t 300 -v You'll see the console fill with messages like: [ 2534.423119] INFO: rcu_sched detected stalls on CPUs/tasks: - [ 2534.428606]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=6770 + [ 2534.428606]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=6770 [ 2534.437029](detected by 0, t=15005 jiffies, g=1479, c=1478, q=473) [ 2714.623691] INFO: rcu_sched detected stalls on CPUs/tasks: - [ 2714.629181]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=12819 + [ 2714.629181]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=12819 [ 2714.637692](detected by 116, t=60058 jiffies, g=1479, c=1478, q=1736) [ 2747.216955] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:5:1464] [ 2775.399061] watchdog: BUG: soft lockup - CPU#13 stuck for 123s! [systemd-network:2936] [Regression Risk] TBD -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1732238 Title: arm64: Unfair rwlock can stall the system Status in linux package in Ubuntu: Confirmed Status in linux source package in Artful: Confirmed Bug description: [Impact] There is a long-standing upstream bug with the ARM64 specific implementation of RW locks. The implementation can starve writers under lock contention leading to RCU stalls, driver timeouts and general system instability. [Test Case] $ stress-ng --kill 0 -t 300 -v You'll see the console fill with messages like: [ 2534.423119] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2534.428606]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=6770 [ 2534.437029](detected by 0, t=15005 jiffies, g=1479, c=1478, q=473) [ 2714.623691] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2714.629181]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=12819 [ 2714.637692](detected by 116, t=60058 jiffies, g=1479, c=1478, q=1736) [ 2747.216955] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:5:1464] [ 2775.399061] watchdog: BUG: soft lockup - CPU#13 stuck for 123s! [systemd-network:2936] [Regression Risk] TBD To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1732238/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1732238] Re: arm64: Unfair rwlock can stall the system
This has been resolved with the following commits upstream: commit d133166146333e1f13fc81c0e6c43c8d99290a8a Author: Will Deaconlocking/qrwlock: Prevent slowpath writers getting held up by fastpath commit 087133ac90763cd339b6b67f2998f87dcc136c52 Author: Will Deacon locking/qrwlock, arm64: Move rwlock implementation over to qrwlocks commit b519b56e378ee82caf9b079b04f5db87dedc3251 Author: Will Deacon locking/qrwlock: Use atomic_cond_read_acquire() when spinning in qrwlock commit 4df714be4dcf40bfb0d4af0f851a6e1977afa02e Author: Will Deacon locking/atomic: Add atomic_cond_read_acquire() commit e0d02285f16e8d5810f3d5d5e8a5886ca0015d3b Author: Will Deacon locking/qrwlock: Use 'struct qrwlock' instead of 'struct __qrwlock -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1732238 Title: arm64: Unfair rwlock can stall the system Status in linux package in Ubuntu: Confirmed Status in linux source package in Artful: Confirmed Bug description: [Impact] There is a long-standing upstream bug with the ARM64 specific implementation of RW locks. The implementation can starve writers under lock contention leading to RCU stalls and general system instability. [Test Case] $ stress-ng --kill 0 -t 300 -v You'll see the console fill with messages like: [ 2534.423119] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2534.428606]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=6770 [ 2534.437029](detected by 0, t=15005 jiffies, g=1479, c=1478, q=473) [ 2714.623691] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2714.629181]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=12819 [ 2714.637692](detected by 116, t=60058 jiffies, g=1479, c=1478, q=1736) [ 2747.216955] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:5:1464] [ 2775.399061] watchdog: BUG: soft lockup - CPU#13 stuck for 123s! [systemd-network:2936] [Regression Risk] TBD To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1732238/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1732238] Re: arm64: Unfair rwlock can stall the system
** Also affects: linux (Ubuntu Artful) Importance: Undecided Status: New ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed ** Changed in: linux (Ubuntu Artful) Status: New => Confirmed ** Changed in: linux (Ubuntu) Assignee: (unassigned) => dann frazier (dannf) ** Changed in: linux (Ubuntu Artful) Assignee: (unassigned) => dann frazier (dannf) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1732238 Title: arm64: Unfair rwlock can stall the system Status in linux package in Ubuntu: Confirmed Status in linux source package in Artful: Confirmed Bug description: [Impact] There is a long-standing upstream bug with the ARM64 specific implementation of RW locks. The implementation can starve writers under lock contention leading to RCU stalls and general system instability. [Test Case] $ stress-ng --kill 0 -t 300 -v You'll see the console fill with messages like: [ 2534.423119] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2534.428606]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=6770 [ 2534.437029](detected by 0, t=15005 jiffies, g=1479, c=1478, q=473) [ 2714.623691] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2714.629181]192-...: (1 ticks this GP) idle=b6e/140/0 softirq=578/578 fqs=12819 [ 2714.637692](detected by 116, t=60058 jiffies, g=1479, c=1478, q=1736) [ 2747.216955] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:5:1464] [ 2775.399061] watchdog: BUG: soft lockup - CPU#13 stuck for 123s! [systemd-network:2936] [Regression Risk] TBD To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1732238/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp