This bug was fixed in the package linux - 3.13.0-51.84
---
linux (3.13.0-51.84) trusty; urgency=low
[ Luis Henriques ]
* Release Tracking Bug
- LP: #1444141
* Merged back Ubuntu-3.13.0-49.83 security release
linux (3.13.0-50.82) trusty; urgency=low
[ Brad Figg ]
*
My deployment is still running strong after over 36 hours. No crashes. I
will leave it running for a few more days to see if it happens after a
few days... and will report back.
@arges, thanks for this fix!
--
You received this bug notification because you are a member of Kernel
Packages, which
@arges
For me it's related at least part of it...
If I don't update the kernel to proposed-updates I have the following messages :
If I use one CPU instead of two, I don't have those messages.
BUG: soft lockup CPU#1 stuck for 22s! [qemu-system-x86:6889]
INFO: rcu_sched detected stalls on
@baco-1
1) What kind of hardware are you running on L0? ('ubuntu-bug linux' and filing
a bug would collect the necessary info)
2) What kind of load are you seeing in L0, L1?
3) Can you give me the output of 'tail /sys/module/kvm_intel/parameters/*' ?
4) You could setup crashdump to dump on a
@baco-1
These backtraces look a bit different than the original bug. Can you
file a new bug with how you are reproducing this and gather complete
logs?
--chris
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
I still have the same issue with kernel 3.16.0-36-generic or 3.13.0-51-generic
(proposed-updates)
# KVM HOST (3.16.0-36-generic)
sudo apt-get install linux-signed-generic-lts-utopic/trusty-proposed
# KVM GUEST (3.16.0-36-generic)
sudo apt-get install linux-virtual-lts-utopic/trusty-proposed
After speaking to Gema, she will re-test with this kernel installed in L0 in
addition to L1.
NOTE: This fix needs to be present for L0/L1 kernels.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
I have been trying to verify this kernel and I haven't seen exactly the
soft lockup crash, but this other one, which may or may not be related
but wanted to make a note of it:
[ 2406.041444] Kernel panic - not syncing: hung_task: blocked tasks
[ 2406.043163] CPU: 1 PID: 35 Comm: khungtaskd Not
Verified on my reproducers. I'm marking the development task as fixed
for this bug. I'll move the upstream investigation to another bug.
** Changed in: linux (Ubuntu)
Assignee: Chris J Arges (arges) = (unassigned)
** Changed in: linux (Ubuntu)
Status: Confirmed = Fix Released
**
This bug is awaiting verification that the kernel in -proposed solves
the problem. Please test the kernel and update this bug with the
results. If the problem is solved, change the tag 'verification-needed-
trusty' to 'verification-done-trusty'.
If verification is not done by 5 working days from
@Andy: So 3.16.0-34 is the kernel with the fix? Any chance that it will
also be backported to the 3.13 series?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1413540
Title:
Trusty soft
** Changed in: linux (Ubuntu Trusty)
Status: In Progress = Fix Committed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1413540
Title:
Trusty soft lockup issues with nested KVM
** Also affects: linux (Ubuntu Trusty)
Importance: Undecided
Status: New
** Changed in: linux (Ubuntu Trusty)
Assignee: (unassigned) = Chris J Arges (arges)
** Changed in: linux (Ubuntu Trusty)
Importance: Undecided = High
** Changed in: linux (Ubuntu Trusty)
Status:
Ran into this bug too on 3.13.0-48. My workaround is to run QEMU on top
of KVM (instead of kvm on top of KVM)
devstack local.conf:
[[post-config|$NOVA_CONF]]
[libvirt]
virt_type = qemu
nova.conf
[libvirt]
virt_type = qemu
--
You received this bug notification because you are a member of Kernel
With a revert of b6b8a145 ('Rework interception of IRQs and NMIs'), the
issue does not occur readily with the test case. I was able to run for
1+ hour. Generally I can reproduce within 15m.
With 9242b5b6 ('KVM: x86: Check for nested events if there is an
injectable interrupt') applied, I can run
** Description changed:
[Impact]
Upstream discussion: https://lkml.org/lkml/2015/2/11/247
Certain workloads that need to execute functions on a non-local CPU
using smp_call_function_* can result in soft lockups with the following
backtrace:
PID: 22262 TASK: 8804274bb000
@arosen,
This looks like a different softlockup, and also the machine seems to recover
from it. Please file a new bug and be sure to attach logs to the bug. Describe
in detail how to reproduce this as well, what kind of host machine do you have?
what VM definition are you using? Etc etc.
**
@chris: done
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1439394
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1413540
Title:
Trusty soft lockup issues with nested KVM
Status
I am also hitting this issue in my CI a lot. Here is the trace I'm
getting in syslog: http://logs2.aaronorosen.com/85/169585/1/check/dsvm-
tempest-full-congress-
nodepool/94f8441/logs/syslog.txt.gz#_Apr__1_02_43_44
Is there a work around for this?
--
You received this bug notification because
Hi,
Just wanted to chime in that this bug also affected me - running
OpenStack Juno w/KVM inside a KVM hypervisor.
CPU on the host machine is:
vendor_id : GenuineIntel
cpu family : 6
model : 58
model name : Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
running 14.04 with the
@fifieldt
Hi, that is the same bug. Things to reduce the hangs right now are:
- Disabling KSM in L1 guest
- Using 3.16 kernel on the L0 host
- Pinning L1 vCPUs to L0 host CPU
Note this doesn't fix the issue, it only decreases (potentially) the frequency
of these lockups.
--chris
--
You
@smb - after repeating the test a few times, I too ran out of space with
the default 8GB VM disk size, resulting in a paused VM. You'll have to
re-create the VMs a little bit differently (--disk GB).
ex:
@L0:
sudo uvt-kvm destroy trusty-vm
sudo uvt-kvm create --memory 2048 --disk 40 trusty-vm
Hrmn... When I repeated the setup I seem to have triggered some kind of
lockup even while bringing up l2. Of course hard to say without details
of Ryan's dump. However mine seems to have backtraces in the log which
remind me an awful lot of an issue related to punching holes into ext4
based qcow
Yeah, will do. Just got distracted and wanted to ensure that the repro
was not accidentally another form of failure path to the out of space
issue.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
Stefan,
This looks like a separate bug (as we discussed). Please file another bug for
this when you have time.
** Description changed:
[Impact]
- Users of nested KVM for testing openstack have soft lockups as follows:
+ Certain workloads that need to execute functions on a non-local CPU using
I've added instructions for a workaround. The code paths I've seen in
crashes has been the following:
kvm_sched_in
- kvm_arch_vcpu_load
- vmx_vcpu_load
- loaded_vmcs_clear
- smp_call_function_single
pmdp_clear_flush
- flush_tlb_mm_range
- native_flush_tlb_others
-
Ideas going forward:
1) Instrument kernel for debugging csd_lock
2) Determine which CPUs exhibit this issue
3) Examine pinning more in depth pin 0-0 1-2 for example
4) Test older kernels , newer kernels to verify issue
--
You received this bug notification because you are a member of Kernel
Hm, following your instructions I rather run into a situation where the
l2 guest gets paused. Likely because l1 runs out of disk space. The
default of uvtool is 7G which I would say the l2 stress run fills as it
grows the l2 qcow image on l1 which has to stuff all the initial cloud-
image and the
Also FYI: I was not able to reproduce this issue when using Vivid as
the bare metal L0.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1413540
Title:
Trusty soft lockup issues with
This does not appear to be specific to OpenStack, nor tempest. I've
reproduced with Trusty on Trusty on Trusty, vanilla qemu/kvm.
Simplified reproducer, with an existing MAAS cluster:
@L0 baremetal:
- Create a Trusty bare metal host from daily images.
- sudo apt-get update -y sudo apt-get -y
** Summary changed:
- soft lockup issues with nested KVM VMs running tempest
+ Trusty soft lockup issues with nested KVM
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1413540
Title:
s/static/sym/ ;-)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1413540
Title:
Trusty soft lockup issues with nested KVM
Status in linux package in Ubuntu:
Confirmed
Bug
A few hrs later, those two L0 bare metal host CPUs are still maxed. In
scenarios where L0 is hosting many VMs, such as in a cloud, this bug can
be expected to cause significant performance, consistency and capacity
issues on the host and in the cloud as a whole.
--
You received this bug
** Attachment added: L0-baremetal-cpu-pegged.png
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1413540/+attachment/4353983/+files/L0-baremetal-cpu-pegged.png
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
** Attachment added: L1-console-log-soft-lockup.png
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1413540/+attachment/4353984/+files/L1-console-log-soft-lockup.png
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in
I've collected crash dumps, and have stored them on an internal
Canonical server as they are 2gb+. Feel free to ping me for access.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1413540
36 matches
Mail list logo
