For posterity: If
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/comments/143
is the cause of this issue for you, dmesg -D (which turns off console
logging of kernel messages) might be a viable workaround until you can
reboot.
--
You received this bug notification because you are a
This bug was fixed in the package linux - 3.13.0-83.127
---
linux (3.13.0-83.127) trusty; urgency=low
[ Brad Figg ]
* Release Tracking Bug
- LP: #1555839
[ Florian Westphal ]
* SAUCE: [nf,v2] netfilter: x_tables: don't rely on well-behaving
userspace
- LP: #1555
This bug was fixed in the package linux - 3.19.0-56.62
---
linux (3.19.0-56.62) vivid; urgency=low
[ Brad Figg ]
* Release Tracking Bug
- LP: #1555832
[ Florian Westphal ]
* SAUCE: [nf,v2] netfilter: x_tables: don't rely on well-behaving
userspace
- LP: #1555338
This bug was fixed in the package linux - 4.2.0-34.39
---
linux (4.2.0-34.39) wily; urgency=low
[ Brad Figg ]
* Release Tracking Bug
- LP: #1555821
[ Florian Westphal ]
* SAUCE: [nf] netfilter: x_tables: check for size overflow
- LP: #1555353
* SAUCE: [nf,v2] netf
Thanks !
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564
Title:
Soft lockup with "block nbdX: Attempted send on closed socket" spam
Status in linux package in Ubuntu:
Fix Relea
verification can be done with this script:
#!/bin/bash
modprobe nbd
qemu-nbd -d /dev/nbd0
truncate /tmp/testfile -s 20G
qemu-nbd -c /dev/nbd0 /tmp/testfile
for n in $( seq 1 250 ) ; do
echo $n
( dd if=/dev/zero of=/dev/nbd0 bs=1 & )
done
qemu-nbd -d /dev/nbd0
after running that, on an u
** Tags removed: verification-needed-trusty
** Tags added: verification-done-trusty
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564
Title:
Soft lockup with "block nbdX: Attempted
** Branch linked: lp:ubuntu/trusty-proposed/linux-lts-vivid
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564
Title:
Soft lockup with "block nbdX: Attempted send on closed socket" s
This bug is awaiting verification that the kernel in -proposed solves
the problem. Please test the kernel and update this bug with the
results. If the problem is solved, change the tag 'verification-needed-
wily' to 'verification-done-wily'.
If verification is not done by 5 working days from today
This bug is awaiting verification that the kernel in -proposed solves
the problem. Please test the kernel and update this bug with the
results. If the problem is solved, change the tag 'verification-needed-
vivid' to 'verification-done-vivid'.
If verification is not done by 5 working days from tod
This bug is awaiting verification that the kernel in -proposed solves
the problem. Please test the kernel and update this bug with the
results. If the problem is solved, change the tag 'verification-needed-
trusty' to 'verification-done-trusty'.
If verification is not done by 5 working days from t
** Also affects: linux (Ubuntu Vivid)
Importance: Undecided
Status: New
** Also affects: linux (Ubuntu Wily)
Importance: Undecided
Status: New
** Changed in: linux (Ubuntu Vivid)
Status: New => Fix Committed
** Changed in: linux (Ubuntu Wily)
Status: New => Fix
** Also affects: linux (Ubuntu Trusty)
Importance: Undecided
Status: New
** Changed in: linux (Ubuntu Trusty)
Status: New => Fix Committed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.laun
This bug was fixed in the package linux - 4.4.0-6.21
---
linux (4.4.0-6.21) xenial; urgency=low
[ Tim Gardner ]
* Release Tracking Bug
- LP: #1546283
* Naples/Zen, NTB Driver (LP: #1542071)
- [Config] CONFIG_NTB_AMD=m
- NTB: Add support for AMD PCI-Express Non-Tr
opened bug 1534216 to track the serial port issue.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564
Title:
Soft lockup with "block nbdX: Attempted send on closed socket" spam
Stat
** Tags added: canonical-bootstack
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564
Title:
Soft lockup with "block nbdX: Attempted send on closed socket" spam
Status in linux pack
Great. I'll send the patch upstream, and open a new bug for the serial
port hanging issue. Thanks!
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564
Title:
Soft lockup with "block
I resumed the tests on Monday, and so far we're looking good. Your
change prevented ~10 locks so far, it would seem.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564
Title:
Soft lo
I applied the patch, and it saved a reboot twice already, I think. dmesg
from one server : http://pastebin.ubuntu.com/14438525/
I have to stop the tests for the weekend though, I'll resume on Monday.
--
You received this bug notification because you are a member of Kernel
Packages, which is subs
Ok, nm about the sosreport - I got the info from some older emails from
axino, nova is using qemu-nbd to locally mount images and access the
partitions inside them. I was able to trivially reproduce this simply
by creating an image, attaching it with qemu-nbd to /dev/nbd0,
partitioning it and mkfs
axino or nick, can either of you attach an sosreport from an affected
system? The crashdump doesn't include any userspace data so I can't see
what exactly the qemu-nbd userspace program is doing, nor can i see what
params it's started with. I'll need that info to be able to debug the
qemu-nbd sid
This memory leak we have so far only seen on arm64, to be clear.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564
Title:
Soft lockup with "block nbdX: Attempted send on closed sock
This problem has caused more serious damage recently. When nbd dies and
printk()s like mad, the serial console is not fast enough to display it.
The kernel keeps allocating buffer space for serial output, which we see
as 13G kmalloc-256 or kmalloc-512 kernel threads.
Eventually the OOMkiller tri
Well, yes I agree, it does look like the serial port causing the
softlockup is probably separate - but caused by - the nbd closed socket
errors. However, the serial port output definitely shouldn't be causing
a softlockup - no matter how much data it has to send, the serial port
driver in the kern
Re-reading comment #318 Dan, I realize that we may be investigating a
symptom and not the root cause.
Whenever the soft-lockup happens, the serial console does get flooded
"block nbdX: Attempted send on closed socket". If the serial console
getting flooded causes soft lockups, then it is indeed a
> It turns out, the kernel booted through kexec fails booting probably because
> of the notsc option :
> https://pastebin.canonical.com/146714/
hmm, that's weird, but if notsc is all that changed i assume it is the
problem.
> I'm a bit worried about the following line :
> [ 0.00] tsc: Kerne
Hi Dan,
Thanks for your investigation. Sorry for the delay, but finally I
managed to reboot the compute nodes with the "notsc" kernel parameter. I
also disabled the qemu-nbd workaround.
Once that was done, it didn't take long for a node to crash, which would
indicate that notsc didn't fix the pro
Ok, here's my analysis of the latest dump.
There are 3 kernel migrate threads waiting; this is the cause of the
softlockup - specifically pid 101 on cpu 13 is where the softlockup (and
then panic, due to panic on softlockup enabled) happens, and the other 2
migrate threads (pid 79 and 151) are als
I've dl'ed the dump and I'm reviewing it.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564
Title:
Soft lockup with "block nbdX: Attempted send on closed socket" spam
Status in lin
Junien,
Sorry for the delay. After sometime dealing with some other priorities,
I'm coming back to this. I'm downloading the dump and will take a look.
Lets see what this bug is related with.
Tks for providing it. Will report something back soon.
--
You received this bug notification because yo
Unfortunately, one server managed to crashdump, even with your patched
kernel. apport is above, crashdump is at https://private-
fileshare.canonical.com/~axino/201511241217.crash.lp1505564.matar.tar.xz
- sha1sum 056fae2554e52989a24094945b297c0c5906be7c
I've diverted qemu-nbd again.
Please let me
apport information
** Description changed:
Some of our nova compute hosts regularly freeze, sometimes for a few
hours, with kern.log getting spammed with:
block nbdX: Attempted send on closed socket
and a few "CPU soft lockup" messages (see attached log). This clears up
when the q
Note to self:
The commit being backported to 3.19 and 3.13 has to contain this race
fix:
commit dcc909d90ccdbb73226397ff6d298f7af35b0e11
Author: Markus Pargmann
Date: Tue Oct 6 20:03:54 2015 +0200
nbd: Add locking for tasks
The timeout handling introduced in
7e2893a16d3e (nbd
** Patch added: "trusty-nbd-restructure-debugging-prints.patch"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+attachment/4524366/+files/trusty-nbd-restructure-debugging-prints.patch
--
You received this bug notification because you are a member of Kernel
Packages, which is sub
** Patch added: "trusty-nbd-fix-timeout-detection.patch"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+attachment/4524367/+files/trusty-nbd-fix-timeout-detection.patch
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to lin
Junien,
That is good feedback. I also received another request to backport this
to 3.13 SO I'll be providing the hotfixed kernel in the same PPA soon
(tomorrow morning most likely).
Attaching the 3.13 patches (just for reference since the SRU process
requires me to send all those patches to kerne
Hi Rafael,
I applied the patch earlier today.
No crash so far, which was nearly impossible before !
This looks very promising, I'll keep you posted tomorrow.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.l
Okay,
PPA is ready:
https://launchpad.net/~inaddy/+archive/ubuntu/lp1505564/+packages
Please upgrade kernel to:
linux-lts-vivid - 3.19.0-36.41~14.04.1hf00090138v20151122b1
By doing:
$ sudo add-apt-repository ppa:inaddy/lp1505564
$ sudo apt-get update
$ sudo apt-get install linux-image-3.19.0-
Hello Junien,
Based on my previous feedbacks, I've created the following PPA:
https://launchpad.net/~inaddy/+archive/ubuntu/lp1505564
With a Trusty HWE kernel (vivid) + 2 patches:
nbd: Restructure debugging prints
nbd: Fix timeout detection
For you to use and provide me feedback.
I've done mi
Testing patches I have attached above:
inaddy@sf00090138trusty(~)$ sudo qemu-img create -f qcow2 -o
preallocation=metadata ./test.qcow2 1G
Formatting './test.qcow2', fmt=qcow2 size=1073741824 encryption=off
cluster_size=65536 preallocation='metadata' lazy_refcounts=off
inaddy@sf00090138trusty(~
** Tags added: patch
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564
Title:
Soft lockup with "block nbdX: Attempted send on closed socket" spam
Status in linux package in Ubuntu:
** Patch added: "vivid-nbd-fix-timeout-detection.patch"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+attachment/4523782/+files/vivid-nbd-fix-timeout-detection.patch
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux
** Patch added: "vivid-nbd-restructure-debugging-prints.patch"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+attachment/4523781/+files/vivid-nbd-restructure-debugging-prints.patch
--
You received this bug notification because you are a member of Kernel
Packages, which is subsc
** Description changed:
- Hi,
+ Some of our nova compute hosts regularly freeze, sometimes for a few
+ hours, with kern.log getting spammed with:
- Some of our nova compute hosts regularly freeze, sometimes for a few hours,
with kern.log getting spammed with :
block nbdX: Attempted send on c
Junien,
I faced minor issues on backport yesterday and today is holiday in
Brazil. I'll get back to this soon. Nevertheless, it is good feedback
that this "qemu-nbd" workaround is probably making the system more
stable.
I'll get back to you soon.
Thank you
Rafael
--
You received this bug not
Hi Rafael,
WIth qemu-nbd symlinked to /bin/true, no crash so far...
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564
Title:
Soft lockup with "block nbdX: Attempted send on closed
Thanks for your update Rafael. Since nova-compute doesn't do anything
useful with qemu-nbd anyway, I'm going to try to "soft-disable" it
(divert + symlink to /bin/true), and we'll see if we can repro the
crashes. I'll keep you posted.
I'll also try your patched kernel as soon as it's ready, of cou
Hello Junien,
After your last crash - similar to previous ones - one thing called my
attention: For the first time we had one CPU RCU stall detected by
another CPU. This made me think that it wasn't only related to the SMP
logic - like I believed - but the stall occurred also somewhere else.
Thank Junien, I'm downloading the crash dump (10GB) and will update you
as soon as I open it.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564
Title:
Soft lockup with "block nbdX:
apport above, crash dump is at
https://chinstrap.canonical.com/~axino/201511171222.crash.lp1505564.druk.tar.xz
- sha1sum 93ae006186b6bc7298afd37d3f759effe08d7ba3
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs
** Tags added: kernel-key
** Changed in: linux (Ubuntu)
Importance: Undecided => High
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564
Title:
Soft lockup with "block nbdX: Atte
apport information
** Description changed:
Hi,
Some of our nova compute hosts regularly freeze, sometimes for a few hours,
with kern.log getting spammed with :
block nbdX: Attempted send on closed socket
and a few "CPU soft lockup" messages (see attached log). This clears up when
Hi Rafael,
For starters, the server Chris mentioned above didn't panic because the
kernel.softlockup_panic wasn't set to 1 on reboot. This is now fixed.
Then, we're still running 3.19 (all the nodes got rebooted to
3.19.0-33-generic). Let me know if you wish us to get back to 3.13.
I verified th
Hello Chris,
Could you clarify the following statement:
"""
So, step 2 was to add "nox2apic intremap=off" to the DL385-G7s. I added it to
only one of them initially. That machine lasted 9 days before we had another
kernel panic ("NMI watchdog: BUG: soft lockup - CPU#27 stuck for 23s!
[migrat
Hi Rafael,
I've been continuing Junien's investigations into this problem. The
machines have had all the BIOS and firmware updates I could find on HP's
website (although in the case of a DL385-G7 the latest appears to be
February 2014!) One of them only lasted a day before crashing again.
So, s
Hello Junien,
(recommendations with *)
I'm replying to you and to the LP bug so it gets proper documentation.
Under comment #91:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/comments/91
You can see my kernel dump analysis, where I am showing you that the
OS is stuck in a "migration
** Changed in: linux (Ubuntu)
Status: Confirmed => In Progress
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564
Title:
Soft lockup with "block nbdX: Attempted send on closed
I'm attaching the crash tool output from the 3.13 kernel dump.
Much likely related to the situation already found in the following case:
-> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1413540
Handled by Chris Arges and I on LKML discussions with Ingo and Linus:
-> http://www.kernelhub.o
** Attachment added: "lp1505564-3.13-kdump-crash-output.txt"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+attachment/4509474/+files/lp1505564-3.13-kdump-crash-output.txt
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to
I'm attaching the crash tool output from the 3.13 kernel dump.
Much likely related to the situation already found in the following case:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1413540
Handled by Chris Arges and I on LKML discussions with Ingo and Linus:
-> http://www.kernelhub.org/?
Junien, I'm on it right now.. will update here asap.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564
Title:
Soft lockup with "block nbdX: Attempted send on closed socket" spam
St
Yet another crash, on another node this time (still a 100% Nova compute
node). apport information is above, crashdump is at
https://chinstrap.canonical.com/~axino/201510301227.crash.lp1505564.phianna.tar.xz
- sha1sum 71353f8c70d009369a61de811c90d6199b341543
Thanks !
--
You received this bug noti
apport information
** Description changed:
Hi,
Some of our nova compute hosts regularly freeze, sometimes for a few hours,
with kern.log getting spammed with :
block nbdX: Attempted send on closed socket
and a few "CPU soft lockup" messages (see attached log). This clears up when
crashdump available at
https://chinstrap.canonical.com/~axino/201510292103.crash.lp1505564.orlo.tar.xz
- sha1sum is 366c0460cceed5938f2a19fc4b925380a33c18a6
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.laun
Upgraded all the kernels to lts-vivid (3.19.0-31-generic), and got a new
crashdump overnight, from the same server just above (orlo). apport +
dump below.
** Description changed:
Hi,
Some of our nova compute hosts regularly freeze, sometimes for a few hours,
with kern.log getting spammed
sha1 sums for all 3 dumps below :
6b63d74566b6df0671ba9e79dca724ddc6d8d6df 201510281259.crash.lp1505564.tar.xz
<= may have been taken after the lockup occured
3a8cbdd9e51af4f6eaba4ff0aacc6f956c706961
201510281618.crash.lp1505564.druk.tar.xz
1ebd57dea13cf655e7ef442951da2aedc33d0046
20151028195
dump at
https://chinstrap.canonical.com/~axino/201510281951.crash.lp1505564.orlo.tar.xz
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564
Title:
Soft lockup with "block nbdX: Attemp
Second apport+dump below
** Description changed:
Hi,
Some of our nova compute hosts regularly freeze, sometimes for a few hours,
with kern.log getting spammed with :
block nbdX: Attempted send on closed socket
and a few "CPU soft lockup" messages (see attached log). This clears up
dump is at
https://chinstrap.canonical.com/~axino/201510281618.crash.lp1505564.druk.tar.xz
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564
Title:
Soft lockup with "block nbdX: Att
I'm just now realizing that the crashdump above may have been taken too
late (when the kernel wasn't locked up anymre), because I could ssh to
the server when I took it.
I was seeing the "block nbdX: Attempted send on closed socket" kernel
log spam on the serial when I sent the NMI, but _perhaps_
First dump + apport (post reboot) below
** Description changed:
Hi,
Some of our nova compute hosts regularly freeze, sometimes for a few hours,
with kern.log getting spammed with :
block nbdX: Attempted send on closed socket
and a few "CPU soft lockup" messages (see attached log).
This issue just hit us again, this time I sent an NMI to the server to
get a dump. It's available at
https://chinstrap.canonical.com/~axino/201510281259.crash.lp1505564.tar.xz
apport information post-reboot is available above.
We've been trying to see if the issue appeared somewhere in the 3.13
s
apport information
** Description changed:
Hi,
Some of our nova compute hosts regularly freeze, sometimes for a few hours,
with kern.log getting spammed with :
block nbdX: Attempted send on closed socket
and a few "CPU soft lockup" messages (see attached log). This clears up when
** Changed in: linux (Ubuntu)
Assignee: (unassigned) => Rafael David Tinoco (inaddy)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564
Title:
Soft lockup with "block nbdX: Atte
I think that this may be a duplicate of #1500739, the symptoms certainly
look the same.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1505564
Title:
Soft lockup with "block nbdX: Attemp
Second host now
** Tags added: staging
** Description changed:
Hi,
Some of our nova compute hosts regularly freeze, sometimes for a few hours,
with kern.log getting spammed with :
block nbdX: Attempted send on closed socket
and a few "CPU soft lockup" messages (see attached log).
apport information
** Tags added: apport-collected trusty uec-images
** Description changed:
Hi,
Some of our nova compute hosts regularly freeze, sometimes for a few hours,
with kern.log getting spammed with :
block nbdX: Attempted send on closed socket
and a few "CPU soft lockup"
77 matches
Mail list logo
