[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Po-Hsu Lin]

15.10 EOL

** Changed in: linux (Ubuntu)
   Status: Triaged => Won't Fix

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Stefan Bader]

Any news here? Also, it will help me to figure out what exact code level
any crashing kernel was if you could post the "uname -a" output after
booting into it on a likely affected machine.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Stefan Bader]

Hm one note, and sorry this probably not very trusting... have you
ensured (uname -a) that you actually booted into the right older kernel.
Its just easy to go wrong as normally only the kernel with the highest
version number is used. One have to fiddle manually with
/etc/default/grub (for HVM guests). Or work with grubenv, though I
usually prefer just to replace GRUB_DEFAULT=0 with
GRUB_DEFAULT="" and run update-grub. The string I copy from the
menuentry I want to use in /boot/grub/grub.cfg...

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Stefan Bader]

Would be great if you were able to get it working again. At least to see
whether the crash happens in the same area (timers). At the moment this
sounds like something in user-space changed in a way that allows it to
mess badly with the kernel. That sounds bad. And if its really somethign
getting done with the timers that leaves a lot to look at.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Will Buckner]

I have now downgraded all of these systems to 3.19.8-031908-generic
(Mainline). We'll know in a couple of days if this fixed it (or as soon
as a few hours if it didn't fix it, possibly). I'll update when I know
anything else; good so far! 4.0.9 definitely didn't fix it completely,
but MAY have made it less frequent.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Will Buckner]

3.19 crashed as well, just now. This is surprising. I don't have a
trace, as the kernel downgrades messed up netconsole somehow, but I'll
try to get it working again.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Will Buckner]

And I should also point out that ifquery is crashing on boot most of the
time on these systems:

xxx.log:[3.909680] ifquery[378]: segfault at 1 ip 00403187 sp 
7fff7078d8c0 error 4 in ifup[40+d000]
xxx.log:[3.008003] ifquery[380]: segfault at 1 ip 00403187 sp 
7ffdf6935bc0 error 4 in ifup[40+d000]
xxx.log:[2.647084] ifquery[370]: segfault at 1 ip 00403187 sp 
7ffda45292f0 error 4 in ifup[40+d000]
xxx.log:[2.868947] ifquery[370]: segfault at 1 ip 00403187 sp 
7ffcd9140230 error 4 in ifup[40+d000]
xxx.log:[2.846450] ifquery[398]: segfault at 1 ip 00403187 sp 
7ffc5da3d670 error 4 in ifup[40+d000]
xxx.log:[3.070464] ifquery[372]: segfault at 1 ip 00403187 sp 
7fff78691b90 error 4 in ifup[40+d000]

But I've been unable to get a coredump and it doesn't save a .crash and
I can't reproduce it once the system is online. And this "checkport"
utility couldn't possibly be running so early in boot--it's triggered by
a web request and represents a relatively low percentage of traffic.
It's unlikely that it would be triggered at all in the first 30 seconds
after boot.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Stefan Bader]

The ifquery segfaults should be safe to ignore. I see those a lot but those 
don't seem to have any impact (yeah, probably should be fixed at some point ... 
if there were not always more pressing matters). Your in-house utility, hard to 
say without knowing more about what causes the segfault there. If the kernel 
interfaces were all well written it should not be possible but of course they 
are not.
If you really get crashes (in the sense guest goes away completely like before) 
with 4.0, could you post me a console stacktrace. That should be different to 
some degree as the timer code changes were not done before 4.2.
Kexec crashdump only works on HVM guests if you were not using PV drivers for 
network and disk. Which could make the problem go away (as well as any good 
performance). To make it work with PV drivers needs a very recent kernel (I 
think 4.4) as well as some support on the host side (which there is no control 
over on AWS).

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Will Buckner]

Hey Stefan,

Just wanted to update you that I've installed the Mainline 4.0.9 kernel
(4.0.9-040009-generic #201507212131) on all affected machines, finishing
about an hour ago, from https://wiki.ubuntu.com/Kernel/MainlineBuilds.
I've still had two crashes in the hour since the downgrade.

I don't know enough about kernel code to provide feedback on the detail
above, but thanks for looking into this!

I doubt that this could be related, but maybe it will help... We also
appear to have one of our C utilities (developed in-house) crashing a
few times a day:

xxx.log:[ 4740.992084] traps: checkport[6624] general protection ip:4c3d07 
sp:7f8441ce8e20 error:0 in checkport[40+ff000]
xxx.log:[60087.824087] traps: checkport[2203] general protection ip:4c3d07 
sp:7fcc7df61e20 error:0 in checkport[40+ff000]
xxx.log:[ 5375.784101] traps: checkport[10795] general protection ip:4c3d07 
sp:7f96ebcbde20 error:0 in checkport[40+ff000]

in addition to numerous segfaults:
xxx.log:[ 1354.456792] checkport[1644]: segfault at a8 ip 0049be3a sp 
7fea90080da0 error 4 in checkport[40+ff000]
xxx.log:[ 1393.894368] checkport[2754]: segfault at a8 ip 0049be3a sp 
7f082e3c6da0 error 4 in checkport[40+ff000]
xxx.log:[ 2474.260293] checkport[24996]: segfault at a8 ip 0049be3a sp 
7f569f6dcda0 error 4 in checkport[40+ff000]
xxx.log:[ 4170.613174] checkport[23278]: segfault at a8 ip 0049be3a sp 
7fcf9b580da0 error 4 in checkport[40+ff000]

The developers are looking into this on our end, but, we still had the
same version of the tool and the same number of segfaults on our 3.19
kernel on 15.04 and it never triggered a kernel crash. Is it possible
that invalid memory access in this binary could be triggering a kernel
GPF? Or would that memory be protected?

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Stefan Bader]

Oh hm, another crazy thought...maybe it would also be worth trying a
3.19 kernel... If the guests then crash as well then maybe something in
the area of user-space (as in the whole release) is causing issues.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Stefan Bader]

More braindump: One thing that seems sure is that the crash occurs in 
__run_timers after optionally cascading pending timers into higher level lists. 
This picks a list of timers from the root vector array (tv1) into a temporary 
location and then removes timers from that (and calls actions) until the 
temporary list is empty. Looking at the registers from the trace in comment #3 
and comparing with the disassembly I would say r15 contains the pointer to the 
currently processed timer (with the list pointers right at the beginning). rax 
is the next pointer and rdx the pointer to the previous elements address. rdx 
seems to be an address on the stack, so I would guess this is the temporary 
lists head. Which makes sense given that the loop always takes the first 
element from the work list and then deletes it from the list.
What should not happen is that this first element is already marked as deleted 
but still taken from the list. Because the poison value for the next pointer is 
only set after the element was unlinked from the list. One detail to note is 
that rdx still points to the previous elements address. If I looked correctly 
that would happen only from migrate_timers (which only happens if a cpu goes 
away, which is unlikely) or via __mod_timer() and that properly locks the base 
and would have taken the element out of list.
Oh wait... just realize that even if the bad list element had pprev set to 
NULL, that would be "fixed up" by hlist_move_list(). So any thoughts trying to 
narrow down potential sources of modification are invalid. Only that it had to 
be somehow through detach_timer() at some point is true... *sigh*

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Stefan Bader]

Note that the following is not final statement but sharing some thoughts
to whoever else is looking at this report (and for me to remember). So
while I did find nothing that really looked odd in the xen-netfront code
I saw there was some change to the generic timer code:

commit 1dabbcec2c0a36fe43509d06499b9e512e70a028
  timer: Use hlist for the timer wheel hash buckets

That change was part of 4.2 but if it would be the cause I would expect
problems not only on AWS instances. But then it might just be that bare-
metal servers with a similarly high traffic tend to be upgraded much
less often anyway... Part of the change above seems to be some
exchange of special meaning of list pointer values. Not sure I grasp the
implications, yet. While using double linked lists before, the pointer
to the next element seemed to serve as pending indicator and the pointer
to the previous element was invalidated with a LIST_POISON2 value. Now
its the other way round. Referring to the detach_timer function which is
called from __run_timers via detached_expired_timer.

The crash happens at offset 0x116 in run_timer_softirq (thats 278
decimal). The disassembly of that function around there is:

   0x810e5c1e <+254>:   mov%r15,0x8(%rbx)
   0x810e5c22 <+258>:   nopl   0x0(%rax,%rax,1)
   // Guest this is __hlist_del(struct hlist_node *n)
   // rax = n->next
   0x810e5c27 <+263>:   mov(%r15),%rax
   // rdx = n->ppev
   0x810e5c2a <+266>:   mov0x8(%r15),%rdx
   0x810e5c2e <+270>:   test   %rax,%rax
   // *(n->pprev) = n->next
   0x810e5c31 <+273>:   mov%rax,(%rdx)
   // if (n->next == NULL) jump
   0x810e5c34 <+276>:   je 0x810e5c3a 

   // (n->next)->pprev = n->pprev (but n->next is LIST_POISON2 / invalid ptr)
   0x810e5c36 <+278>:   mov%rdx,0x8(%rax)
   0x810e5c3a <+282>:   testb  $0x10,0x2a(%r15)
   // here we seem back at detach_timer inlined and clear_pending assumed true
   // entry->next = LIST_POISON2 and entry->pprev = NULL
   0x810e5c3f <+287>:   movabs $0xdead00200200,%rax
   0x810e5c49 <+297>:   movq   $0x0,0x8(%r15)
   0x810e5c51 <+305>:   mov%rax,(%r15)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Stefan Bader]

Having written that down, it feels quite dangerous to use LIST_POISON in
combination with __hlist_del as the latter only protects against the
next pointer being NULL (indicating the last list entry). That
definitively breaks when trying to detach the same timer twice. Need to
think about whether this would have been ok before (with the double
linked lists)...

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Stefan Bader]

Looking at both traces this looks to be consistently happen inside 
run_timer_softirq() and from the offset I would guess we are in the inlined 
__run_timers. Another noteworthy part is the value of RAX. This is the value of 
LIST_POISON2 which is used to mark an invalid pointer of a (hlist_node 
*)->pprev.
So I would guess something modified the list of pending timers (those exist 
per-cpu) while softirq processing was working on them. The problem is to say 
what. Not sure a dump will help as often in those races the clues go away just 
after causing problems.
I would maybe suspect the area of xen-netfront, given that, as far as I can 
tell, this has not happened on bare-metal servers and from the description 
rather seems to affect high traffic instances.
Would it be possible to volunteer one affected instance and try mainline 
kernels (https://wiki.ubuntu.com/Kernel/MainlineBuilds) between 3.19 and 4.2 
(4.0, 4.1) and/or after (4.3, maybe 4.4)? That would give a smaller delta to 
look at for what broke things (using the 4.0 and 4.1 kernels) or whether maybe 
it got fixed but not identified as a stable patch (when using 4.3 or 4.4).

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Will Buckner]

Thanks for looking into this Stefan! We were completely fine with 15.05
and 3.19. If it won't break anything terribly, I can try to put 3.19,
4.0, and 4.1 on these machines, but each one crashes every 24-48 hours,
so it might take me several days. Which kernel would you recommend
starting with, say, 4.0 or 4.4?

Another thing that I didn't find relevant before, but seems to confirm
what you're saying about the per-CPU timers--AWS told me the following
after a crash where I disabled my auto-reboot-on-alarm triggers:

I was able to successfully get a trace - most of the vCPU were just in a
halted state, so nothing there, but one had some potentially useful
information:


VCPU 1
rip: 810c3ef5 __pv_queued_spin_lock_slowpath+0xc5
flags: 0206 i nz p
rsp: 8803ff243e78
rax: 0a2a   rcx: fffa   rdx: 0003
rbx: 8803f7ef2e38   rsi: 8803ff243df8   rdi: 0003
rbp: 8803ff243ea8r8: r9: 8803fe80
r10:    r11: 813ef2b0   r12: 8803ff2571c0
r13: 0008   r14: 88040ffa30c0   r15: 0001
cs: 0010 ss: ds: es: 
fs:  @ 7fc1867b8700
gs:  @ 8803ff24/

cr0: 80050033
cr2: 00a8
cr3: de15f000
cr4: 001406e0

dr0: 
dr1: 
dr2: 
dr3: 
dr6: 0ff0
dr7: 0400
Code (instr addr 810c3ef5)
41 bf 01 00 00 00 48 0f af c3 48 89 45 d0 b8 00 80 00 00 eb 0b  90 83 e8 01 
0f 84 d4 00 00 00

Stack:
8c2fa8473f0f2e38 8803ff2577c0 8803f7ef2e10 
8803f7ef2e10 000101155691 8803ff243eb8 817f0021
8803ff243f38 816e48f4 000101155693 0040012c
0024 8803ff243ee0 8803ff243ee0 8803ff243ef0

Call Trace:
  [] __pv_queued_spin_lock_slowpath+0xc5  <--
  [] _raw_spin_lock+0x21
  [] net_rx_action+0xe4
  [] __do_softirq+0xf6
  [] do_softirq_own_stack+0x1c


-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Stefan Bader]

Hi Will and thanks for volunteering. That backtrace at least looks to confirm 
the educated guess about being related to network. *If* it actually is related, 
then I could imagine that processing incoming network traffic in softirq 
context of cpu#1 might cancel a timer which was set to wait for a packet to 
arrive and maybe that at or around the same time expired. Still hard to say for 
sure.
Looking at the xen-netfont driver between 3.19 and 4.2 there was not any change 
that stuck out as suspicious but then it might as well be a change in the 
network stack which the xen driver would need to adapt for.
About the mainline kernels. While I never would *promise* nothing horrible to 
happen, I would not expect anything so bad that there was not a chance to 
switch back to a different kernel. The mainline kernels miss some of the few 
special drivers (mostly overlayfs) which I don't think you have in use. As you 
say 3.19 is ok, I would suggest to go for 4.0.9 first and 4.1.15 after that. 
Both contain the latest upstream stable patches.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Robert C Jennings]

Leann, can the kernel team take a look at this bug?

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Alberto Salvia Novella]

** Changed in: linux (Ubuntu)
 Assignee: (unassigned) => Alberto Salvia Novella (es20490446e)

** Changed in: linux (Ubuntu)
   Status: Confirmed => In Progress

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Alberto Salvia Novella]

** Changed in: linux (Ubuntu)
   Status: In Progress => Triaged

** Changed in: linux (Ubuntu)
 Assignee: Alberto Salvia Novella (es20490446e) => (unassigned)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Na3iL]

** Changed in: linux (Ubuntu)
 Assignee: (unassigned) => Na3iL (naeilzoueidi)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Will Buckner]

Do you guys need a vmcore? I'm working on getting one from AWS.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Alberto Salvia Novella]

** Changed in: linux (Ubuntu)
   Importance: Undecided => Critical

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking

[Will Buckner]

And now we've got a second trace from the same machine with a bit more
info:

And one more:

[14032.676085] general protection fault:  [#1] SMP
[14032.678409] Modules linked in: isofs xt_CHECKSUM iptable_mangle 
ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc 
iptable_filter ip_tables x_tables ppdev xen_fbfront fb_sys_fops intel_rapl 
iosf_mbi input_leds i2c_piix4 parport_pc 8250_fintek serio_raw parport mac_hid 
netconsole configfs autofs4 cirrus syscopyarea sysfillrect sysimgblt ttm 
crct10dif_pclmul crc32_pclmul aesni_intel drm_kms_helper aes_x86_64 lrw 
gf128mul glue_helper ablk_helper drm cryptd ixgbevf psmouse pata_acpi floppy
[14032.680061] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.2.0-23-generic 
#28-Ubuntu
[14032.680061] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/07/2015
[14032.680061] task: 81c13500 ti: 81c0 task.ti: 
81c0
[14032.680061] RIP: 0010:[]  [] 
run_timer_softirq+0x116/0x2d0
[14032.680061] RSP: 0018:8803ff203e98  EFLAGS: 00010086
[14032.680061] RAX: dead00200200 RBX: 8803ff20e9c0 RCX: 000e1785
[14032.680061] RDX: 8803ff203ec8 RSI: 8803ff21be00 RDI: 8803ff20e9c0
[14032.680061] RBP: 8803ff203f08 R08: 0001be00 R09: 
[14032.680061] R10: ea00036dc680 R11:  R12: 00d2
[14032.680061] R13: 8172aaf0 R14:  R15: 8800db71a3d0
[14032.680061] FS:  () GS:8803ff20() 
knlGS:
[14032.680061] CS:  0010 DS:  ES:  CR0: 80050033
[14032.680061] CR2: 7f97cd08ec64 CR3: 0003f5834000 CR4: 001406f0
[14032.680061] Stack:
[14032.680061]  8803ff203eb8 8803ff20f5f8 8803ff20f3f8 
8803ff20f1f8
[14032.680061]  8803ff20e9f8 8800db71a348 dead00200200 
08c9e9276e6f82e9
[14032.680061]  00011fc0 0001 81c0b0c8 
0001
[14032.680061] Call Trace:
[14032.680061]  
[14032.680061]  [] __do_softirq+0xf6/0x250
[14032.680061]  [] irq_exit+0xa3/0xb0
[14032.680061]  [] xen_evtchn_do_upcall+0x39/0x50
[14032.680061]  [] xen_hvm_callback_vector+0x6b/0x70
[14032.680061]  
[14032.680061]  [] ? get_next_timer_interrupt+0xf2/0x240
[14032.680061]  [] ? native_safe_halt+0x6/0x10
[14032.680061]  [] default_idle+0x1e/0xa0
[14032.680061]  [] arch_cpu_idle+0xf/0x20
[14032.680061]  [] default_idle_call+0x2a/0x40
[14032.680061]  [] cpu_startup_entry+0x2c9/0x320
[14032.680061]  [] rest_init+0x7c/0x80
[14032.680061]  [] start_kernel+0x48b/0x4ac
[14032.680061]  [] ? early_idt_handler_array+0x120/0x120
[14032.680061]  [] x86_64_start_reservations+0x2a/0x2c
[14032.680061]  [] x86_64_start_kernel+0x14a/0x16d
[14032.680061] Code: 81 e6 00 00 20 00 48 85 d2 48 89 45 b8 0f 85 30 01 00 00 
4c 89 7b 08 0f 1f 44 00 00 49 8b 07 49 8b 57 08 48 85 c0 48 89 02 74 04 <48> 89 
50 08 41 f6 47 2a 10 48 b8 00 02 20 00 00 00 ad de 49 c7
[14032.680061] RIP  [] run_timer_softirq+0x116/0x2d0
[14032.680061]  RSP 

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1534345

Title:
  Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced
  Networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1534345/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs