[Kernel-packages] [Bug 1651199] Re: 16.04 - Kernel panic on docker server
After upgrading to mainline builds everywhere, I haven't seen a restart in a least a day. Seems this is fixed in the mainline kernel. ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed ** Changed in: linux (Ubuntu Xenial) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1651199 Title: 16.04 - Kernel panic on docker server Status in Linux: New Status in linux package in Ubuntu: Confirmed Status in linux source package in Xenial: Confirmed Bug description: [31016.057405] BUG: unable to handle kernel paging request at 82c4801e6bda [31016.061249] IP: [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] PGD 0 [31016.061380] Oops: 0010 [#1] SMP [31016.061380] Modules linked in: binfmt_misc xt_REDIRECT nf_nat_redirect veth xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo xt_addrtype iptable_filter xt_conntrack br_netfilter bridge stp llc overlay xt_nat xt_tcpudp iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables isofs ppdev serio_raw parport_pc parport ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ixgbevf psmouse floppy [31016.061380] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.4.0-53-generic #74-Ubuntu [31016.061380] Hardware name: Xen HVM domU, BIOS 4.2.amazon 11/11/2016 [31016.061380] task: 880406496040 ti: 8804064e task.ti: 8804064e [31016.061380] RIP: 0010:[] [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] RSP: 0018:88040fc43f78 EFLAGS: 00010082 [31016.061380] RAX: RBX: 88040fc4b840 RCX: 0001 [31016.061380] RDX: fffe RSI: 000123a0 RDI: 88040fc43f30 [31016.061380] RBP: 88040fc43f90 R08: f24a R09: 0001 [31016.061380] R10: 0001 R11: R12: 0001 [31016.061380] R13: 0003 R14: R15: 8804064e [31016.178419] FS: () GS:88040fc4() knlGS: [31016.178419] CS: 0010 DS: ES: CR0: 80050033 [31016.178419] CR2: 82c4801e6bda CR3: 000204f2f000 CR4: 001406e0 [31016.178419] DR0: DR1: DR2: [31016.178419] DR3: DR6: fffe0ff0 DR7: 0400 [31016.178419] Stack: [31016.178419] 0003 88040fc43fa8 [31016.178419] 814d6bc0 81f36a00 8804064e3e90 81837f32 [31016.178419] 8804064e3de8 8804064e [31016.178419] Call Trace: [31016.178419] [31016.178419] [] xen_evtchn_do_upcall+0x30/0x40 [31016.178419] [] xen_hvm_callback_vector+0x82/0x90 [31016.178419] [31016.178419] [] ? native_safe_halt+0x6/0x10 [31016.178419] [] default_idle+0x1e/0xe0 [31016.239315] [] arch_cpu_idle+0xf/0x20 [31016.239899] [] default_idle_call+0x2a/0x40 [31016.239899] [] cpu_startup_entry+0x2f1/0x350 [31016.239899] [] start_secondary+0x154/0x190 [31016.239899] Code: 01 00 00 00 65 44 8b 2d dc 56 b3 7e c6 03 00 44 89 e0 65 0f c1 05 2e d8 b3 7e 85 c0 75 35 48 8b 05 63 ce d0 00 44 89 ef ff 50 50 <9c> 58 0f 1f 44 00 00 f6 c4 02 75 23 65 8b 05 0a d8 b3 7e 65 c7 [31016.239899] RIP [] __xen_evtchn_do_upcall+0x43/0x80 [31016.239899] RSP [31016.239899] CR2: 82c4801e6bda [31016.239899] ---[ end trace 5b3e8ea32013e327 ]--- [31016.239899] Kernel panic - not syncing: Fatal exception in interrupt [31016.239899] Kernel Offset: disabled We believe this appeared in the last 1-2mo of releases, since this started happening after we did a `apt-get upgrade` on our machines after a bit of a pause. These are EC2 m4.xlarge servers running Ubuntu 16.04.1. They are Kubernetes minions, so I assume docker is likely the trigger. Unfortunately there aren't really any interesting logs in journalctl from the previous boot prior to the panic. Let me know what I can do to debug further. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-53-generic 4.4.0-53.74 ProcVersionSignature: Ubuntu 4.4.0-53.74-generic 4.4.30 Uname: Linux 4.4.0-53-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Dec 19 15:56 seq crw-rw 1 root audio 116, 33 Dec 19 15:56 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.4
[Kernel-packages] [Bug 1651199] Re: 16.04 - Kernel panic on docker server
** Tags added: kernel-fixed-upstream -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1651199 Title: 16.04 - Kernel panic on docker server Status in Linux: New Status in linux package in Ubuntu: Confirmed Status in linux source package in Xenial: Confirmed Bug description: [31016.057405] BUG: unable to handle kernel paging request at 82c4801e6bda [31016.061249] IP: [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] PGD 0 [31016.061380] Oops: 0010 [#1] SMP [31016.061380] Modules linked in: binfmt_misc xt_REDIRECT nf_nat_redirect veth xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo xt_addrtype iptable_filter xt_conntrack br_netfilter bridge stp llc overlay xt_nat xt_tcpudp iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables isofs ppdev serio_raw parport_pc parport ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ixgbevf psmouse floppy [31016.061380] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.4.0-53-generic #74-Ubuntu [31016.061380] Hardware name: Xen HVM domU, BIOS 4.2.amazon 11/11/2016 [31016.061380] task: 880406496040 ti: 8804064e task.ti: 8804064e [31016.061380] RIP: 0010:[] [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] RSP: 0018:88040fc43f78 EFLAGS: 00010082 [31016.061380] RAX: RBX: 88040fc4b840 RCX: 0001 [31016.061380] RDX: fffe RSI: 000123a0 RDI: 88040fc43f30 [31016.061380] RBP: 88040fc43f90 R08: f24a R09: 0001 [31016.061380] R10: 0001 R11: R12: 0001 [31016.061380] R13: 0003 R14: R15: 8804064e [31016.178419] FS: () GS:88040fc4() knlGS: [31016.178419] CS: 0010 DS: ES: CR0: 80050033 [31016.178419] CR2: 82c4801e6bda CR3: 000204f2f000 CR4: 001406e0 [31016.178419] DR0: DR1: DR2: [31016.178419] DR3: DR6: fffe0ff0 DR7: 0400 [31016.178419] Stack: [31016.178419] 0003 88040fc43fa8 [31016.178419] 814d6bc0 81f36a00 8804064e3e90 81837f32 [31016.178419] 8804064e3de8 8804064e [31016.178419] Call Trace: [31016.178419] [31016.178419] [] xen_evtchn_do_upcall+0x30/0x40 [31016.178419] [] xen_hvm_callback_vector+0x82/0x90 [31016.178419] [31016.178419] [] ? native_safe_halt+0x6/0x10 [31016.178419] [] default_idle+0x1e/0xe0 [31016.239315] [] arch_cpu_idle+0xf/0x20 [31016.239899] [] default_idle_call+0x2a/0x40 [31016.239899] [] cpu_startup_entry+0x2f1/0x350 [31016.239899] [] start_secondary+0x154/0x190 [31016.239899] Code: 01 00 00 00 65 44 8b 2d dc 56 b3 7e c6 03 00 44 89 e0 65 0f c1 05 2e d8 b3 7e 85 c0 75 35 48 8b 05 63 ce d0 00 44 89 ef ff 50 50 <9c> 58 0f 1f 44 00 00 f6 c4 02 75 23 65 8b 05 0a d8 b3 7e 65 c7 [31016.239899] RIP [] __xen_evtchn_do_upcall+0x43/0x80 [31016.239899] RSP [31016.239899] CR2: 82c4801e6bda [31016.239899] ---[ end trace 5b3e8ea32013e327 ]--- [31016.239899] Kernel panic - not syncing: Fatal exception in interrupt [31016.239899] Kernel Offset: disabled We believe this appeared in the last 1-2mo of releases, since this started happening after we did a `apt-get upgrade` on our machines after a bit of a pause. These are EC2 m4.xlarge servers running Ubuntu 16.04.1. They are Kubernetes minions, so I assume docker is likely the trigger. Unfortunately there aren't really any interesting logs in journalctl from the previous boot prior to the panic. Let me know what I can do to debug further. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-53-generic 4.4.0-53.74 ProcVersionSignature: Ubuntu 4.4.0-53.74-generic 4.4.30 Uname: Linux 4.4.0-53-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Dec 19 15:56 seq crw-rw 1 root audio 116, 33 Dec 19 15:56 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Mon Dec 19 18:12:12 2016
[Kernel-packages] [Bug 1651199] Re: 16.04 - Kernel panic on docker server
I also noticed that one of the boxes wasn't upgraded (for whatever reason) and has not had any panics in the last few days. The kernel version on that box is: Linux ip-10-0-26-30.us-west-2.compute.internal 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1651199 Title: 16.04 - Kernel panic on docker server Status in Linux: New Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Incomplete Bug description: [31016.057405] BUG: unable to handle kernel paging request at 82c4801e6bda [31016.061249] IP: [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] PGD 0 [31016.061380] Oops: 0010 [#1] SMP [31016.061380] Modules linked in: binfmt_misc xt_REDIRECT nf_nat_redirect veth xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo xt_addrtype iptable_filter xt_conntrack br_netfilter bridge stp llc overlay xt_nat xt_tcpudp iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables isofs ppdev serio_raw parport_pc parport ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ixgbevf psmouse floppy [31016.061380] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.4.0-53-generic #74-Ubuntu [31016.061380] Hardware name: Xen HVM domU, BIOS 4.2.amazon 11/11/2016 [31016.061380] task: 880406496040 ti: 8804064e task.ti: 8804064e [31016.061380] RIP: 0010:[] [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] RSP: 0018:88040fc43f78 EFLAGS: 00010082 [31016.061380] RAX: RBX: 88040fc4b840 RCX: 0001 [31016.061380] RDX: fffe RSI: 000123a0 RDI: 88040fc43f30 [31016.061380] RBP: 88040fc43f90 R08: f24a R09: 0001 [31016.061380] R10: 0001 R11: R12: 0001 [31016.061380] R13: 0003 R14: R15: 8804064e [31016.178419] FS: () GS:88040fc4() knlGS: [31016.178419] CS: 0010 DS: ES: CR0: 80050033 [31016.178419] CR2: 82c4801e6bda CR3: 000204f2f000 CR4: 001406e0 [31016.178419] DR0: DR1: DR2: [31016.178419] DR3: DR6: fffe0ff0 DR7: 0400 [31016.178419] Stack: [31016.178419] 0003 88040fc43fa8 [31016.178419] 814d6bc0 81f36a00 8804064e3e90 81837f32 [31016.178419] 8804064e3de8 8804064e [31016.178419] Call Trace: [31016.178419] [31016.178419] [] xen_evtchn_do_upcall+0x30/0x40 [31016.178419] [] xen_hvm_callback_vector+0x82/0x90 [31016.178419] [31016.178419] [] ? native_safe_halt+0x6/0x10 [31016.178419] [] default_idle+0x1e/0xe0 [31016.239315] [] arch_cpu_idle+0xf/0x20 [31016.239899] [] default_idle_call+0x2a/0x40 [31016.239899] [] cpu_startup_entry+0x2f1/0x350 [31016.239899] [] start_secondary+0x154/0x190 [31016.239899] Code: 01 00 00 00 65 44 8b 2d dc 56 b3 7e c6 03 00 44 89 e0 65 0f c1 05 2e d8 b3 7e 85 c0 75 35 48 8b 05 63 ce d0 00 44 89 ef ff 50 50 <9c> 58 0f 1f 44 00 00 f6 c4 02 75 23 65 8b 05 0a d8 b3 7e 65 c7 [31016.239899] RIP [] __xen_evtchn_do_upcall+0x43/0x80 [31016.239899] RSP [31016.239899] CR2: 82c4801e6bda [31016.239899] ---[ end trace 5b3e8ea32013e327 ]--- [31016.239899] Kernel panic - not syncing: Fatal exception in interrupt [31016.239899] Kernel Offset: disabled We believe this appeared in the last 1-2mo of releases, since this started happening after we did a `apt-get upgrade` on our machines after a bit of a pause. These are EC2 m4.xlarge servers running Ubuntu 16.04.1. They are Kubernetes minions, so I assume docker is likely the trigger. Unfortunately there aren't really any interesting logs in journalctl from the previous boot prior to the panic. Let me know what I can do to debug further. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-53-generic 4.4.0-53.74 ProcVersionSignature: Ubuntu 4.4.0-53.74-generic 4.4.30 Uname: Linux 4.4.0-53-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Dec 19 15:56 seq crw-rw 1 root audio 116, 33 Dec 19 15:56 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion:
[Kernel-packages] [Bug 1651199] Re: 16.04 - Kernel panic on docker server
My canary box is running 4.9.0-040900.201612111631 now and hasn't rebooted in ~1.5 days. I'll roll this out further, but this may be fixed in mainline. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1651199 Title: 16.04 - Kernel panic on docker server Status in Linux: New Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Incomplete Bug description: [31016.057405] BUG: unable to handle kernel paging request at 82c4801e6bda [31016.061249] IP: [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] PGD 0 [31016.061380] Oops: 0010 [#1] SMP [31016.061380] Modules linked in: binfmt_misc xt_REDIRECT nf_nat_redirect veth xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo xt_addrtype iptable_filter xt_conntrack br_netfilter bridge stp llc overlay xt_nat xt_tcpudp iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables isofs ppdev serio_raw parport_pc parport ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ixgbevf psmouse floppy [31016.061380] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.4.0-53-generic #74-Ubuntu [31016.061380] Hardware name: Xen HVM domU, BIOS 4.2.amazon 11/11/2016 [31016.061380] task: 880406496040 ti: 8804064e task.ti: 8804064e [31016.061380] RIP: 0010:[] [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] RSP: 0018:88040fc43f78 EFLAGS: 00010082 [31016.061380] RAX: RBX: 88040fc4b840 RCX: 0001 [31016.061380] RDX: fffe RSI: 000123a0 RDI: 88040fc43f30 [31016.061380] RBP: 88040fc43f90 R08: f24a R09: 0001 [31016.061380] R10: 0001 R11: R12: 0001 [31016.061380] R13: 0003 R14: R15: 8804064e [31016.178419] FS: () GS:88040fc4() knlGS: [31016.178419] CS: 0010 DS: ES: CR0: 80050033 [31016.178419] CR2: 82c4801e6bda CR3: 000204f2f000 CR4: 001406e0 [31016.178419] DR0: DR1: DR2: [31016.178419] DR3: DR6: fffe0ff0 DR7: 0400 [31016.178419] Stack: [31016.178419] 0003 88040fc43fa8 [31016.178419] 814d6bc0 81f36a00 8804064e3e90 81837f32 [31016.178419] 8804064e3de8 8804064e [31016.178419] Call Trace: [31016.178419] [31016.178419] [] xen_evtchn_do_upcall+0x30/0x40 [31016.178419] [] xen_hvm_callback_vector+0x82/0x90 [31016.178419] [31016.178419] [] ? native_safe_halt+0x6/0x10 [31016.178419] [] default_idle+0x1e/0xe0 [31016.239315] [] arch_cpu_idle+0xf/0x20 [31016.239899] [] default_idle_call+0x2a/0x40 [31016.239899] [] cpu_startup_entry+0x2f1/0x350 [31016.239899] [] start_secondary+0x154/0x190 [31016.239899] Code: 01 00 00 00 65 44 8b 2d dc 56 b3 7e c6 03 00 44 89 e0 65 0f c1 05 2e d8 b3 7e 85 c0 75 35 48 8b 05 63 ce d0 00 44 89 ef ff 50 50 <9c> 58 0f 1f 44 00 00 f6 c4 02 75 23 65 8b 05 0a d8 b3 7e 65 c7 [31016.239899] RIP [] __xen_evtchn_do_upcall+0x43/0x80 [31016.239899] RSP [31016.239899] CR2: 82c4801e6bda [31016.239899] ---[ end trace 5b3e8ea32013e327 ]--- [31016.239899] Kernel panic - not syncing: Fatal exception in interrupt [31016.239899] Kernel Offset: disabled We believe this appeared in the last 1-2mo of releases, since this started happening after we did a `apt-get upgrade` on our machines after a bit of a pause. These are EC2 m4.xlarge servers running Ubuntu 16.04.1. They are Kubernetes minions, so I assume docker is likely the trigger. Unfortunately there aren't really any interesting logs in journalctl from the previous boot prior to the panic. Let me know what I can do to debug further. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-53-generic 4.4.0-53.74 ProcVersionSignature: Ubuntu 4.4.0-53.74-generic 4.4.30 Uname: Linux 4.4.0-53-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Dec 19 15:56 seq crw-rw 1 root audio 116, 33 Dec 19 15:56 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command
[Kernel-packages] [Bug 1651199] Re: 16.04 - Kernel panic on docker server
Joseph - I have not identified a version of the kernel without the bug, though we have not seen this behavior from the released kernel ~2mo ago. I can try to pinpoint which kernel release it was, though I'd like to wait and see if the mainline kernel panics as well before bisecting. Unfortunately these happen several times a day so the turnaround cycle is a bit long. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1651199 Title: 16.04 - Kernel panic on docker server Status in Linux: New Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Incomplete Bug description: [31016.057405] BUG: unable to handle kernel paging request at 82c4801e6bda [31016.061249] IP: [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] PGD 0 [31016.061380] Oops: 0010 [#1] SMP [31016.061380] Modules linked in: binfmt_misc xt_REDIRECT nf_nat_redirect veth xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo xt_addrtype iptable_filter xt_conntrack br_netfilter bridge stp llc overlay xt_nat xt_tcpudp iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables isofs ppdev serio_raw parport_pc parport ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ixgbevf psmouse floppy [31016.061380] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.4.0-53-generic #74-Ubuntu [31016.061380] Hardware name: Xen HVM domU, BIOS 4.2.amazon 11/11/2016 [31016.061380] task: 880406496040 ti: 8804064e task.ti: 8804064e [31016.061380] RIP: 0010:[] [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] RSP: 0018:88040fc43f78 EFLAGS: 00010082 [31016.061380] RAX: RBX: 88040fc4b840 RCX: 0001 [31016.061380] RDX: fffe RSI: 000123a0 RDI: 88040fc43f30 [31016.061380] RBP: 88040fc43f90 R08: f24a R09: 0001 [31016.061380] R10: 0001 R11: R12: 0001 [31016.061380] R13: 0003 R14: R15: 8804064e [31016.178419] FS: () GS:88040fc4() knlGS: [31016.178419] CS: 0010 DS: ES: CR0: 80050033 [31016.178419] CR2: 82c4801e6bda CR3: 000204f2f000 CR4: 001406e0 [31016.178419] DR0: DR1: DR2: [31016.178419] DR3: DR6: fffe0ff0 DR7: 0400 [31016.178419] Stack: [31016.178419] 0003 88040fc43fa8 [31016.178419] 814d6bc0 81f36a00 8804064e3e90 81837f32 [31016.178419] 8804064e3de8 8804064e [31016.178419] Call Trace: [31016.178419] [31016.178419] [] xen_evtchn_do_upcall+0x30/0x40 [31016.178419] [] xen_hvm_callback_vector+0x82/0x90 [31016.178419] [31016.178419] [] ? native_safe_halt+0x6/0x10 [31016.178419] [] default_idle+0x1e/0xe0 [31016.239315] [] arch_cpu_idle+0xf/0x20 [31016.239899] [] default_idle_call+0x2a/0x40 [31016.239899] [] cpu_startup_entry+0x2f1/0x350 [31016.239899] [] start_secondary+0x154/0x190 [31016.239899] Code: 01 00 00 00 65 44 8b 2d dc 56 b3 7e c6 03 00 44 89 e0 65 0f c1 05 2e d8 b3 7e 85 c0 75 35 48 8b 05 63 ce d0 00 44 89 ef ff 50 50 <9c> 58 0f 1f 44 00 00 f6 c4 02 75 23 65 8b 05 0a d8 b3 7e 65 c7 [31016.239899] RIP [] __xen_evtchn_do_upcall+0x43/0x80 [31016.239899] RSP [31016.239899] CR2: 82c4801e6bda [31016.239899] ---[ end trace 5b3e8ea32013e327 ]--- [31016.239899] Kernel panic - not syncing: Fatal exception in interrupt [31016.239899] Kernel Offset: disabled We believe this appeared in the last 1-2mo of releases, since this started happening after we did a `apt-get upgrade` on our machines after a bit of a pause. These are EC2 m4.xlarge servers running Ubuntu 16.04.1. They are Kubernetes minions, so I assume docker is likely the trigger. Unfortunately there aren't really any interesting logs in journalctl from the previous boot prior to the panic. Let me know what I can do to debug further. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-53-generic 4.4.0-53.74 ProcVersionSignature: Ubuntu 4.4.0-53.74-generic 4.4.30 Uname: Linux 4.4.0-53-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Dec 19 15:56 seq crw-rw 1 root audio 116, 33 Dec 19 15:56 timer AplayDevices: Error:
[Kernel-packages] [Bug 1651199] Re: 16.04 - Kernel panic on docker server
Joseph - I switched one of the 6 machines that previously showed this behavior to the mainline kernel: Linux ip-10-0-32-196.us- west-2.compute.internal 4.9.0-040900-generic #201612111631 SMP Sun Dec 11 21:33:00 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux . I'll let you know the results of this experiment tomorrow. Fumihito - I saw that docker bug when researching this issue, and I downgraded docker-engine on those machines to 1.12.2. However, that didn't help. Hopefully the mainline kernel test will shed some light on this. For reference this is my docker config: $ sudo docker info Containers: 102 Running: 13 Paused: 0 Stopped: 89 Images: 13 Server Version: 1.12.2 Storage Driver: overlay2 Backing Filesystem: extfs Logging Driver: journald Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host overlay null Swarm: inactive Runtimes: runc Default Runtime: runc Security Options: apparmor seccomp Kernel Version: 4.9.0-040900-generic Operating System: Ubuntu 16.04.1 LTS OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 15.67 GiB Name: ip-10-0-32-196.us-west-2.compute.internal ID: TSIL:W3HI:APTU:DSVF:QDEF:HOJ6:PRB7:L7EL:QRQ2:K63C:CH22:RVQR Docker Root Dir: /data/docker Debug Mode (client): false Debug Mode (server): true File Descriptors: 59 Goroutines: 116 System Time: 2016-12-20T07:49:15.79205206Z EventsListeners: 0 Registry: https://index.docker.io/v1/ WARNING: No swap limit support Insecure Registries: 127.0.0.0/8 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1651199 Title: 16.04 - Kernel panic on docker server Status in Linux: New Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Incomplete Bug description: [31016.057405] BUG: unable to handle kernel paging request at 82c4801e6bda [31016.061249] IP: [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] PGD 0 [31016.061380] Oops: 0010 [#1] SMP [31016.061380] Modules linked in: binfmt_misc xt_REDIRECT nf_nat_redirect veth xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo xt_addrtype iptable_filter xt_conntrack br_netfilter bridge stp llc overlay xt_nat xt_tcpudp iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables isofs ppdev serio_raw parport_pc parport ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ixgbevf psmouse floppy [31016.061380] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.4.0-53-generic #74-Ubuntu [31016.061380] Hardware name: Xen HVM domU, BIOS 4.2.amazon 11/11/2016 [31016.061380] task: 880406496040 ti: 8804064e task.ti: 8804064e [31016.061380] RIP: 0010:[] [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] RSP: 0018:88040fc43f78 EFLAGS: 00010082 [31016.061380] RAX: RBX: 88040fc4b840 RCX: 0001 [31016.061380] RDX: fffe RSI: 000123a0 RDI: 88040fc43f30 [31016.061380] RBP: 88040fc43f90 R08: f24a R09: 0001 [31016.061380] R10: 0001 R11: R12: 0001 [31016.061380] R13: 0003 R14: R15: 8804064e [31016.178419] FS: () GS:88040fc4() knlGS: [31016.178419] CS: 0010 DS: ES: CR0: 80050033 [31016.178419] CR2: 82c4801e6bda CR3: 000204f2f000 CR4: 001406e0 [31016.178419] DR0: DR1: DR2: [31016.178419] DR3: DR6: fffe0ff0 DR7: 0400 [31016.178419] Stack: [31016.178419] 0003 88040fc43fa8 [31016.178419] 814d6bc0 81f36a00 8804064e3e90 81837f32 [31016.178419] 8804064e3de8 8804064e [31016.178419] Call Trace: [31016.178419] [31016.178419] [] xen_evtchn_do_upcall+0x30/0x40 [31016.178419] [] xen_hvm_callback_vector+0x82/0x90 [31016.178419] [31016.178419] [] ? native_safe_halt+0x6/0x10 [31016.178419] [] default_idle+0x1e/0xe0 [31016.239315] [] arch_cpu_idle+0xf/0x20 [31016.239899] [] default_idle_call+0x2a/0x40 [31016.239899] [] cpu_startup_entry+0x2f1/0x350 [31016.239899] [] start_secondary+0x154/0x190 [31016.239899] Code: 01 00 00 00 65 44 8b 2d dc 56 b3 7e c6 03 00 44 89 e0 65 0f c1 05 2e d8 b3 7e 85 c0 75 35 48 8b 05 63 ce d0 00 44 89 ef ff 50 50 <9c> 58 0f 1f 44 00 00 f6 c4 02 75 23 65 8b 05 0a d8 b3 7e 65 c7
[Kernel-packages] [Bug 1651199] [NEW] 16.04 - Kernel panic on docker server
Public bug reported: [31016.057405] BUG: unable to handle kernel paging request at 82c4801e6bda [31016.061249] IP: [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] PGD 0 [31016.061380] Oops: 0010 [#1] SMP [31016.061380] Modules linked in: binfmt_misc xt_REDIRECT nf_nat_redirect veth xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo xt_addrtype iptable_filter xt_conntrack br_netfilter bridge stp llc overlay xt_nat xt_tcpudp iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables isofs ppdev serio_raw parport_pc parport ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ixgbevf psmouse floppy [31016.061380] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.4.0-53-generic #74-Ubuntu [31016.061380] Hardware name: Xen HVM domU, BIOS 4.2.amazon 11/11/2016 [31016.061380] task: 880406496040 ti: 8804064e task.ti: 8804064e [31016.061380] RIP: 0010:[] [] __xen_evtchn_do_upcall+0x43/0x80 [31016.061380] RSP: 0018:88040fc43f78 EFLAGS: 00010082 [31016.061380] RAX: RBX: 88040fc4b840 RCX: 0001 [31016.061380] RDX: fffe RSI: 000123a0 RDI: 88040fc43f30 [31016.061380] RBP: 88040fc43f90 R08: f24a R09: 0001 [31016.061380] R10: 0001 R11: R12: 0001 [31016.061380] R13: 0003 R14: R15: 8804064e [31016.178419] FS: () GS:88040fc4() knlGS: [31016.178419] CS: 0010 DS: ES: CR0: 80050033 [31016.178419] CR2: 82c4801e6bda CR3: 000204f2f000 CR4: 001406e0 [31016.178419] DR0: DR1: DR2: [31016.178419] DR3: DR6: fffe0ff0 DR7: 0400 [31016.178419] Stack: [31016.178419] 0003 88040fc43fa8 [31016.178419] 814d6bc0 81f36a00 8804064e3e90 81837f32 [31016.178419] 8804064e3de8 8804064e [31016.178419] Call Trace: [31016.178419] [31016.178419] [] xen_evtchn_do_upcall+0x30/0x40 [31016.178419] [] xen_hvm_callback_vector+0x82/0x90 [31016.178419] [31016.178419] [] ? native_safe_halt+0x6/0x10 [31016.178419] [] default_idle+0x1e/0xe0 [31016.239315] [] arch_cpu_idle+0xf/0x20 [31016.239899] [] default_idle_call+0x2a/0x40 [31016.239899] [] cpu_startup_entry+0x2f1/0x350 [31016.239899] [] start_secondary+0x154/0x190 [31016.239899] Code: 01 00 00 00 65 44 8b 2d dc 56 b3 7e c6 03 00 44 89 e0 65 0f c1 05 2e d8 b3 7e 85 c0 75 35 48 8b 05 63 ce d0 00 44 89 ef ff 50 50 <9c> 58 0f 1f 44 00 00 f6 c4 02 75 23 65 8b 05 0a d8 b3 7e 65 c7 [31016.239899] RIP [] __xen_evtchn_do_upcall+0x43/0x80 [31016.239899] RSP [31016.239899] CR2: 82c4801e6bda [31016.239899] ---[ end trace 5b3e8ea32013e327 ]--- [31016.239899] Kernel panic - not syncing: Fatal exception in interrupt [31016.239899] Kernel Offset: disabled We believe this appeared in the last 1-2mo of releases, since this started happening after we did a `apt-get upgrade` on our machines after a bit of a pause. These are EC2 m4.xlarge servers running Ubuntu 16.04.1. They are Kubernetes minions, so I assume docker is likely the trigger. Unfortunately there aren't really any interesting logs in journalctl from the previous boot prior to the panic. Let me know what I can do to debug further. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-53-generic 4.4.0-53.74 ProcVersionSignature: Ubuntu 4.4.0-53.74-generic 4.4.30 Uname: Linux 4.4.0-53-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Dec 19 15:56 seq crw-rw 1 root audio 116, 33 Dec 19 15:56 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Mon Dec 19 18:12:12 2016 Ec2AMI: ami-f1ca1091 Ec2AMIManifest: (unknown) Ec2AvailabilityZone: us-west-2c Ec2InstanceType: m4.xlarge Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' JournalErrors: Error: command ['journalctl', '-b', '--priority=warning', '--lines=1000'] failed with exit code 1: Hint: You are currently not seeing messages from other users and the system. Users in the 'systemd-journal' group can see all messages. Pass -q to turn off this
