[Kernel-packages] [Bug 2049234] Re: non-Ethernet tunnel interface creation fails on jammy 5.15.0-81 or later
I’m unsure this is the right way but adding linux package as this was also caused by kernel upgrade. ** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2049234 Title: non-Ethernet tunnel interface creation fails on jammy 5.15.0-81 or later Status in linux package in Ubuntu: New Status in systemd package in Ubuntu: New Bug description: jammy kernel 5.15.0-81.90 includes the following backport: https://github.com/torvalds/linux/commit/b0ad3c179059089d809b477a1d445c1183a7b8fe https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/commit/?h=Ubuntu-5.15.0-91.101&id=1dcafa2582f335f0d929f3e9e736775a95271c43 https://bugs.launchpad.net/bugs/2029401 This causes creation of ip6tnl tunnel interface to fail through systemd-networkd. The following systemd commit also has to be applied to jammy: https://github.com/systemd/systemd/commit/3be64aa462642de9fbe9f95536f8d167c19994bb (Other commits might be required along with above) Since the systemd commit is missing on jammy, Kind=ip6tnl netdev configuration fails as follows: > eno1: Could not create stacked netdev: Invalid argument I believe this is a regression; We encountered this linux- image-5.15.0-91-generic installed via unattended-upgrade -- Also EINVAL during creating tunnel interfaces let AcceptIPv6RA function not to start, so we lose connection to our server due to missing IP address. ``` # eno1.network [Match] Name=eno1 [Network] Tunnel=test_tun ``` ``` # test_tun.netdev [NetDev] Name=test_tun Kind=ip6tnl [Tunnel] Local=2001:db8::a Remote=2001:db8::b Mode=any ``` ``` # test_tun.network [Match] Name=test_tun [Network] Address=10.0.0.1/24 ``` To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2049234/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1702665] Re: 4.4.0-83-generic + Docker + EC2 frequently crashes at cgroup_rmdir GPF
Not seeing crashes at 4.4.76-040476-generic for these few days. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1702665 Title: 4.4.0-83-generic + Docker + EC2 frequently crashes at cgroup_rmdir GPF Status in docker package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: We run xenial-based Docker container hosts on EC2 with Amazon ECS. Recently we refreshed our base image, we started to see frequent panic. Hosts run Amazon ECS Agent, and the agent automatically creates or destroys Docker container based on requests onto ECS cluster. I think this crash is caused by Docker-related operations, because crashing at cgroups. Also, we're running several different cluster with another EC2 instance types, using same image. This problem is only reproducing at t2.small instances. (We also run c4.large and m4.* clusters) Our previous image ran 4.4.0-79-generic, and we see no problem with 79. [30558.783899] general protection fault: [#1] SMP [30558.784056] Modules linked in: veth binfmt_misc xt_nat xt_comment xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc isofs ppdev input_leds serio_raw parport_pc parport autofs4 btrfs xor raid6_pq crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse floppy [30558.784056] CPU: 0 PID: 1 Comm: systemd Not tainted 4.4.0-83-generic #106-Ubuntu [30558.784056] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017 [30558.784056] task: 88007c4e8000 ti: 88007c4e4000 task.ti: 88007c4e4000 [30558.784056] RIP: 0010:[] [] cgroup_destroy_locked+0x5f/0xf0 [30558.784056] RSP: 0018:88007c4e7e40 EFLAGS: 00010212 [30558.784056] RAX: 8800114481bd RBX: 88002827ba50 RCX: 88007ab8d150 [30558.784056] RDX: 00111e7e0088 RSI: 88002827ba54 RDI: 8217745c [30558.784056] RBP: 88007c4e7e60 R08: 0020 R09: 88007c4e7e70 [30558.784056] R10: 0637760b R11: 880011829a80 R12: 88007ab8d000 [30558.784056] R13: R14: 559b48b2dcc0 R15: ff9c [30558.784056] FS: 7f29cf0db8c0() GS:88007d20() knlGS: [30558.784056] CS: 0010 DS: ES: CR0: 80050033 [30558.784056] CR2: 7fa466d58180 CR3: 7c19 CR4: 001406f0 [30558.784056] Stack: [30558.784056] 88002827ba50 88002827ba50 8800373e70d0 559b48b2dcc0 [30558.784056] 88007c4e7e80 811194b3 88002827ba50 [30558.784056] 88007c4e7ea0 8128ddcd 880011829a80 [30558.784056] Call Trace: [30558.784056] [] cgroup_rmdir+0x23/0x40 [30558.784056] [] kernfs_iop_rmdir+0x4d/0x80 [30558.784056] [] vfs_rmdir+0xb4/0x130 [30558.784056] [] do_rmdir+0x1df/0x200 [30558.784056] [] SyS_rmdir+0x16/0x20 [30558.784056] [] entry_SYSCALL_64_fastpath+0x16/0x71 [30558.784056] Code: 74 fd 48 c7 c7 5c 74 17 82 e8 8e 72 72 00 49 8b 94 24 50 01 00 00 49 8d 8c 24 50 01 00 00 48 39 d1 48 8d 42 f0 74 18 48 8b 50 08 82 b0 01 00 00 01 48 8b 50 10 48 39 d1 48 8d 42 f0 75 e8 49 [30558.784056] RIP [] cgroup_destroy_locked+0x5f/0xf0 [30558.784056] RSP [30558.960828] ---[ end trace 7634e03ff94e8934 ]--- [30558.964811] Kernel panic - not syncing: Fatal exception in interrupt [30558.968805] Kernel Offset: disabled ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-83-generic 4.4.0-83.106 ProcVersionSignature: Ubuntu 4.4.0-83.106-generic 4.4.70 Uname: Linux 4.4.0-83-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jul 6 10:22 seq crw-rw 1 root audio 116, 33 Jul 6 10:22 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.6 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Thu Jul 6 10:27:37 2017 Ec2AMI: ami-34100353 Ec2AMIManifest: (unknown) Ec2AvailabilityZone: ap-northeast-1c Ec2InstanceType: t2.small Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU PciMultimedia: ProcEnviron: SHELL=/bin/bash TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-83-generic root=UUID=f76be987-234f-4071-87d4-06318cfc2135 ro cgroup_
[Kernel-packages] [Bug 1702665] Re: 4.4.0-83-generic + Docker + EC2 frequently crashes at cgroup_rmdir GPF
Sure. Upgraded one of an instance running 4.4.76-040476-generic, aside of existing 4.12 instance. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1702665 Title: 4.4.0-83-generic + Docker + EC2 frequently crashes at cgroup_rmdir GPF Status in docker package in Ubuntu: New Status in linux package in Ubuntu: Confirmed Bug description: We run xenial-based Docker container hosts on EC2 with Amazon ECS. Recently we refreshed our base image, we started to see frequent panic. Hosts run Amazon ECS Agent, and the agent automatically creates or destroys Docker container based on requests onto ECS cluster. I think this crash is caused by Docker-related operations, because crashing at cgroups. Also, we're running several different cluster with another EC2 instance types, using same image. This problem is only reproducing at t2.small instances. (We also run c4.large and m4.* clusters) Our previous image ran 4.4.0-79-generic, and we see no problem with 79. [30558.783899] general protection fault: [#1] SMP [30558.784056] Modules linked in: veth binfmt_misc xt_nat xt_comment xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc isofs ppdev input_leds serio_raw parport_pc parport autofs4 btrfs xor raid6_pq crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse floppy [30558.784056] CPU: 0 PID: 1 Comm: systemd Not tainted 4.4.0-83-generic #106-Ubuntu [30558.784056] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017 [30558.784056] task: 88007c4e8000 ti: 88007c4e4000 task.ti: 88007c4e4000 [30558.784056] RIP: 0010:[] [] cgroup_destroy_locked+0x5f/0xf0 [30558.784056] RSP: 0018:88007c4e7e40 EFLAGS: 00010212 [30558.784056] RAX: 8800114481bd RBX: 88002827ba50 RCX: 88007ab8d150 [30558.784056] RDX: 00111e7e0088 RSI: 88002827ba54 RDI: 8217745c [30558.784056] RBP: 88007c4e7e60 R08: 0020 R09: 88007c4e7e70 [30558.784056] R10: 0637760b R11: 880011829a80 R12: 88007ab8d000 [30558.784056] R13: R14: 559b48b2dcc0 R15: ff9c [30558.784056] FS: 7f29cf0db8c0() GS:88007d20() knlGS: [30558.784056] CS: 0010 DS: ES: CR0: 80050033 [30558.784056] CR2: 7fa466d58180 CR3: 7c19 CR4: 001406f0 [30558.784056] Stack: [30558.784056] 88002827ba50 88002827ba50 8800373e70d0 559b48b2dcc0 [30558.784056] 88007c4e7e80 811194b3 88002827ba50 [30558.784056] 88007c4e7ea0 8128ddcd 880011829a80 [30558.784056] Call Trace: [30558.784056] [] cgroup_rmdir+0x23/0x40 [30558.784056] [] kernfs_iop_rmdir+0x4d/0x80 [30558.784056] [] vfs_rmdir+0xb4/0x130 [30558.784056] [] do_rmdir+0x1df/0x200 [30558.784056] [] SyS_rmdir+0x16/0x20 [30558.784056] [] entry_SYSCALL_64_fastpath+0x16/0x71 [30558.784056] Code: 74 fd 48 c7 c7 5c 74 17 82 e8 8e 72 72 00 49 8b 94 24 50 01 00 00 49 8d 8c 24 50 01 00 00 48 39 d1 48 8d 42 f0 74 18 48 8b 50 08 82 b0 01 00 00 01 48 8b 50 10 48 39 d1 48 8d 42 f0 75 e8 49 [30558.784056] RIP [] cgroup_destroy_locked+0x5f/0xf0 [30558.784056] RSP [30558.960828] ---[ end trace 7634e03ff94e8934 ]--- [30558.964811] Kernel panic - not syncing: Fatal exception in interrupt [30558.968805] Kernel Offset: disabled ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-83-generic 4.4.0-83.106 ProcVersionSignature: Ubuntu 4.4.0-83.106-generic 4.4.70 Uname: Linux 4.4.0-83-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jul 6 10:22 seq crw-rw 1 root audio 116, 33 Jul 6 10:22 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.6 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Thu Jul 6 10:27:37 2017 Ec2AMI: ami-34100353 Ec2AMIManifest: (unknown) Ec2AvailabilityZone: ap-northeast-1c Ec2InstanceType: t2.small Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU PciMultimedia: ProcEnviron: SHELL=/bin/bash TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-83-generic root=UUID=f76be987-234f-4071-
[Kernel-packages] [Bug 1702665] Re: 4.4.0-83-generic + Docker + EC2 frequently crashes at cgroup_rmdir GPF
Clarify: Upgraded one of an instance running 4.4.0-83 to 4.4.76-040476-generic, * -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1702665 Title: 4.4.0-83-generic + Docker + EC2 frequently crashes at cgroup_rmdir GPF Status in docker package in Ubuntu: New Status in linux package in Ubuntu: Confirmed Bug description: We run xenial-based Docker container hosts on EC2 with Amazon ECS. Recently we refreshed our base image, we started to see frequent panic. Hosts run Amazon ECS Agent, and the agent automatically creates or destroys Docker container based on requests onto ECS cluster. I think this crash is caused by Docker-related operations, because crashing at cgroups. Also, we're running several different cluster with another EC2 instance types, using same image. This problem is only reproducing at t2.small instances. (We also run c4.large and m4.* clusters) Our previous image ran 4.4.0-79-generic, and we see no problem with 79. [30558.783899] general protection fault: [#1] SMP [30558.784056] Modules linked in: veth binfmt_misc xt_nat xt_comment xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc isofs ppdev input_leds serio_raw parport_pc parport autofs4 btrfs xor raid6_pq crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse floppy [30558.784056] CPU: 0 PID: 1 Comm: systemd Not tainted 4.4.0-83-generic #106-Ubuntu [30558.784056] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017 [30558.784056] task: 88007c4e8000 ti: 88007c4e4000 task.ti: 88007c4e4000 [30558.784056] RIP: 0010:[] [] cgroup_destroy_locked+0x5f/0xf0 [30558.784056] RSP: 0018:88007c4e7e40 EFLAGS: 00010212 [30558.784056] RAX: 8800114481bd RBX: 88002827ba50 RCX: 88007ab8d150 [30558.784056] RDX: 00111e7e0088 RSI: 88002827ba54 RDI: 8217745c [30558.784056] RBP: 88007c4e7e60 R08: 0020 R09: 88007c4e7e70 [30558.784056] R10: 0637760b R11: 880011829a80 R12: 88007ab8d000 [30558.784056] R13: R14: 559b48b2dcc0 R15: ff9c [30558.784056] FS: 7f29cf0db8c0() GS:88007d20() knlGS: [30558.784056] CS: 0010 DS: ES: CR0: 80050033 [30558.784056] CR2: 7fa466d58180 CR3: 7c19 CR4: 001406f0 [30558.784056] Stack: [30558.784056] 88002827ba50 88002827ba50 8800373e70d0 559b48b2dcc0 [30558.784056] 88007c4e7e80 811194b3 88002827ba50 [30558.784056] 88007c4e7ea0 8128ddcd 880011829a80 [30558.784056] Call Trace: [30558.784056] [] cgroup_rmdir+0x23/0x40 [30558.784056] [] kernfs_iop_rmdir+0x4d/0x80 [30558.784056] [] vfs_rmdir+0xb4/0x130 [30558.784056] [] do_rmdir+0x1df/0x200 [30558.784056] [] SyS_rmdir+0x16/0x20 [30558.784056] [] entry_SYSCALL_64_fastpath+0x16/0x71 [30558.784056] Code: 74 fd 48 c7 c7 5c 74 17 82 e8 8e 72 72 00 49 8b 94 24 50 01 00 00 49 8d 8c 24 50 01 00 00 48 39 d1 48 8d 42 f0 74 18 48 8b 50 08 82 b0 01 00 00 01 48 8b 50 10 48 39 d1 48 8d 42 f0 75 e8 49 [30558.784056] RIP [] cgroup_destroy_locked+0x5f/0xf0 [30558.784056] RSP [30558.960828] ---[ end trace 7634e03ff94e8934 ]--- [30558.964811] Kernel panic - not syncing: Fatal exception in interrupt [30558.968805] Kernel Offset: disabled ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-83-generic 4.4.0-83.106 ProcVersionSignature: Ubuntu 4.4.0-83.106-generic 4.4.70 Uname: Linux 4.4.0-83-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jul 6 10:22 seq crw-rw 1 root audio 116, 33 Jul 6 10:22 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.6 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Thu Jul 6 10:27:37 2017 Ec2AMI: ami-34100353 Ec2AMIManifest: (unknown) Ec2AvailabilityZone: ap-northeast-1c Ec2InstanceType: t2.small Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU PciMultimedia: ProcEnviron: SHELL=/bin/bash TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-83-generic root=UUID=f76be987-234f-4071-87d4-06318cfc213
[Kernel-packages] [Bug 1702665] Re: 4.4.0-83-generic + Docker + EC2 frequently crashes at cgroup_rmdir GPF
4.12 instance is still running without crash, survived this weekend. Added kernel-fixed-upstream tag. ** Tags added: kernel-fixed-upstream -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1702665 Title: 4.4.0-83-generic + Docker + EC2 frequently crashes at cgroup_rmdir GPF Status in docker package in Ubuntu: New Status in linux package in Ubuntu: Confirmed Bug description: We run xenial-based Docker container hosts on EC2 with Amazon ECS. Recently we refreshed our base image, we started to see frequent panic. Hosts run Amazon ECS Agent, and the agent automatically creates or destroys Docker container based on requests onto ECS cluster. I think this crash is caused by Docker-related operations, because crashing at cgroups. Also, we're running several different cluster with another EC2 instance types, using same image. This problem is only reproducing at t2.small instances. (We also run c4.large and m4.* clusters) Our previous image ran 4.4.0-79-generic, and we see no problem with 79. [30558.783899] general protection fault: [#1] SMP [30558.784056] Modules linked in: veth binfmt_misc xt_nat xt_comment xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc isofs ppdev input_leds serio_raw parport_pc parport autofs4 btrfs xor raid6_pq crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse floppy [30558.784056] CPU: 0 PID: 1 Comm: systemd Not tainted 4.4.0-83-generic #106-Ubuntu [30558.784056] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017 [30558.784056] task: 88007c4e8000 ti: 88007c4e4000 task.ti: 88007c4e4000 [30558.784056] RIP: 0010:[] [] cgroup_destroy_locked+0x5f/0xf0 [30558.784056] RSP: 0018:88007c4e7e40 EFLAGS: 00010212 [30558.784056] RAX: 8800114481bd RBX: 88002827ba50 RCX: 88007ab8d150 [30558.784056] RDX: 00111e7e0088 RSI: 88002827ba54 RDI: 8217745c [30558.784056] RBP: 88007c4e7e60 R08: 0020 R09: 88007c4e7e70 [30558.784056] R10: 0637760b R11: 880011829a80 R12: 88007ab8d000 [30558.784056] R13: R14: 559b48b2dcc0 R15: ff9c [30558.784056] FS: 7f29cf0db8c0() GS:88007d20() knlGS: [30558.784056] CS: 0010 DS: ES: CR0: 80050033 [30558.784056] CR2: 7fa466d58180 CR3: 7c19 CR4: 001406f0 [30558.784056] Stack: [30558.784056] 88002827ba50 88002827ba50 8800373e70d0 559b48b2dcc0 [30558.784056] 88007c4e7e80 811194b3 88002827ba50 [30558.784056] 88007c4e7ea0 8128ddcd 880011829a80 [30558.784056] Call Trace: [30558.784056] [] cgroup_rmdir+0x23/0x40 [30558.784056] [] kernfs_iop_rmdir+0x4d/0x80 [30558.784056] [] vfs_rmdir+0xb4/0x130 [30558.784056] [] do_rmdir+0x1df/0x200 [30558.784056] [] SyS_rmdir+0x16/0x20 [30558.784056] [] entry_SYSCALL_64_fastpath+0x16/0x71 [30558.784056] Code: 74 fd 48 c7 c7 5c 74 17 82 e8 8e 72 72 00 49 8b 94 24 50 01 00 00 49 8d 8c 24 50 01 00 00 48 39 d1 48 8d 42 f0 74 18 48 8b 50 08 82 b0 01 00 00 01 48 8b 50 10 48 39 d1 48 8d 42 f0 75 e8 49 [30558.784056] RIP [] cgroup_destroy_locked+0x5f/0xf0 [30558.784056] RSP [30558.960828] ---[ end trace 7634e03ff94e8934 ]--- [30558.964811] Kernel panic - not syncing: Fatal exception in interrupt [30558.968805] Kernel Offset: disabled ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-83-generic 4.4.0-83.106 ProcVersionSignature: Ubuntu 4.4.0-83.106-generic 4.4.70 Uname: Linux 4.4.0-83-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jul 6 10:22 seq crw-rw 1 root audio 116, 33 Jul 6 10:22 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.6 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Thu Jul 6 10:27:37 2017 Ec2AMI: ami-34100353 Ec2AMIManifest: (unknown) Ec2AvailabilityZone: ap-northeast-1c Ec2InstanceType: t2.small Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU PciMultimedia: ProcEnviron: SHELL=/bin/bash TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.
[Kernel-packages] [Bug 1702665] Re: 4.4.0-83-generic + Docker + EC2 t2.small frequently crashes at cgroup_rmdir GPF
Additional logs from 4.4.0 machine by running `dmesg -w` until crash. [ 3031.276097] [ cut here ] [ 3031.276106] WARNING: CPU: 0 PID: 31804 at /build/linux-0uniEn/linux-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_finish_destroy+0x6b/0xc0() [ 3031.276108] Modules linked in: veth binfmt_misc xt_nat xt_comment xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc aufs isofs ppdev input_leds serio_raw parport_pc parport autofs4 btrfs xor raid6_pq crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse floppy [ 3031.276131] CPU: 0 PID: 31804 Comm: kworker/u30:3 Not tainted 4.4.0-83-generic #106-Ubuntu [ 3031.276132] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017 [ 3031.276136] Workqueue: netns cleanup_net [ 3031.276137] 0286 e9d24b1b 880043383ba0 813f9513 [ 3031.276140] 81d75940 880043383bd8 81081322 [ 3031.276141] 880070297800 0006 880043383ca8 [ 3031.276143] Call Trace: [ 3031.276148] [] dump_stack+0x63/0x90 [ 3031.276151] [] warn_slowpath_common+0x82/0xc0 [ 3031.276153] [] warn_slowpath_null+0x1a/0x20 [ 3031.276154] [] in6_dev_finish_destroy+0x6b/0xc0 [ 3031.276157] [] ip6_route_dev_notify+0x116/0x130 [ 3031.276159] [] notifier_call_chain+0x4a/0x70 [ 3031.276161] [] raw_notifier_call_chain+0x16/0x20 [ 3031.276163] [] call_netdevice_notifiers_info+0x35/0x60 [ 3031.276165] [] netdev_run_todo+0x16d/0x320 [ 3031.276168] [] ? rollback_registered_many+0x2c9/0x340 [ 3031.276171] [] rtnl_unlock+0xe/0x10 [ 3031.276173] [] default_device_exit_batch+0x147/0x170 [ 3031.276176] [] ? __wake_up_sync+0x20/0x20 [ 3031.276178] [] ops_exit_list.isra.4+0x52/0x60 [ 3031.276179] [] cleanup_net+0x1c2/0x2a0 [ 3031.276182] [] process_one_work+0x165/0x480 [ 3031.276184] [] worker_thread+0x4b/0x4c0 [ 3031.276186] [] ? process_one_work+0x480/0x480 [ 3031.276187] [] kthread+0xe5/0x100 [ 3031.276189] [] ? kthread_create_on_node+0x1e0/0x1e0 [ 3031.276192] [] ret_from_fork+0x3f/0x70 [ 3031.276193] [] ? kthread_create_on_node+0x1e0/0x1e0 [ 3031.276194] ---[ end trace 5d921dfe814c9c7c ]--- [ 3031.276195] [ cut here ] [ 3031.276197] WARNING: CPU: 0 PID: 31804 at /build/linux-0uniEn/linux-4.4.0/net/ipv6/addrconf_core.c:160 in6_dev_finish_destroy+0x83/0xc0() [ 3031.276198] Modules linked in: veth binfmt_misc xt_nat xt_comment xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc aufs isofs ppdev input_leds serio_raw parport_pc parport autofs4 btrfs xor raid6_pq crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse floppy [ 3031.276211] CPU: 0 PID: 31804 Comm: kworker/u30:3 Tainted: GW 4.4.0-83-generic #106-Ubuntu [ 3031.276212] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017 [ 3031.276213] Workqueue: netns cleanup_net [ 3031.276214] 0286 e9d24b1b 880043383ba0 813f9513 [ 3031.276216] 81d75940 880043383bd8 81081322 [ 3031.276217] 880070297800 0006 880043383ca8 [ 3031.276219] Call Trace: [ 3031.276220] [] dump_stack+0x63/0x90 [ 3031.276222] [] warn_slowpath_common+0x82/0xc0 [ 3031.276224] [] warn_slowpath_null+0x1a/0x20 [ 3031.276225] [] in6_dev_finish_destroy+0x83/0xc0 [ 3031.276226] [] ip6_route_dev_notify+0x116/0x130 [ 3031.276228] [] notifier_call_chain+0x4a/0x70 [ 3031.276229] [] raw_notifier_call_chain+0x16/0x20 [ 3031.276231] [] call_netdevice_notifiers_info+0x35/0x60 [ 3031.276233] [] netdev_run_todo+0x16d/0x320 [ 3031.276234] [] ? rollback_registered_many+0x2c9/0x340 [ 3031.276237] [] rtnl_unlock+0xe/0x10 [ 3031.276239] [] default_device_exit_batch+0x147/0x170 [ 3031.276240] [] ? __wake_up_sync+0x20/0x20 [ 3031.276242] [] ops_exit_list.isra.4+0x52/0x60 [ 3031.276243] [] cleanup_net+0x1c2/0x2a0 [ 3031.276245] [] process_one_work+0x165/0x480 [ 3031.276247] [] worker_thread+0x4b/0x4c0 [ 3031.276249] [] ? process_one_work+0x480/0x480 [ 3031.276250] [] kthread+0xe5/0x100 [ 3031.276251] [] ? kthread_create_on_node+0x1e0/0x1e0 [ 3031.276253] [] ret_from_fork+0x3f/0x70 [ 3031.276254] [] ? kthread_create_on_node+0x1e0/0x1e0 [ 3031.276255] ---[ end trace 5d921dfe814c9c7d ]--- [ 3031.276260] BUG: unable to handle kernel NULL pointer dereference at 0448 [ 3031.280059] IP: [] in6_dev_finish_destroy+0x2d/0xc0 [ 3031.280059] PGD 7bbf6067 PUD 7ba63067 PMD 0 [ 3031.280059] Oops: [#1] SMP [ 3031.280059] Modules linked in: veth binfm
[Kernel-packages] [Bug 1702665] Re: 4.4.0-83-generic + Docker + EC2 t2.small frequently crashes at cgroup_rmdir GPF
Noticed that these log appears before panic occurs for both kind of panics (GPF and "unable to handle kernel paging request"). [ 210.064089] unregister_netdevice: waiting for lo to become free. Usage count = 1 [ 1260.052073] unregister_netdevice: waiting for lo to become free. Usage count = 1 [ 3042.312072] unregister_netdevice: waiting for lo to become free. Usage count = 1 [ 4330.228094] unregister_netdevice: waiting for lo to become free. Usage count = 1 [ 4340.484089] unregister_netdevice: waiting for lo to become free. Usage count = 1 [ 5430.912074] unregister_netdevice: waiting for lo to become free. Usage count = 1 [ 7233.044085] unregister_netdevice: waiting for lo to become free. Usage count = 1 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1702665 Title: 4.4.0-83-generic + Docker + EC2 t2.small frequently crashes at cgroup_rmdir GPF Status in docker package in Ubuntu: New Status in linux package in Ubuntu: Confirmed Bug description: We run xenial-based Docker container hosts on EC2 with Amazon ECS. Recently we refreshed our base image, we started to see frequent panic. Hosts run Amazon ECS Agent, and the agent automatically creates or destroys Docker container based on requests onto ECS cluster. I think this crash is caused by Docker-related operations, because crashing at cgroups. Also, we're running several different cluster with another EC2 instance types, using same image. This problem is only reproducing at t2.small instances. (We also run c4.large and m4.* clusters) Our previous image ran 4.4.0-79-generic, and we see no problem with 79. [30558.783899] general protection fault: [#1] SMP [30558.784056] Modules linked in: veth binfmt_misc xt_nat xt_comment xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc isofs ppdev input_leds serio_raw parport_pc parport autofs4 btrfs xor raid6_pq crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse floppy [30558.784056] CPU: 0 PID: 1 Comm: systemd Not tainted 4.4.0-83-generic #106-Ubuntu [30558.784056] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017 [30558.784056] task: 88007c4e8000 ti: 88007c4e4000 task.ti: 88007c4e4000 [30558.784056] RIP: 0010:[] [] cgroup_destroy_locked+0x5f/0xf0 [30558.784056] RSP: 0018:88007c4e7e40 EFLAGS: 00010212 [30558.784056] RAX: 8800114481bd RBX: 88002827ba50 RCX: 88007ab8d150 [30558.784056] RDX: 00111e7e0088 RSI: 88002827ba54 RDI: 8217745c [30558.784056] RBP: 88007c4e7e60 R08: 0020 R09: 88007c4e7e70 [30558.784056] R10: 0637760b R11: 880011829a80 R12: 88007ab8d000 [30558.784056] R13: R14: 559b48b2dcc0 R15: ff9c [30558.784056] FS: 7f29cf0db8c0() GS:88007d20() knlGS: [30558.784056] CS: 0010 DS: ES: CR0: 80050033 [30558.784056] CR2: 7fa466d58180 CR3: 7c19 CR4: 001406f0 [30558.784056] Stack: [30558.784056] 88002827ba50 88002827ba50 8800373e70d0 559b48b2dcc0 [30558.784056] 88007c4e7e80 811194b3 88002827ba50 [30558.784056] 88007c4e7ea0 8128ddcd 880011829a80 [30558.784056] Call Trace: [30558.784056] [] cgroup_rmdir+0x23/0x40 [30558.784056] [] kernfs_iop_rmdir+0x4d/0x80 [30558.784056] [] vfs_rmdir+0xb4/0x130 [30558.784056] [] do_rmdir+0x1df/0x200 [30558.784056] [] SyS_rmdir+0x16/0x20 [30558.784056] [] entry_SYSCALL_64_fastpath+0x16/0x71 [30558.784056] Code: 74 fd 48 c7 c7 5c 74 17 82 e8 8e 72 72 00 49 8b 94 24 50 01 00 00 49 8d 8c 24 50 01 00 00 48 39 d1 48 8d 42 f0 74 18 48 8b 50 08 82 b0 01 00 00 01 48 8b 50 10 48 39 d1 48 8d 42 f0 75 e8 49 [30558.784056] RIP [] cgroup_destroy_locked+0x5f/0xf0 [30558.784056] RSP [30558.960828] ---[ end trace 7634e03ff94e8934 ]--- [30558.964811] Kernel panic - not syncing: Fatal exception in interrupt [30558.968805] Kernel Offset: disabled ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-83-generic 4.4.0-83.106 ProcVersionSignature: Ubuntu 4.4.0-83.106-generic 4.4.70 Uname: Linux 4.4.0-83-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jul 6 10:22 seq crw-rw 1 root audio 116, 33 Jul 6 10:22 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.6 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/s
[Kernel-packages] [Bug 1702665] Re: 4.4.0-83-generic + Docker + EC2 t2.small frequently crashes at cgroup_rmdir GPF
We've switched most of instances in the cluster to c4.large instance type, but another panic occur. Panic happened at different trace, so this may not relate to the first GPF, but pasting log below for information. Also, t2.small 4.12.x instance is now running 16 hours+, we have not seen panics yet. still keeping eyes how it goes. [ 7236.612035] BUG: unable to handle kernel paging request at 0001000d [ 7236.614750] IP: [] free_pipe_info+0x57/0x90 [ 7236.615155] PGD 0 [ 7236.615155] Oops: [#1] SMP [ 7236.615155] Modules linked in: veth binfmt_misc xt_nat xt_comment xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc aufs isofs ppdev input_leds serio_raw parport_pc parport autofs4 btrfs xor raid6_pq crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul psmouse glue_helper ablk_helper cryptd ixgbevf floppy [ 7236.615155] CPU: 0 PID: 1 Comm: systemd Not tainted 4.4.0-83-generic #106-Ubuntu [ 7236.615155] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017 [ 7236.615155] task: 8800eabf ti: 8800eabec000 task.ti: 8800eabec000 [ 7236.615155] RIP: 0010:[] [] free_pipe_info+0x57/0x90 [ 7236.615155] RSP: 0018:8800eabefdf8 EFLAGS: 00010202 [ 7236.615155] RAX: fffd RBX: 0008 RCX: 012c [ 7236.615155] RDX: 0028 RSI: 88005d5cc940 RDI: 8800e9840180 [ 7236.615155] RBP: 8800eabefe08 R08: R09: [ 7236.615155] R10: 8800e9e125b8 R11: 8800b5fc8510 R12: 8800e9840180 [ 7236.615155] R13: 8800e9e125b8 R14: 8800eab44c20 R15: 8800e9ee80c0 [ 7236.615155] FS: 7fb3ddc0c8c0() GS:8800eb60() knlGS: [ 7236.615155] CS: 0010 DS: ES: CR0: 80050033 [ 7236.615155] CR2: 0001000d CR3: 3641f000 CR4: 001406f0 [ 7236.615155] Stack: [ 7236.615155] 8800e9e12640 8800e9840180 8800eabefe30 812182dc [ 7236.615155] 8800e9840180 8800b5fc8500 8800e9e125b8 8800eabefe58 [ 7236.615155] 81218390 8800b5fc8500 0008 8800e9e125b8 [ 7236.615155] Call Trace: [ 7236.615155] [] put_pipe_info+0x5c/0x70 [ 7236.615155] [] pipe_release+0xa0/0xb0 [ 7236.615155] [] __fput+0xe4/0x220 [ 7236.615155] [] fput+0xe/0x10 [ 7236.615155] [] task_work_run+0x81/0xa0 [ 7236.615155] [] exit_to_usermode_loop+0xc2/0xd0 [ 7236.615155] [] syscall_return_slowpath+0x4e/0x60 [ 7236.615155] [] int_ret_from_sys_call+0x25/0x8f [ 7236.615155] Code: 4a e7 ff 41 8b 44 24 48 85 c0 74 2c 48 63 c3 48 8d 14 80 49 8b 84 24 80 00 00 00 48 8d 34 d0 48 8b 46 10 48 85 c0 74 06 4c 89 e7 50 10 83 c3 01 41 39 5c 24 48 77 d4 49 8b 7c 24 68 48 85 ff [ 7236.615155] RIP [] free_pipe_info+0x57/0x90 [ 7236.615155] RSP [ 7236.615155] CR2: 0001000d [ 7236.723319] ---[ end trace aca2b9bb73327372 ]--- [ 7236.726141] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0009 [ 7236.726141] [ 7236.729702] Kernel Offset: disabled -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1702665 Title: 4.4.0-83-generic + Docker + EC2 t2.small frequently crashes at cgroup_rmdir GPF Status in docker package in Ubuntu: New Status in linux package in Ubuntu: Confirmed Bug description: We run xenial-based Docker container hosts on EC2 with Amazon ECS. Recently we refreshed our base image, we started to see frequent panic. Hosts run Amazon ECS Agent, and the agent automatically creates or destroys Docker container based on requests onto ECS cluster. I think this crash is caused by Docker-related operations, because crashing at cgroups. Also, we're running several different cluster with another EC2 instance types, using same image. This problem is only reproducing at t2.small instances. (We also run c4.large and m4.* clusters) Our previous image ran 4.4.0-79-generic, and we see no problem with 79. [30558.783899] general protection fault: [#1] SMP [30558.784056] Modules linked in: veth binfmt_misc xt_nat xt_comment xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc isofs ppdev input_leds serio_raw parport_pc parport autofs4 btrfs xor raid6_pq crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse floppy [30558.784056] CPU: 0 PID: 1 Comm: systemd Not tainted 4.4.0-83-generic #106-Ubuntu [30558.784056] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017 [30558.784056] task: 88007c4e8000 ti: 88007c
[Kernel-packages] [Bug 1702665] Re: 4.4.0-83-generic + Docker + EC2 t2.small frequently crashes at cgroup_rmdir GPF
Created new instance and bumped to the 4.12 latest.
$ uname -a
Linux ${HOSTNAME} 4.12.0-041200-generic #201707022031 SMP Mon Jul 3 00:32:52
UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1702665
Title:
4.4.0-83-generic + Docker + EC2 t2.small frequently crashes at
cgroup_rmdir GPF
Status in docker package in Ubuntu:
New
Status in linux package in Ubuntu:
Confirmed
Bug description:
We run xenial-based Docker container hosts on EC2 with Amazon ECS.
Recently we refreshed our base image, we started to see frequent
panic.
Hosts run Amazon ECS Agent, and the agent automatically creates or
destroys Docker container based on requests onto ECS cluster.
I think this crash is caused by Docker-related operations, because
crashing at cgroups.
Also, we're running several different cluster with another EC2
instance types, using same image. This problem is only reproducing at
t2.small instances. (We also run c4.large and m4.* clusters)
Our previous image ran 4.4.0-79-generic, and we see no problem with
79.
[30558.783899] general protection fault: [#1] SMP
[30558.784056] Modules linked in: veth binfmt_misc xt_nat xt_comment
xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack
x_tables nf_nat nf_conntrack br_netfilter bridge stp llc isofs ppdev input_leds
serio_raw parport_pc parport autofs4 btrfs xor raid6_pq crct10dif_pclmul
crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul
glue_helper ablk_helper cryptd psmouse floppy
[30558.784056] CPU: 0 PID: 1 Comm: systemd Not tainted 4.4.0-83-generic
#106-Ubuntu
[30558.784056] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017
[30558.784056] task: 88007c4e8000 ti: 88007c4e4000 task.ti:
88007c4e4000
[30558.784056] RIP: 0010:[] []
cgroup_destroy_locked+0x5f/0xf0
[30558.784056] RSP: 0018:88007c4e7e40 EFLAGS: 00010212
[30558.784056] RAX: 8800114481bd RBX: 88002827ba50 RCX:
88007ab8d150
[30558.784056] RDX: 00111e7e0088 RSI: 88002827ba54 RDI:
8217745c
[30558.784056] RBP: 88007c4e7e60 R08: 0020 R09:
88007c4e7e70
[30558.784056] R10: 0637760b R11: 880011829a80 R12:
88007ab8d000
[30558.784056] R13: R14: 559b48b2dcc0 R15:
ff9c
[30558.784056] FS: 7f29cf0db8c0() GS:88007d20()
knlGS:
[30558.784056] CS: 0010 DS: ES: CR0: 80050033
[30558.784056] CR2: 7fa466d58180 CR3: 7c19 CR4:
001406f0
[30558.784056] Stack:
[30558.784056] 88002827ba50 88002827ba50 8800373e70d0
559b48b2dcc0
[30558.784056] 88007c4e7e80 811194b3 88002827ba50
[30558.784056] 88007c4e7ea0 8128ddcd 880011829a80
[30558.784056] Call Trace:
[30558.784056] [] cgroup_rmdir+0x23/0x40
[30558.784056] [] kernfs_iop_rmdir+0x4d/0x80
[30558.784056] [] vfs_rmdir+0xb4/0x130
[30558.784056] [] do_rmdir+0x1df/0x200
[30558.784056] [] SyS_rmdir+0x16/0x20
[30558.784056] [] entry_SYSCALL_64_fastpath+0x16/0x71
[30558.784056] Code: 74 fd 48 c7 c7 5c 74 17 82 e8 8e 72 72 00 49 8b 94 24 50
01 00 00 49 8d 8c 24 50 01 00 00 48 39 d1 48 8d 42 f0 74 18 48 8b 50 08 82
b0 01 00 00 01 48 8b 50 10 48 39 d1 48 8d 42 f0 75 e8 49
[30558.784056] RIP [] cgroup_destroy_locked+0x5f/0xf0
[30558.784056] RSP
[30558.960828] ---[ end trace 7634e03ff94e8934 ]---
[30558.964811] Kernel panic - not syncing: Fatal exception in interrupt
[30558.968805] Kernel Offset: disabled
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-83-generic 4.4.0-83.106
ProcVersionSignature: Ubuntu 4.4.0-83.106-generic 4.4.70
Uname: Linux 4.4.0-83-generic x86_64
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 6 10:22 seq
crw-rw 1 root audio 116, 33 Jul 6 10:22 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.1-0ubuntu2.6
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
Date: Thu Jul 6 10:27:37 2017
Ec2AMI: ami-34100353
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: ap-northeast-1c
Ec2InstanceType: t2.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: Xen HVM domU
PciMultimedia:
ProcEnviron:
SHELL=/bin/bash
TERM=screen-256color
PATH=(custom, no user)
LANG=en_US.UTF-8
ProcFB:
[Kernel-packages] [Bug 1702665] [NEW] 4.4.0-83-generic + Docker + EC2 t2.small frequently crashes at cgroup_rmdir GPF
Public bug reported: We run xenial-based Docker container hosts on EC2 with Amazon ECS. Recently we refreshed our base image, we started to see frequent panic. Hosts run Amazon ECS Agent, and the agent automatically creates or destroys Docker container based on requests onto ECS cluster. I think this crash is caused by Docker-related operations, because crashing at cgroups. Also, we're running several different cluster with another EC2 instance types, using same image. This problem is only reproducing at t2.small instances. (We also run c4.large and m4.* clusters) Our previous image ran 4.4.0-79-generic, and we see no problem with 79. [30558.783899] general protection fault: [#1] SMP [30558.784056] Modules linked in: veth binfmt_misc xt_nat xt_comment xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc isofs ppdev input_leds serio_raw parport_pc parport autofs4 btrfs xor raid6_pq crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse floppy [30558.784056] CPU: 0 PID: 1 Comm: systemd Not tainted 4.4.0-83-generic #106-Ubuntu [30558.784056] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017 [30558.784056] task: 88007c4e8000 ti: 88007c4e4000 task.ti: 88007c4e4000 [30558.784056] RIP: 0010:[] [] cgroup_destroy_locked+0x5f/0xf0 [30558.784056] RSP: 0018:88007c4e7e40 EFLAGS: 00010212 [30558.784056] RAX: 8800114481bd RBX: 88002827ba50 RCX: 88007ab8d150 [30558.784056] RDX: 00111e7e0088 RSI: 88002827ba54 RDI: 8217745c [30558.784056] RBP: 88007c4e7e60 R08: 0020 R09: 88007c4e7e70 [30558.784056] R10: 0637760b R11: 880011829a80 R12: 88007ab8d000 [30558.784056] R13: R14: 559b48b2dcc0 R15: ff9c [30558.784056] FS: 7f29cf0db8c0() GS:88007d20() knlGS: [30558.784056] CS: 0010 DS: ES: CR0: 80050033 [30558.784056] CR2: 7fa466d58180 CR3: 7c19 CR4: 001406f0 [30558.784056] Stack: [30558.784056] 88002827ba50 88002827ba50 8800373e70d0 559b48b2dcc0 [30558.784056] 88007c4e7e80 811194b3 88002827ba50 [30558.784056] 88007c4e7ea0 8128ddcd 880011829a80 [30558.784056] Call Trace: [30558.784056] [] cgroup_rmdir+0x23/0x40 [30558.784056] [] kernfs_iop_rmdir+0x4d/0x80 [30558.784056] [] vfs_rmdir+0xb4/0x130 [30558.784056] [] do_rmdir+0x1df/0x200 [30558.784056] [] SyS_rmdir+0x16/0x20 [30558.784056] [] entry_SYSCALL_64_fastpath+0x16/0x71 [30558.784056] Code: 74 fd 48 c7 c7 5c 74 17 82 e8 8e 72 72 00 49 8b 94 24 50 01 00 00 49 8d 8c 24 50 01 00 00 48 39 d1 48 8d 42 f0 74 18 48 8b 50 08 82 b0 01 00 00 01 48 8b 50 10 48 39 d1 48 8d 42 f0 75 e8 49 [30558.784056] RIP [] cgroup_destroy_locked+0x5f/0xf0 [30558.784056] RSP [30558.960828] ---[ end trace 7634e03ff94e8934 ]--- [30558.964811] Kernel panic - not syncing: Fatal exception in interrupt [30558.968805] Kernel Offset: disabled ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-83-generic 4.4.0-83.106 ProcVersionSignature: Ubuntu 4.4.0-83.106-generic 4.4.70 Uname: Linux 4.4.0-83-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jul 6 10:22 seq crw-rw 1 root audio 116, 33 Jul 6 10:22 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.6 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Thu Jul 6 10:27:37 2017 Ec2AMI: ami-34100353 Ec2AMIManifest: (unknown) Ec2AvailabilityZone: ap-northeast-1c Ec2InstanceType: t2.small Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU PciMultimedia: ProcEnviron: SHELL=/bin/bash TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-83-generic root=UUID=f76be987-234f-4071-87d4-06318cfc2135 ro cgroup_enable=memory swapaccount=1 console=tty1 console=ttyS0 RelatedPackageVersions: linux-restricted-modules-4.4.0-83-generic N/A linux-backports-modules-4.4.0-83-generic N/A linux-firmwareN/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) WifiSyslog: dmi.bios.date: 02/16/2017 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd02/16/2017:svnXen:pnHVMd
