[Kernel-packages] [Bug 1702910] Re: System freeze after in6_dev_finish_destroy errors
Can folks affected by this bug test the kernel located at: http://kernel.ubuntu.com/~jsalisbury/lp1702910/ To test the kernel, please install both the linux-image and linux-image- extra .deb packages. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1702910 Title: System freeze after in6_dev_finish_destroy errors Status in linux package in Ubuntu: Triaged Status in linux source package in Xenial: Triaged Bug description: After the upgrade to the `4.4.0-83-generic #106~14.04.1-Ubuntu` kernel we are seeing system hangs with the following logs. Prior to this upgrade there were no crashes or kernel traces of this sort. We have ruled out physical hardware issues by moving the VM to another physical host. In the logs there's reference to docker instances on the host. This results in veth devices being brought up and torn down frequently, which is likely related. As a possible workaround we have disabled IPv6 for now. ``` kernel: [337352.274907] vethf317e0d: renamed from eth0 kernel: [337352.386446] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.393444] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.395408] device veth160ed4a left promiscuous mode kernel: [337352.395417] docker0: port 1(veth160ed4a) entered disabled state kernel: [337359.301576] vethee2599d: renamed from eth0 kernel: [337359.339582] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.343059] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.344987] device veth3100aab left promiscuous mode kernel: [337359.344997] docker0: port 2(veth3100aab) entered disabled state kernel: [337475.398358] [ cut here ] kernel: [337475.398370] WARNING: CPU: 3 PID: 23671 at /build/linux-lts-xenial-ep3zLI/linux-lts-xenial-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_f kernel: [337475.398372] Modules linked in: btrfs xor raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c veth ip6table_filter ip6_tab comment ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_fil filter bridge stp llc aufs tcp_diag inet_diag coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel ppdev aes_x86_64 lrw gf128mul input_leds serio_raw drm_kms_helper parport_pc drm lp fb_sys_fops syscopyarea sysfillrect sysimgblt vmw_vmci parport shpchp 8250_fintek i2c_piix4 kernel: [337475.398416] CPU: 3 PID: 23671 Comm: kworker/u8:0 Not tainted 4.4.0-83-generic #106~14.04.1-Ubuntu kernel: [337475.398418] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015 kernel: [337475.398423] Workqueue: netns cleanup_net kernel: [337475.398424] 8801b197fbb8 813ddefc kernel: [337475.398427] 81d7aa10 8801b197fbf0 8107ea36 88016f97d400 kernel: [337475.398428] 88016f97e800 0006 8801b197fcc0 kernel: [337475.398430] Call Trace: kernel: [337475.398435] [] dump_stack+0x63/0x87 kernel: [337475.398440] [] warn_slowpath_common+0x86/0xc0 kernel: [337475.398442] [] warn_slowpath_null+0x1a/0x20 kernel: [337475.398444] [] in6_dev_finish_destroy+0x6b/0xb0 kernel: [337475.398447] [] ip6_route_dev_notify+0x110/0x130 kernel: [337475.398450] [] notifier_call_chain+0x49/0x70 kernel: [337475.398452] [] raw_notifier_call_chain+0x16/0x20 kernel: [337475.398455] [] call_netdevice_notifiers_info+0x35/0x60 kernel: [337475.398458] [] netdev_run_todo+0x157/0x300 kernel: [337475.398461] [] ? rollback_registered_many+0x22e/0x2e0 kernel: [337475.398463] [] rtnl_unlock+0xe/0x10 kernel: [337475.398464] [] default_device_exit_batch+0x138/0x150 kernel: [337475.398468] [] ? __wake_up_sync+0x20/0x20 kernel: [337475.398470] [] ops_exit_list.isra.4+0x52/0x60 kernel: [337475.398472] [] cleanup_net+0x1b3/0x280 kernel: [337475.398476] [] process_one_work+0x150/0x3f0 kernel: [337475.398478] [] worker_thread+0x11a/0x470 kernel: [337475.398481] [] ? __schedule+0x359/0x980 kernel: [337475.398483] [] ? rescuer_thread+0x310/0x310 kernel: [337475.398484] [] kthread+0xd6/0xf0 kernel: [337475.398486] [] ? kthread_park+0x60/0x60 kernel: [337475.398488] [] ret_from_fork+0x3f/0x70 kernel: [337475.398490] [] ? kthread_park+0x60/0x60 kernel: [337475.398491] ---[ end trace 6d63274dc9a3dbec ]--- kernel: [337475.398492] [ cut here ] ``` ProblemType: Bug ApportVersion: 2.14.1-0ubuntu3.24 Architecture: amd64 Date: Fri Jul 7 13:59:36 2017 Dependencies: adduser 3.113+nmu3ubuntu3 apt-utils 1.0.1ubuntu2.17 base-passwd 3.5.33 busybox-initramfs 1:1.21.0-1ubuntu1 coreutils 8.21-1ubuntu5.4
[Kernel-packages] [Bug 1702910] Re: System freeze after in6_dev_finish_destroy errors
** Changed in: linux (Ubuntu) Status: Incomplete => Triaged ** Also affects: linux (Ubuntu Xenial) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Xenial) Status: New => Triaged ** Changed in: linux (Ubuntu Xenial) Importance: Undecided => High -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1702910 Title: System freeze after in6_dev_finish_destroy errors Status in linux package in Ubuntu: Triaged Status in linux source package in Xenial: Triaged Bug description: After the upgrade to the `4.4.0-83-generic #106~14.04.1-Ubuntu` kernel we are seeing system hangs with the following logs. Prior to this upgrade there were no crashes or kernel traces of this sort. We have ruled out physical hardware issues by moving the VM to another physical host. In the logs there's reference to docker instances on the host. This results in veth devices being brought up and torn down frequently, which is likely related. As a possible workaround we have disabled IPv6 for now. ``` kernel: [337352.274907] vethf317e0d: renamed from eth0 kernel: [337352.386446] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.393444] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.395408] device veth160ed4a left promiscuous mode kernel: [337352.395417] docker0: port 1(veth160ed4a) entered disabled state kernel: [337359.301576] vethee2599d: renamed from eth0 kernel: [337359.339582] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.343059] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.344987] device veth3100aab left promiscuous mode kernel: [337359.344997] docker0: port 2(veth3100aab) entered disabled state kernel: [337475.398358] [ cut here ] kernel: [337475.398370] WARNING: CPU: 3 PID: 23671 at /build/linux-lts-xenial-ep3zLI/linux-lts-xenial-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_f kernel: [337475.398372] Modules linked in: btrfs xor raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c veth ip6table_filter ip6_tab comment ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_fil filter bridge stp llc aufs tcp_diag inet_diag coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel ppdev aes_x86_64 lrw gf128mul input_leds serio_raw drm_kms_helper parport_pc drm lp fb_sys_fops syscopyarea sysfillrect sysimgblt vmw_vmci parport shpchp 8250_fintek i2c_piix4 kernel: [337475.398416] CPU: 3 PID: 23671 Comm: kworker/u8:0 Not tainted 4.4.0-83-generic #106~14.04.1-Ubuntu kernel: [337475.398418] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015 kernel: [337475.398423] Workqueue: netns cleanup_net kernel: [337475.398424] 8801b197fbb8 813ddefc kernel: [337475.398427] 81d7aa10 8801b197fbf0 8107ea36 88016f97d400 kernel: [337475.398428] 88016f97e800 0006 8801b197fcc0 kernel: [337475.398430] Call Trace: kernel: [337475.398435] [] dump_stack+0x63/0x87 kernel: [337475.398440] [] warn_slowpath_common+0x86/0xc0 kernel: [337475.398442] [] warn_slowpath_null+0x1a/0x20 kernel: [337475.398444] [] in6_dev_finish_destroy+0x6b/0xb0 kernel: [337475.398447] [] ip6_route_dev_notify+0x110/0x130 kernel: [337475.398450] [] notifier_call_chain+0x49/0x70 kernel: [337475.398452] [] raw_notifier_call_chain+0x16/0x20 kernel: [337475.398455] [] call_netdevice_notifiers_info+0x35/0x60 kernel: [337475.398458] [] netdev_run_todo+0x157/0x300 kernel: [337475.398461] [] ? rollback_registered_many+0x22e/0x2e0 kernel: [337475.398463] [] rtnl_unlock+0xe/0x10 kernel: [337475.398464] [] default_device_exit_batch+0x138/0x150 kernel: [337475.398468] [] ? __wake_up_sync+0x20/0x20 kernel: [337475.398470] [] ops_exit_list.isra.4+0x52/0x60 kernel: [337475.398472] [] cleanup_net+0x1b3/0x280 kernel: [337475.398476] [] process_one_work+0x150/0x3f0 kernel: [337475.398478] [] worker_thread+0x11a/0x470 kernel: [337475.398481] [] ? __schedule+0x359/0x980 kernel: [337475.398483] [] ? rescuer_thread+0x310/0x310 kernel: [337475.398484] [] kthread+0xd6/0xf0 kernel: [337475.398486] [] ? kthread_park+0x60/0x60 kernel: [337475.398488] [] ret_from_fork+0x3f/0x70 kernel: [337475.398490] [] ? kthread_park+0x60/0x60 kernel: [337475.398491] ---[ end trace 6d63274dc9a3dbec ]--- kernel: [337475.398492] [ cut here ] ``` ProblemType: Bug ApportVersion: 2.14.1-0ubuntu3.24 Architecture: amd64 Date: Fri Jul 7 13:59:36 2017 Dependencies: adduser 3.113+nmu3ubuntu3 apt-utils 1.0.1ubuntu2.17
[Kernel-packages] [Bug 1702910] Re: System freeze after in6_dev_finish_destroy errors
There's a discussion here possibly about the same problem: https://patchwork.ozlabs.org/patch/801533/ And they referred to this patch: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux- stable.git/commit/?id=242d3a49a2a1a71d8eb9f953db1bcaa9d698ce00 The changes look like they line up with the call trace order, but I could be wrong. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1702910 Title: System freeze after in6_dev_finish_destroy errors Status in linux package in Ubuntu: Incomplete Bug description: After the upgrade to the `4.4.0-83-generic #106~14.04.1-Ubuntu` kernel we are seeing system hangs with the following logs. Prior to this upgrade there were no crashes or kernel traces of this sort. We have ruled out physical hardware issues by moving the VM to another physical host. In the logs there's reference to docker instances on the host. This results in veth devices being brought up and torn down frequently, which is likely related. As a possible workaround we have disabled IPv6 for now. ``` kernel: [337352.274907] vethf317e0d: renamed from eth0 kernel: [337352.386446] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.393444] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.395408] device veth160ed4a left promiscuous mode kernel: [337352.395417] docker0: port 1(veth160ed4a) entered disabled state kernel: [337359.301576] vethee2599d: renamed from eth0 kernel: [337359.339582] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.343059] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.344987] device veth3100aab left promiscuous mode kernel: [337359.344997] docker0: port 2(veth3100aab) entered disabled state kernel: [337475.398358] [ cut here ] kernel: [337475.398370] WARNING: CPU: 3 PID: 23671 at /build/linux-lts-xenial-ep3zLI/linux-lts-xenial-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_f kernel: [337475.398372] Modules linked in: btrfs xor raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c veth ip6table_filter ip6_tab comment ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_fil filter bridge stp llc aufs tcp_diag inet_diag coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel ppdev aes_x86_64 lrw gf128mul input_leds serio_raw drm_kms_helper parport_pc drm lp fb_sys_fops syscopyarea sysfillrect sysimgblt vmw_vmci parport shpchp 8250_fintek i2c_piix4 kernel: [337475.398416] CPU: 3 PID: 23671 Comm: kworker/u8:0 Not tainted 4.4.0-83-generic #106~14.04.1-Ubuntu kernel: [337475.398418] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015 kernel: [337475.398423] Workqueue: netns cleanup_net kernel: [337475.398424] 8801b197fbb8 813ddefc kernel: [337475.398427] 81d7aa10 8801b197fbf0 8107ea36 88016f97d400 kernel: [337475.398428] 88016f97e800 0006 8801b197fcc0 kernel: [337475.398430] Call Trace: kernel: [337475.398435] [] dump_stack+0x63/0x87 kernel: [337475.398440] [] warn_slowpath_common+0x86/0xc0 kernel: [337475.398442] [] warn_slowpath_null+0x1a/0x20 kernel: [337475.398444] [] in6_dev_finish_destroy+0x6b/0xb0 kernel: [337475.398447] [] ip6_route_dev_notify+0x110/0x130 kernel: [337475.398450] [] notifier_call_chain+0x49/0x70 kernel: [337475.398452] [] raw_notifier_call_chain+0x16/0x20 kernel: [337475.398455] [] call_netdevice_notifiers_info+0x35/0x60 kernel: [337475.398458] [] netdev_run_todo+0x157/0x300 kernel: [337475.398461] [] ? rollback_registered_many+0x22e/0x2e0 kernel: [337475.398463] [] rtnl_unlock+0xe/0x10 kernel: [337475.398464] [] default_device_exit_batch+0x138/0x150 kernel: [337475.398468] [] ? __wake_up_sync+0x20/0x20 kernel: [337475.398470] [] ops_exit_list.isra.4+0x52/0x60 kernel: [337475.398472] [] cleanup_net+0x1b3/0x280 kernel: [337475.398476] [] process_one_work+0x150/0x3f0 kernel: [337475.398478] [] worker_thread+0x11a/0x470 kernel: [337475.398481] [] ? __schedule+0x359/0x980 kernel: [337475.398483] [] ? rescuer_thread+0x310/0x310 kernel: [337475.398484] [] kthread+0xd6/0xf0 kernel: [337475.398486] [] ? kthread_park+0x60/0x60 kernel: [337475.398488] [] ret_from_fork+0x3f/0x70 kernel: [337475.398490] [] ? kthread_park+0x60/0x60 kernel: [337475.398491] ---[ end trace 6d63274dc9a3dbec ]--- kernel: [337475.398492] [ cut here ] ``` ProblemType: Bug ApportVersion: 2.14.1-0ubuntu3.24 Architecture: amd64 Date: Fri Jul 7 13:59:36 2017 Dependencies: adduser 3.113+nmu3ubuntu3 apt-utils
[Kernel-packages] [Bug 1702910] Re: System freeze after in6_dev_finish_destroy errors
Were you able to reproduce on a 4.12 kernel? Maybe even a 4.14 kernel (even though it's just the first rc that's been cut, it is the next lts). 4.9 is another lts kernel series that could be tested. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1702910 Title: System freeze after in6_dev_finish_destroy errors Status in linux package in Ubuntu: Incomplete Bug description: After the upgrade to the `4.4.0-83-generic #106~14.04.1-Ubuntu` kernel we are seeing system hangs with the following logs. Prior to this upgrade there were no crashes or kernel traces of this sort. We have ruled out physical hardware issues by moving the VM to another physical host. In the logs there's reference to docker instances on the host. This results in veth devices being brought up and torn down frequently, which is likely related. As a possible workaround we have disabled IPv6 for now. ``` kernel: [337352.274907] vethf317e0d: renamed from eth0 kernel: [337352.386446] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.393444] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.395408] device veth160ed4a left promiscuous mode kernel: [337352.395417] docker0: port 1(veth160ed4a) entered disabled state kernel: [337359.301576] vethee2599d: renamed from eth0 kernel: [337359.339582] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.343059] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.344987] device veth3100aab left promiscuous mode kernel: [337359.344997] docker0: port 2(veth3100aab) entered disabled state kernel: [337475.398358] [ cut here ] kernel: [337475.398370] WARNING: CPU: 3 PID: 23671 at /build/linux-lts-xenial-ep3zLI/linux-lts-xenial-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_f kernel: [337475.398372] Modules linked in: btrfs xor raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c veth ip6table_filter ip6_tab comment ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_fil filter bridge stp llc aufs tcp_diag inet_diag coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel ppdev aes_x86_64 lrw gf128mul input_leds serio_raw drm_kms_helper parport_pc drm lp fb_sys_fops syscopyarea sysfillrect sysimgblt vmw_vmci parport shpchp 8250_fintek i2c_piix4 kernel: [337475.398416] CPU: 3 PID: 23671 Comm: kworker/u8:0 Not tainted 4.4.0-83-generic #106~14.04.1-Ubuntu kernel: [337475.398418] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015 kernel: [337475.398423] Workqueue: netns cleanup_net kernel: [337475.398424] 8801b197fbb8 813ddefc kernel: [337475.398427] 81d7aa10 8801b197fbf0 8107ea36 88016f97d400 kernel: [337475.398428] 88016f97e800 0006 8801b197fcc0 kernel: [337475.398430] Call Trace: kernel: [337475.398435] [] dump_stack+0x63/0x87 kernel: [337475.398440] [] warn_slowpath_common+0x86/0xc0 kernel: [337475.398442] [] warn_slowpath_null+0x1a/0x20 kernel: [337475.398444] [] in6_dev_finish_destroy+0x6b/0xb0 kernel: [337475.398447] [] ip6_route_dev_notify+0x110/0x130 kernel: [337475.398450] [] notifier_call_chain+0x49/0x70 kernel: [337475.398452] [] raw_notifier_call_chain+0x16/0x20 kernel: [337475.398455] [] call_netdevice_notifiers_info+0x35/0x60 kernel: [337475.398458] [] netdev_run_todo+0x157/0x300 kernel: [337475.398461] [] ? rollback_registered_many+0x22e/0x2e0 kernel: [337475.398463] [] rtnl_unlock+0xe/0x10 kernel: [337475.398464] [] default_device_exit_batch+0x138/0x150 kernel: [337475.398468] [] ? __wake_up_sync+0x20/0x20 kernel: [337475.398470] [] ops_exit_list.isra.4+0x52/0x60 kernel: [337475.398472] [] cleanup_net+0x1b3/0x280 kernel: [337475.398476] [] process_one_work+0x150/0x3f0 kernel: [337475.398478] [] worker_thread+0x11a/0x470 kernel: [337475.398481] [] ? __schedule+0x359/0x980 kernel: [337475.398483] [] ? rescuer_thread+0x310/0x310 kernel: [337475.398484] [] kthread+0xd6/0xf0 kernel: [337475.398486] [] ? kthread_park+0x60/0x60 kernel: [337475.398488] [] ret_from_fork+0x3f/0x70 kernel: [337475.398490] [] ? kthread_park+0x60/0x60 kernel: [337475.398491] ---[ end trace 6d63274dc9a3dbec ]--- kernel: [337475.398492] [ cut here ] ``` ProblemType: Bug ApportVersion: 2.14.1-0ubuntu3.24 Architecture: amd64 Date: Fri Jul 7 13:59:36 2017 Dependencies: adduser 3.113+nmu3ubuntu3 apt-utils 1.0.1ubuntu2.17 base-passwd 3.5.33 busybox-initramfs 1:1.21.0-1ubuntu1 coreutils 8.21-1ubuntu5.4 cpio 2.11+dfsg-1ubuntu1.2 dbus 1.6.18-0ubuntu4.5
[Kernel-packages] [Bug 1702910] Re: System freeze after in6_dev_finish_destroy errors
Here is additional debug information: KERNEL: /usr/lib/debug/boot/vmlinux-4.4.0-87-generic DUMPFILE: dump.201709151609 [PARTIAL DUMP] CPUS: 12 DATE: Fri Sep 15 16:05:34 2017 UPTIME: 1 days, 17:34:46 LOAD AVERAGE: 4.17, 5.02, 4.48 TASKS: 2572 NODENAME: 543231-infra02 RELEASE: 4.4.0-87-generic VERSION: #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017 MACHINE: x86_64 (3000 Mhz) MEMORY: 32 GB PANIC: "BUG: unable to handle kernel paging request at 0040" PID: 4189 COMMAND: "dnsmasq" TASK: 8808299dd400 [THREAD_INFO: 88042502c000] CPU: 11 STATE: TASK_RUNNING (PANIC) [149668.845048] WARNING: CPU: 0 PID: 11227 at /build/linux-5EyXrQ/linux-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_finish_destroy+0x6b/0xc0() [149668.845050] Modules linked in: nf_conntrack_netlink ebt_arp ebt_among xt_CT xt_mac xt_physdev xt_set ip_set_hash_net ip_set nfnetlink xt_REDIRECT nf_nat_redirect nf_conntrack_ipv6 nf_defrag_ipv6 xt_nat xt_conntrack xt_mark xt_connmark tcp_diag udp_diag inet_diag unix_diag ebtable_filter xt_comment ip6table_raw ip6table_mangle sunrpc binfmt_misc veth xt_CHECKSUM xt_tcpudp iptable_raw ipmi_ssif kvm_amd ib_iser kvm amd64_edac_mod rdma_cm iw_cm 8250_fintek edac_mce_amd irqbypass ib_cm fam15h_power edac_core ib_sa serio_raw ib_mad i2c_piix4 ipmi_si ipmi_msghandler k10temp hpilo ib_core shpchp ib_addr vhost_net vhost macvtap macvlan nbd mac_hid iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_vs iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_mangle iptable_filter ipt_REJECT nf_reject_ipv4 [149668.845092] ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_nat nf_conntrack ip_tables ip6table_filter ip6_tables ebtables x_tables dm_snapshot dm_bufio dm_multipath br_netfilter bridge 8021q garp mrp stp llc ixgbe mdio tg3 bonding autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx4_en vxlan ip6_udp_tunnel udp_tunnel crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ttm aesni_intel drm_kms_helper aes_x86_64 lrw gf128mul syscopyarea glue_helper sysfillrect ablk_helper sysimgblt fb_sys_fops cryptd drm psmouse pata_acpi igb pata_atiixp hpsa dca ahci mlx4_core ptp libahci scsi_transport_sas pps_core i2c_algo_bit fjes [149668.845132] CPU: 0 PID: 11227 Comm: kworker/u24:1 Not tainted 4.4.0-87-generic #110-Ubuntu [149668.845134] Hardware name: HP ProLiant SE4255e /, BIOS A31 01/05/2013 [149668.845138] Workqueue: netns cleanup_net [149668.845139] 0286 e218996f8a60a1fa 8802f3637ba0 813f9903 [149668.845142] 81d75978 8802f3637bd8 81081332 [149668.845144] 8806dd999400 8806dd99e800 0006 8802f3637ca8 [149668.845146] Call Trace: [149668.845151] [] dump_stack+0x63/0x90 [149668.845154] [] warn_slowpath_common+0x82/0xc0 [149668.845156] [] warn_slowpath_null+0x1a/0x20 [149668.845158] [] in6_dev_finish_destroy+0x6b/0xc0 [149668.845161] [] ip6_route_dev_notify+0x116/0x130 [149668.845163] [] notifier_call_chain+0x4a/0x70 [149668.845165] [] raw_notifier_call_chain+0x16/0x20 [149668.845168] [] call_netdevice_notifiers_info+0x35/0x60 [149668.845170] [] netdev_run_todo+0x16d/0x320 [149668.845173] [] rtnl_unlock+0xe/0x10 [149668.845175] [] default_device_exit_batch+0x147/0x170 [149668.845179] [] ? __wake_up_sync+0x20/0x20 [149668.845181] [] ops_exit_list.isra.4+0x52/0x60 [149668.845183] [] cleanup_net+0x1c2/0x2a0 [149668.845185] [] process_one_work+0x165/0x480 [149668.845188] [] worker_thread+0x4b/0x4c0 [149668.845190] [] ? process_one_work+0x480/0x480 [149668.845192] [] kthread+0xe5/0x100 [149668.845194] [] ? kthread_create_on_node+0x1e0/0x1e0 [149668.845197] [] ret_from_fork+0x3f/0x70 [149668.845199] [] ? kthread_create_on_node+0x1e0/0x1e0 [149668.845200] ---[ end trace a70573408f354326 ]--- -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1702910 Title: System freeze after in6_dev_finish_destroy errors Status in linux package in Ubuntu: Incomplete Bug description: After the upgrade to the `4.4.0-83-generic #106~14.04.1-Ubuntu` kernel we are seeing system hangs with the following logs. Prior to this upgrade there were no crashes or kernel traces of this sort. We have ruled out physical hardware issues by moving the VM to another physical host. In the logs there's reference to docker instances on the host. This results in veth devices being brought up and torn down frequently, which is likely related. As a possible workaround we have disabled IPv6 for now. ``` kernel: [337352.274907] vethf317e0d: renamed from eth0 kernel: [337352.386446] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.393444] docker0: port
[Kernel-packages] [Bug 1702910] Re: System freeze after in6_dev_finish_destroy errors
I can confirm that this also happens with 4.4.0-87-generic. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1702910 Title: System freeze after in6_dev_finish_destroy errors Status in linux package in Ubuntu: Incomplete Bug description: After the upgrade to the `4.4.0-83-generic #106~14.04.1-Ubuntu` kernel we are seeing system hangs with the following logs. Prior to this upgrade there were no crashes or kernel traces of this sort. We have ruled out physical hardware issues by moving the VM to another physical host. In the logs there's reference to docker instances on the host. This results in veth devices being brought up and torn down frequently, which is likely related. As a possible workaround we have disabled IPv6 for now. ``` kernel: [337352.274907] vethf317e0d: renamed from eth0 kernel: [337352.386446] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.393444] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.395408] device veth160ed4a left promiscuous mode kernel: [337352.395417] docker0: port 1(veth160ed4a) entered disabled state kernel: [337359.301576] vethee2599d: renamed from eth0 kernel: [337359.339582] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.343059] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.344987] device veth3100aab left promiscuous mode kernel: [337359.344997] docker0: port 2(veth3100aab) entered disabled state kernel: [337475.398358] [ cut here ] kernel: [337475.398370] WARNING: CPU: 3 PID: 23671 at /build/linux-lts-xenial-ep3zLI/linux-lts-xenial-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_f kernel: [337475.398372] Modules linked in: btrfs xor raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c veth ip6table_filter ip6_tab comment ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_fil filter bridge stp llc aufs tcp_diag inet_diag coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel ppdev aes_x86_64 lrw gf128mul input_leds serio_raw drm_kms_helper parport_pc drm lp fb_sys_fops syscopyarea sysfillrect sysimgblt vmw_vmci parport shpchp 8250_fintek i2c_piix4 kernel: [337475.398416] CPU: 3 PID: 23671 Comm: kworker/u8:0 Not tainted 4.4.0-83-generic #106~14.04.1-Ubuntu kernel: [337475.398418] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015 kernel: [337475.398423] Workqueue: netns cleanup_net kernel: [337475.398424] 8801b197fbb8 813ddefc kernel: [337475.398427] 81d7aa10 8801b197fbf0 8107ea36 88016f97d400 kernel: [337475.398428] 88016f97e800 0006 8801b197fcc0 kernel: [337475.398430] Call Trace: kernel: [337475.398435] [] dump_stack+0x63/0x87 kernel: [337475.398440] [] warn_slowpath_common+0x86/0xc0 kernel: [337475.398442] [] warn_slowpath_null+0x1a/0x20 kernel: [337475.398444] [] in6_dev_finish_destroy+0x6b/0xb0 kernel: [337475.398447] [] ip6_route_dev_notify+0x110/0x130 kernel: [337475.398450] [] notifier_call_chain+0x49/0x70 kernel: [337475.398452] [] raw_notifier_call_chain+0x16/0x20 kernel: [337475.398455] [] call_netdevice_notifiers_info+0x35/0x60 kernel: [337475.398458] [] netdev_run_todo+0x157/0x300 kernel: [337475.398461] [] ? rollback_registered_many+0x22e/0x2e0 kernel: [337475.398463] [] rtnl_unlock+0xe/0x10 kernel: [337475.398464] [] default_device_exit_batch+0x138/0x150 kernel: [337475.398468] [] ? __wake_up_sync+0x20/0x20 kernel: [337475.398470] [] ops_exit_list.isra.4+0x52/0x60 kernel: [337475.398472] [] cleanup_net+0x1b3/0x280 kernel: [337475.398476] [] process_one_work+0x150/0x3f0 kernel: [337475.398478] [] worker_thread+0x11a/0x470 kernel: [337475.398481] [] ? __schedule+0x359/0x980 kernel: [337475.398483] [] ? rescuer_thread+0x310/0x310 kernel: [337475.398484] [] kthread+0xd6/0xf0 kernel: [337475.398486] [] ? kthread_park+0x60/0x60 kernel: [337475.398488] [] ret_from_fork+0x3f/0x70 kernel: [337475.398490] [] ? kthread_park+0x60/0x60 kernel: [337475.398491] ---[ end trace 6d63274dc9a3dbec ]--- kernel: [337475.398492] [ cut here ] ``` ProblemType: Bug ApportVersion: 2.14.1-0ubuntu3.24 Architecture: amd64 Date: Fri Jul 7 13:59:36 2017 Dependencies: adduser 3.113+nmu3ubuntu3 apt-utils 1.0.1ubuntu2.17 base-passwd 3.5.33 busybox-initramfs 1:1.21.0-1ubuntu1 coreutils 8.21-1ubuntu5.4 cpio 2.11+dfsg-1ubuntu1.2 dbus 1.6.18-0ubuntu4.5 debconf 1.5.51ubuntu2 debconf-i18n 1.5.51ubuntu2 debianutils 4.4 dmsetup 2:1.02.77-6ubuntu2 dpkg 1.17.5ubuntu5.7 e2fslibs
[Kernel-packages] [Bug 1702910] Re: System freeze after in6_dev_finish_destroy errors
Can you also see if this bug happens with the latest upstream stable 4.4 kernel? It is available from: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.85/ -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1702910 Title: System freeze after in6_dev_finish_destroy errors Status in linux package in Ubuntu: Incomplete Bug description: After the upgrade to the `4.4.0-83-generic #106~14.04.1-Ubuntu` kernel we are seeing system hangs with the following logs. Prior to this upgrade there were no crashes or kernel traces of this sort. We have ruled out physical hardware issues by moving the VM to another physical host. In the logs there's reference to docker instances on the host. This results in veth devices being brought up and torn down frequently, which is likely related. As a possible workaround we have disabled IPv6 for now. ``` kernel: [337352.274907] vethf317e0d: renamed from eth0 kernel: [337352.386446] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.393444] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.395408] device veth160ed4a left promiscuous mode kernel: [337352.395417] docker0: port 1(veth160ed4a) entered disabled state kernel: [337359.301576] vethee2599d: renamed from eth0 kernel: [337359.339582] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.343059] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.344987] device veth3100aab left promiscuous mode kernel: [337359.344997] docker0: port 2(veth3100aab) entered disabled state kernel: [337475.398358] [ cut here ] kernel: [337475.398370] WARNING: CPU: 3 PID: 23671 at /build/linux-lts-xenial-ep3zLI/linux-lts-xenial-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_f kernel: [337475.398372] Modules linked in: btrfs xor raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c veth ip6table_filter ip6_tab comment ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_fil filter bridge stp llc aufs tcp_diag inet_diag coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel ppdev aes_x86_64 lrw gf128mul input_leds serio_raw drm_kms_helper parport_pc drm lp fb_sys_fops syscopyarea sysfillrect sysimgblt vmw_vmci parport shpchp 8250_fintek i2c_piix4 kernel: [337475.398416] CPU: 3 PID: 23671 Comm: kworker/u8:0 Not tainted 4.4.0-83-generic #106~14.04.1-Ubuntu kernel: [337475.398418] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015 kernel: [337475.398423] Workqueue: netns cleanup_net kernel: [337475.398424] 8801b197fbb8 813ddefc kernel: [337475.398427] 81d7aa10 8801b197fbf0 8107ea36 88016f97d400 kernel: [337475.398428] 88016f97e800 0006 8801b197fcc0 kernel: [337475.398430] Call Trace: kernel: [337475.398435] [] dump_stack+0x63/0x87 kernel: [337475.398440] [] warn_slowpath_common+0x86/0xc0 kernel: [337475.398442] [] warn_slowpath_null+0x1a/0x20 kernel: [337475.398444] [] in6_dev_finish_destroy+0x6b/0xb0 kernel: [337475.398447] [] ip6_route_dev_notify+0x110/0x130 kernel: [337475.398450] [] notifier_call_chain+0x49/0x70 kernel: [337475.398452] [] raw_notifier_call_chain+0x16/0x20 kernel: [337475.398455] [] call_netdevice_notifiers_info+0x35/0x60 kernel: [337475.398458] [] netdev_run_todo+0x157/0x300 kernel: [337475.398461] [] ? rollback_registered_many+0x22e/0x2e0 kernel: [337475.398463] [] rtnl_unlock+0xe/0x10 kernel: [337475.398464] [] default_device_exit_batch+0x138/0x150 kernel: [337475.398468] [] ? __wake_up_sync+0x20/0x20 kernel: [337475.398470] [] ops_exit_list.isra.4+0x52/0x60 kernel: [337475.398472] [] cleanup_net+0x1b3/0x280 kernel: [337475.398476] [] process_one_work+0x150/0x3f0 kernel: [337475.398478] [] worker_thread+0x11a/0x470 kernel: [337475.398481] [] ? __schedule+0x359/0x980 kernel: [337475.398483] [] ? rescuer_thread+0x310/0x310 kernel: [337475.398484] [] kthread+0xd6/0xf0 kernel: [337475.398486] [] ? kthread_park+0x60/0x60 kernel: [337475.398488] [] ret_from_fork+0x3f/0x70 kernel: [337475.398490] [] ? kthread_park+0x60/0x60 kernel: [337475.398491] ---[ end trace 6d63274dc9a3dbec ]--- kernel: [337475.398492] [ cut here ] ``` ProblemType: Bug ApportVersion: 2.14.1-0ubuntu3.24 Architecture: amd64 Date: Fri Jul 7 13:59:36 2017 Dependencies: adduser 3.113+nmu3ubuntu3 apt-utils 1.0.1ubuntu2.17 base-passwd 3.5.33 busybox-initramfs 1:1.21.0-1ubuntu1 coreutils 8.21-1ubuntu5.4 cpio 2.11+dfsg-1ubuntu1.2 dbus 1.6.18-0ubuntu4.5 debconf 1.5.51ubuntu2 debconf-i18n
[Kernel-packages] [Bug 1702910] Re: System freeze after in6_dev_finish_destroy errors
We've successfully reproduced the error on a newly built instance. We'll proceed to run the mainline kernel on the same instance. Worth noting that it took 441529 seconds (5.1 days) since boot for this to manifest in this case. On our previous instance the time to failure was in the order of a hours to days. [441529.300238] [ cut here ] [441529.300288] WARNING: CPU: 1 PID: 10409 at /build/linux-lts-xenial-N0BD4p/linux-lts-xenial-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_finish_destroy+0x6b/0xb0() [441529.300359] CPU: 1 PID: 10409 Comm: kworker/u4:0 Not tainted 4.4.0-87-generic #110~14.04.1-Ubuntu [441529.300361] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015 [441529.300374] Workqueue: netns cleanup_net [441529.300376] 880088ddbbb8 813de2dc [441529.300378] 81d7aa60 880088ddbbf0 8107ea46 880139e5a400 [441529.300380] 0006 880088ddbcc0 [441529.300382] Call Trace: [441529.300398] [] dump_stack+0x63/0x87 [441529.300413] [] warn_slowpath_common+0x86/0xc0 [441529.300415] [] warn_slowpath_null+0x1a/0x20 [441529.300417] [] in6_dev_finish_destroy+0x6b/0xb0 [441529.300421] [] ip6_route_dev_notify+0x110/0x130 [441529.300425] [] notifier_call_chain+0x49/0x70 [441529.300427] [] raw_notifier_call_chain+0x16/0x20 [441529.300429] [] call_netdevice_notifiers_info+0x35/0x60 [441529.300432] [] netdev_run_todo+0x157/0x300 [441529.300436] [] ? rollback_registered_many+0x22e/0x2e0 [441529.300440] [] rtnl_unlock+0xe/0x10 [441529.300442] [] default_device_exit_batch+0x138/0x150 [441529.300452] [] ? __wake_up_sync+0x20/0x20 [441529.300454] [] ops_exit_list.isra.4+0x52/0x60 [441529.300456] [] cleanup_net+0x1b3/0x280 [441529.300461] [] process_one_work+0x150/0x3f0 [441529.300463] [] worker_thread+0x11a/0x470 [441529.300471] [] ? __schedule+0x359/0x980 [441529.300473] [] ? rescuer_thread+0x310/0x310 [441529.300475] [] kthread+0xd6/0xf0 [441529.300477] [] ? do_exit+0x655/0xae0 [441529.300479] [] ? kthread_park+0x60/0x60 [441529.300481] [] ret_from_fork+0x3f/0x70 [441529.300483] [] ? kthread_park+0x60/0x60 [441529.300484] ---[ end trace 5e09761f8674a0af ]--- [441529.300485] [ cut here ] [441529.300487] WARNING: CPU: 1 PID: 10409 at /build/linux-lts-xenial-N0BD4p/linux-lts-xenial-4.4.0/net/ipv6/addrconf_core.c:160 in6_dev_finish_destroy+0xa4/0xb0() [441529.300510] CPU: 1 PID: 10409 Comm: kworker/u4:0 Tainted: GW 4.4.0-87-generic #110~14.04.1-Ubuntu [441529.300511] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015 [441529.300513] Workqueue: netns cleanup_net [441529.300514] 880088ddbbb8 813de2dc [441529.300515] 81d7aa60 880088ddbbf0 8107ea46 880139e5a400 [441529.300517] 0006 880088ddbcc0 [441529.300518] Call Trace: [441529.300520] [] dump_stack+0x63/0x87 [441529.300522] [] warn_slowpath_common+0x86/0xc0 [441529.300524] [] warn_slowpath_null+0x1a/0x20 [441529.300526] [] in6_dev_finish_destroy+0xa4/0xb0 [441529.300528] [] ip6_route_dev_notify+0x110/0x130 [441529.300529] [] notifier_call_chain+0x49/0x70 [441529.300531] [] raw_notifier_call_chain+0x16/0x20 [441529.300533] [] call_netdevice_notifiers_info+0x35/0x60 [441529.300535] [] netdev_run_todo+0x157/0x300 [441529.300538] [] ? rollback_registered_many+0x22e/0x2e0 [441529.300540] [] rtnl_unlock+0xe/0x10 [441529.300543] [] default_device_exit_batch+0x138/0x150 [441529.300546] [] ? __wake_up_sync+0x20/0x20 [441529.300549] [] ops_exit_list.isra.4+0x52/0x60 [441529.300552] [] cleanup_net+0x1b3/0x280 [441529.300555] [] process_one_work+0x150/0x3f0 [441529.300558] [] worker_thread+0x11a/0x470 [441529.300561] [] ? __schedule+0x359/0x980 [441529.300563] [] ? rescuer_thread+0x310/0x310 [441529.300564] [] kthread+0xd6/0xf0 [441529.300566] [] ? do_exit+0x655/0xae0 [441529.300568] [] ? kthread_park+0x60/0x60 [441529.300570] [] ret_from_fork+0x3f/0x70 [441529.300571] [] ? kthread_park+0x60/0x60 [441529.300573] ---[ end trace 5e09761f8674a0b0 ]--- [441529.300775] PGD b97cd067 PUD b97ad067 PMD 0 [441529.300877] Oops: [#1] SMP [441529.301242] CPU: 1 PID: 10409 Comm: kworker/u4:0 Tainted: GW 4.4.0-87-generic #110~14.04.1-Ubuntu [441529.301281] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015 [441529.302097] Workqueue: netns cleanup_net [441529.302114] task: 880057de0e00 ti: 880088dd8000 task.ti: 880088dd8000 [441529.302143] RIP: 0010:[] [] in6_dev_finish_destroy+0x2d/0xb0 [441529.328032] RSP: 0018:880088ddbc10 EFLAGS: 00010246 [441529.328064] RAX: 0024 RBX: 880139e5a400 RCX: 0006 [441529.328092]
[Kernel-packages] [Bug 1702910] Re: System freeze after in6_dev_finish_destroy errors
I can confirm the bug: Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320092] [ cut here ] Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320100] WARNING: CPU: 1 PID: 42381 at /build/linux-0uniEn/linux-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_finish_destroy+0x6b/0xc0() Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320102] Modules linked in: cmac arc4 md4 nls_utf8 cifs fscache xt_statistic sch_htb ebt_ip ebtable_filter xt_nat xt_recent ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_mark xt_comment veth ebtable_nat ebtables ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc overlay i2c_piix4 8250_fintek input_leds joydev mac_hid serio_raw ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr sunrpc iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic crct10dif_pclmul crc32_pclmul ghash_clmulni_intel hv_netvsc hv_utils hid_hyperv hv_storvsc hid scsi_transport_fc hyperv_keyboard aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_he lper hyperv_fb cryptd psmouse pata_acpi hv_vmbus floppy fjes Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320149] CPU: 1 PID: 42381 Comm: kworker/u128:3 Not tainted 4.4.0-83-generic #106-Ubuntu Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320151] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090006 05/23/2012 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320155] Workqueue: netns cleanup_net Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320156] 0286 174b4ae5 880686863ba0 813f9513 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320158] 81d75940 880686863bd8 81081322 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320160] 88048ff82800 88048ff8 0006 880686863ca8 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320162] Call Trace: Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320166] [] dump_stack+0x63/0x90 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320170] [] warn_slowpath_common+0x82/0xc0 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320172] [] warn_slowpath_null+0x1a/0x20 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320173] [] in6_dev_finish_destroy+0x6b/0xc0 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320176] [] ip6_route_dev_notify+0x116/0x130 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320178] [] notifier_call_chain+0x4a/0x70 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320180] [] raw_notifier_call_chain+0x16/0x20 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320182] [] call_netdevice_notifiers_info+0x35/0x60 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320185] [] netdev_run_todo+0x16d/0x320 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320187] [] rtnl_unlock+0xe/0x10 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320189] [] default_device_exit_batch+0x147/0x170 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320193] [] ? __wake_up_sync+0x20/0x20 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320195] [] ops_exit_list.isra.4+0x52/0x60 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320196] [] cleanup_net+0x1c2/0x2a0 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320199] [] process_one_work+0x165/0x480 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320201] [] worker_thread+0x4b/0x4c0 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320202] [] ? process_one_work+0x480/0x480 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320204] [] kthread+0xe5/0x100 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320205] [] ? kthread_create_on_node+0x1e0/0x1e0 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320209] [] ret_from_fork+0x3f/0x70 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320210] [] ? kthread_create_on_node+0x1e0/0x1e0 Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320211] ---[ end trace b549322d7d3fedbf ]--- Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320212] [ cut here ] Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320214] WARNING: CPU: 1 PID: 42381 at /build/linux-0uniEn/linux-4.4.0/net/ipv6/addrconf_core.c:160 in6_dev_finish_destroy+0x83/0xc0() Jul 12 16:45:39 k8s-agent-C5675ACA-0 kernel: [117514.320214] Modules linked in: cmac arc4 md4 nls_utf8 cifs fscache xt_statistic sch_htb ebt_ip ebtable_filter xt_nat xt_recent ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_mark xt_comment veth ebtable_nat ebtables ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
[Kernel-packages] [Bug 1702910] Re: System freeze after in6_dev_finish_destroy errors
There is a similar stack trace on https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1702665. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1702910 Title: System freeze after in6_dev_finish_destroy errors Status in linux package in Ubuntu: Incomplete Bug description: After the upgrade to the `4.4.0-83-generic #106~14.04.1-Ubuntu` kernel we are seeing system hangs with the following logs. Prior to this upgrade there were no crashes or kernel traces of this sort. We have ruled out physical hardware issues by moving the VM to another physical host. In the logs there's reference to docker instances on the host. This results in veth devices being brought up and torn down frequently, which is likely related. As a possible workaround we have disabled IPv6 for now. ``` kernel: [337352.274907] vethf317e0d: renamed from eth0 kernel: [337352.386446] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.393444] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.395408] device veth160ed4a left promiscuous mode kernel: [337352.395417] docker0: port 1(veth160ed4a) entered disabled state kernel: [337359.301576] vethee2599d: renamed from eth0 kernel: [337359.339582] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.343059] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.344987] device veth3100aab left promiscuous mode kernel: [337359.344997] docker0: port 2(veth3100aab) entered disabled state kernel: [337475.398358] [ cut here ] kernel: [337475.398370] WARNING: CPU: 3 PID: 23671 at /build/linux-lts-xenial-ep3zLI/linux-lts-xenial-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_f kernel: [337475.398372] Modules linked in: btrfs xor raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c veth ip6table_filter ip6_tab comment ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_fil filter bridge stp llc aufs tcp_diag inet_diag coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel ppdev aes_x86_64 lrw gf128mul input_leds serio_raw drm_kms_helper parport_pc drm lp fb_sys_fops syscopyarea sysfillrect sysimgblt vmw_vmci parport shpchp 8250_fintek i2c_piix4 kernel: [337475.398416] CPU: 3 PID: 23671 Comm: kworker/u8:0 Not tainted 4.4.0-83-generic #106~14.04.1-Ubuntu kernel: [337475.398418] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015 kernel: [337475.398423] Workqueue: netns cleanup_net kernel: [337475.398424] 8801b197fbb8 813ddefc kernel: [337475.398427] 81d7aa10 8801b197fbf0 8107ea36 88016f97d400 kernel: [337475.398428] 88016f97e800 0006 8801b197fcc0 kernel: [337475.398430] Call Trace: kernel: [337475.398435] [] dump_stack+0x63/0x87 kernel: [337475.398440] [] warn_slowpath_common+0x86/0xc0 kernel: [337475.398442] [] warn_slowpath_null+0x1a/0x20 kernel: [337475.398444] [] in6_dev_finish_destroy+0x6b/0xb0 kernel: [337475.398447] [] ip6_route_dev_notify+0x110/0x130 kernel: [337475.398450] [] notifier_call_chain+0x49/0x70 kernel: [337475.398452] [] raw_notifier_call_chain+0x16/0x20 kernel: [337475.398455] [] call_netdevice_notifiers_info+0x35/0x60 kernel: [337475.398458] [] netdev_run_todo+0x157/0x300 kernel: [337475.398461] [] ? rollback_registered_many+0x22e/0x2e0 kernel: [337475.398463] [] rtnl_unlock+0xe/0x10 kernel: [337475.398464] [] default_device_exit_batch+0x138/0x150 kernel: [337475.398468] [] ? __wake_up_sync+0x20/0x20 kernel: [337475.398470] [] ops_exit_list.isra.4+0x52/0x60 kernel: [337475.398472] [] cleanup_net+0x1b3/0x280 kernel: [337475.398476] [] process_one_work+0x150/0x3f0 kernel: [337475.398478] [] worker_thread+0x11a/0x470 kernel: [337475.398481] [] ? __schedule+0x359/0x980 kernel: [337475.398483] [] ? rescuer_thread+0x310/0x310 kernel: [337475.398484] [] kthread+0xd6/0xf0 kernel: [337475.398486] [] ? kthread_park+0x60/0x60 kernel: [337475.398488] [] ret_from_fork+0x3f/0x70 kernel: [337475.398490] [] ? kthread_park+0x60/0x60 kernel: [337475.398491] ---[ end trace 6d63274dc9a3dbec ]--- kernel: [337475.398492] [ cut here ] ``` ProblemType: Bug ApportVersion: 2.14.1-0ubuntu3.24 Architecture: amd64 Date: Fri Jul 7 13:59:36 2017 Dependencies: adduser 3.113+nmu3ubuntu3 apt-utils 1.0.1ubuntu2.17 base-passwd 3.5.33 busybox-initramfs 1:1.21.0-1ubuntu1 coreutils 8.21-1ubuntu5.4 cpio 2.11+dfsg-1ubuntu1.2 dbus 1.6.18-0ubuntu4.5 debconf 1.5.51ubuntu2 debconf-i18n 1.5.51ubuntu2 debianutils 4.4 dmsetup 2:1.02.77-6ubuntu2
[Kernel-packages] [Bug 1702910] Re: System freeze after in6_dev_finish_destroy errors
We're aiming to reproduce this issue on another instance, then will switch to the mainline kernel to gather the info you need. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1702910 Title: System freeze after in6_dev_finish_destroy errors Status in linux package in Ubuntu: Incomplete Bug description: After the upgrade to the `4.4.0-83-generic #106~14.04.1-Ubuntu` kernel we are seeing system hangs with the following logs. Prior to this upgrade there were no crashes or kernel traces of this sort. We have ruled out physical hardware issues by moving the VM to another physical host. In the logs there's reference to docker instances on the host. This results in veth devices being brought up and torn down frequently, which is likely related. As a possible workaround we have disabled IPv6 for now. ``` kernel: [337352.274907] vethf317e0d: renamed from eth0 kernel: [337352.386446] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.393444] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.395408] device veth160ed4a left promiscuous mode kernel: [337352.395417] docker0: port 1(veth160ed4a) entered disabled state kernel: [337359.301576] vethee2599d: renamed from eth0 kernel: [337359.339582] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.343059] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.344987] device veth3100aab left promiscuous mode kernel: [337359.344997] docker0: port 2(veth3100aab) entered disabled state kernel: [337475.398358] [ cut here ] kernel: [337475.398370] WARNING: CPU: 3 PID: 23671 at /build/linux-lts-xenial-ep3zLI/linux-lts-xenial-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_f kernel: [337475.398372] Modules linked in: btrfs xor raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c veth ip6table_filter ip6_tab comment ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_fil filter bridge stp llc aufs tcp_diag inet_diag coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel ppdev aes_x86_64 lrw gf128mul input_leds serio_raw drm_kms_helper parport_pc drm lp fb_sys_fops syscopyarea sysfillrect sysimgblt vmw_vmci parport shpchp 8250_fintek i2c_piix4 kernel: [337475.398416] CPU: 3 PID: 23671 Comm: kworker/u8:0 Not tainted 4.4.0-83-generic #106~14.04.1-Ubuntu kernel: [337475.398418] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015 kernel: [337475.398423] Workqueue: netns cleanup_net kernel: [337475.398424] 8801b197fbb8 813ddefc kernel: [337475.398427] 81d7aa10 8801b197fbf0 8107ea36 88016f97d400 kernel: [337475.398428] 88016f97e800 0006 8801b197fcc0 kernel: [337475.398430] Call Trace: kernel: [337475.398435] [] dump_stack+0x63/0x87 kernel: [337475.398440] [] warn_slowpath_common+0x86/0xc0 kernel: [337475.398442] [] warn_slowpath_null+0x1a/0x20 kernel: [337475.398444] [] in6_dev_finish_destroy+0x6b/0xb0 kernel: [337475.398447] [] ip6_route_dev_notify+0x110/0x130 kernel: [337475.398450] [] notifier_call_chain+0x49/0x70 kernel: [337475.398452] [] raw_notifier_call_chain+0x16/0x20 kernel: [337475.398455] [] call_netdevice_notifiers_info+0x35/0x60 kernel: [337475.398458] [] netdev_run_todo+0x157/0x300 kernel: [337475.398461] [] ? rollback_registered_many+0x22e/0x2e0 kernel: [337475.398463] [] rtnl_unlock+0xe/0x10 kernel: [337475.398464] [] default_device_exit_batch+0x138/0x150 kernel: [337475.398468] [] ? __wake_up_sync+0x20/0x20 kernel: [337475.398470] [] ops_exit_list.isra.4+0x52/0x60 kernel: [337475.398472] [] cleanup_net+0x1b3/0x280 kernel: [337475.398476] [] process_one_work+0x150/0x3f0 kernel: [337475.398478] [] worker_thread+0x11a/0x470 kernel: [337475.398481] [] ? __schedule+0x359/0x980 kernel: [337475.398483] [] ? rescuer_thread+0x310/0x310 kernel: [337475.398484] [] kthread+0xd6/0xf0 kernel: [337475.398486] [] ? kthread_park+0x60/0x60 kernel: [337475.398488] [] ret_from_fork+0x3f/0x70 kernel: [337475.398490] [] ? kthread_park+0x60/0x60 kernel: [337475.398491] ---[ end trace 6d63274dc9a3dbec ]--- kernel: [337475.398492] [ cut here ] ``` ProblemType: Bug ApportVersion: 2.14.1-0ubuntu3.24 Architecture: amd64 Date: Fri Jul 7 13:59:36 2017 Dependencies: adduser 3.113+nmu3ubuntu3 apt-utils 1.0.1ubuntu2.17 base-passwd 3.5.33 busybox-initramfs 1:1.21.0-1ubuntu1 coreutils 8.21-1ubuntu5.4 cpio 2.11+dfsg-1ubuntu1.2 dbus 1.6.18-0ubuntu4.5 debconf 1.5.51ubuntu2 debconf-i18n 1.5.51ubuntu2 debianutils 4.4
[Kernel-packages] [Bug 1702910] Re: System freeze after in6_dev_finish_destroy errors
Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.12 kernel[0]. If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'. If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'. Once testing of the upstream kernel is complete, please mark this bug as "Confirmed". Thanks in advance. [0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12 ** Changed in: linux (Ubuntu) Importance: Undecided => High ** Changed in: linux (Ubuntu) Status: Confirmed => Incomplete ** Tags added: kernel-da-key -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1702910 Title: System freeze after in6_dev_finish_destroy errors Status in linux package in Ubuntu: Incomplete Bug description: After the upgrade to the `4.4.0-83-generic #106~14.04.1-Ubuntu` kernel we are seeing system hangs with the following logs. Prior to this upgrade there were no crashes or kernel traces of this sort. We have ruled out physical hardware issues by moving the VM to another physical host. In the logs there's reference to docker instances on the host. This results in veth devices being brought up and torn down frequently, which is likely related. As a possible workaround we have disabled IPv6 for now. ``` kernel: [337352.274907] vethf317e0d: renamed from eth0 kernel: [337352.386446] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.393444] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.395408] device veth160ed4a left promiscuous mode kernel: [337352.395417] docker0: port 1(veth160ed4a) entered disabled state kernel: [337359.301576] vethee2599d: renamed from eth0 kernel: [337359.339582] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.343059] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.344987] device veth3100aab left promiscuous mode kernel: [337359.344997] docker0: port 2(veth3100aab) entered disabled state kernel: [337475.398358] [ cut here ] kernel: [337475.398370] WARNING: CPU: 3 PID: 23671 at /build/linux-lts-xenial-ep3zLI/linux-lts-xenial-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_f kernel: [337475.398372] Modules linked in: btrfs xor raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c veth ip6table_filter ip6_tab comment ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_fil filter bridge stp llc aufs tcp_diag inet_diag coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel ppdev aes_x86_64 lrw gf128mul input_leds serio_raw drm_kms_helper parport_pc drm lp fb_sys_fops syscopyarea sysfillrect sysimgblt vmw_vmci parport shpchp 8250_fintek i2c_piix4 kernel: [337475.398416] CPU: 3 PID: 23671 Comm: kworker/u8:0 Not tainted 4.4.0-83-generic #106~14.04.1-Ubuntu kernel: [337475.398418] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015 kernel: [337475.398423] Workqueue: netns cleanup_net kernel: [337475.398424] 8801b197fbb8 813ddefc kernel: [337475.398427] 81d7aa10 8801b197fbf0 8107ea36 88016f97d400 kernel: [337475.398428] 88016f97e800 0006 8801b197fcc0 kernel: [337475.398430] Call Trace: kernel: [337475.398435] [] dump_stack+0x63/0x87 kernel: [337475.398440] [] warn_slowpath_common+0x86/0xc0 kernel: [337475.398442] [] warn_slowpath_null+0x1a/0x20 kernel: [337475.398444] [] in6_dev_finish_destroy+0x6b/0xb0 kernel: [337475.398447] [] ip6_route_dev_notify+0x110/0x130 kernel: [337475.398450] [] notifier_call_chain+0x49/0x70 kernel: [337475.398452] [] raw_notifier_call_chain+0x16/0x20 kernel: [337475.398455] [] call_netdevice_notifiers_info+0x35/0x60 kernel: [337475.398458] [] netdev_run_todo+0x157/0x300 kernel: [337475.398461] [] ? rollback_registered_many+0x22e/0x2e0 kernel: [337475.398463] [] rtnl_unlock+0xe/0x10 kernel: [337475.398464] [] default_device_exit_batch+0x138/0x150 kernel: [337475.398468] [] ? __wake_up_sync+0x20/0x20 kernel: [337475.398470] [] ops_exit_list.isra.4+0x52/0x60 kernel: [337475.398472] [] cleanup_net+0x1b3/0x280 kernel: [337475.398476] [] process_one_work+0x150/0x3f0 kernel: [337475.398478] [] worker_thread+0x11a/0x470 kernel: [337475.398481] [] ? __schedule+0x359/0x980 kernel: [337475.398483] [] ? rescuer_thread+0x310/0x310 kernel: [337475.398484] [] kthread+0xd6/0xf0 kernel: [337475.398486] [] ? kthread_park+0x60/0x60 kernel: [337475.398488] [] ret_from_fork+0x3f/0x70 kernel: [337475.398490] [] ?
[Kernel-packages] [Bug 1702910] Re: System freeze after in6_dev_finish_destroy errors
With the sysctl net.ipv6.conf.all.disable_ipv6=1 we have still seen another kernel error as follows: [97989.224382] [ cut here ] [97989.224395] WARNING: CPU: 1 PID: 19570 at /build/linux-lts-xenial-ep3zLI/linux-lts-xenial-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_finish_destroy+0x6b/0xb0() [97989.224456] CPU: 1 PID: 19570 Comm: kworker/u8:3 Not tainted 4.4.0-83-generic #106~14.04.1-Ubuntu [97989.224457] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015 [97989.224462] Workqueue: netns cleanup_net [97989.224463] 88011b75bbb8 813ddefc [97989.224466] 81d7aa10 88011b75bbf0 8107ea36 8800960e7c00 [97989.224467] 8800960e6c00 0006 88011b75bcc0 [97989.224469] Call Trace: [97989.224474] [] dump_stack+0x63/0x87 [97989.224478] [] warn_slowpath_common+0x86/0xc0 [97989.224479] [] warn_slowpath_null+0x1a/0x20 [97989.224481] [] in6_dev_finish_destroy+0x6b/0xb0 [97989.224487] [] ip6_route_dev_notify+0x110/0x130 [97989.224490] [] notifier_call_chain+0x49/0x70 [97989.224492] [] raw_notifier_call_chain+0x16/0x20 [97989.224495] [] call_netdevice_notifiers_info+0x35/0x60 [97989.224498] [] netdev_run_todo+0x157/0x300 [97989.224500] [] rtnl_unlock+0xe/0x10 [97989.224502] [] default_device_exit_batch+0x138/0x150 [97989.224507] [] ? __wake_up_sync+0x20/0x20 [97989.224509] [] ops_exit_list.isra.4+0x52/0x60 [97989.224510] [] cleanup_net+0x1b3/0x280 [97989.224514] [] process_one_work+0x150/0x3f0 [97989.224516] [] worker_thread+0x11a/0x470 [97989.224518] [] ? rescuer_thread+0x310/0x310 [97989.224520] [] kthread+0xd6/0xf0 [97989.224523] [] ? do_exit+0x655/0xae0 [97989.224525] [] ? kthread_park+0x60/0x60 [97989.224527] [] ret_from_fork+0x3f/0x70 [97989.224529] [] ? kthread_park+0x60/0x60 [97989.224530] ---[ end trace dfb865796f5b5681 ]--- [97989.224531] [ cut here ] -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1702910 Title: System freeze after in6_dev_finish_destroy errors Status in linux package in Ubuntu: Confirmed Bug description: After the upgrade to the `4.4.0-83-generic #106~14.04.1-Ubuntu` kernel we are seeing system hangs with the following logs. Prior to this upgrade there were no crashes or kernel traces of this sort. We have ruled out physical hardware issues by moving the VM to another physical host. In the logs there's reference to docker instances on the host. This results in veth devices being brought up and torn down frequently, which is likely related. As a possible workaround we have disabled IPv6 for now. ``` kernel: [337352.274907] vethf317e0d: renamed from eth0 kernel: [337352.386446] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.393444] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.395408] device veth160ed4a left promiscuous mode kernel: [337352.395417] docker0: port 1(veth160ed4a) entered disabled state kernel: [337359.301576] vethee2599d: renamed from eth0 kernel: [337359.339582] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.343059] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.344987] device veth3100aab left promiscuous mode kernel: [337359.344997] docker0: port 2(veth3100aab) entered disabled state kernel: [337475.398358] [ cut here ] kernel: [337475.398370] WARNING: CPU: 3 PID: 23671 at /build/linux-lts-xenial-ep3zLI/linux-lts-xenial-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_f kernel: [337475.398372] Modules linked in: btrfs xor raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c veth ip6table_filter ip6_tab comment ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_fil filter bridge stp llc aufs tcp_diag inet_diag coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel ppdev aes_x86_64 lrw gf128mul input_leds serio_raw drm_kms_helper parport_pc drm lp fb_sys_fops syscopyarea sysfillrect sysimgblt vmw_vmci parport shpchp 8250_fintek i2c_piix4 kernel: [337475.398416] CPU: 3 PID: 23671 Comm: kworker/u8:0 Not tainted 4.4.0-83-generic #106~14.04.1-Ubuntu kernel: [337475.398418] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015 kernel: [337475.398423] Workqueue: netns cleanup_net kernel: [337475.398424] 8801b197fbb8 813ddefc kernel: [337475.398427] 81d7aa10 8801b197fbf0 8107ea36 88016f97d400 kernel: [337475.398428] 88016f97e800 0006 8801b197fcc0 kernel: [337475.398430] Call Trace:
[Kernel-packages] [Bug 1702910] Re: System freeze after in6_dev_finish_destroy errors
Logs are inline. If anything else is needed I'll capture and attach. ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1702910 Title: System freeze after in6_dev_finish_destroy errors Status in linux package in Ubuntu: Confirmed Bug description: After the upgrade to the `4.4.0-83-generic #106~14.04.1-Ubuntu` kernel we are seeing system hangs with the following logs. Prior to this upgrade there were no crashes or kernel traces of this sort. We have ruled out physical hardware issues by moving the VM to another physical host. In the logs there's reference to docker instances on the host. This results in veth devices being brought up and torn down frequently, which is likely related. As a possible workaround we have disabled IPv6 for now. ``` kernel: [337352.274907] vethf317e0d: renamed from eth0 kernel: [337352.386446] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.393444] docker0: port 1(veth160ed4a) entered disabled state kernel: [337352.395408] device veth160ed4a left promiscuous mode kernel: [337352.395417] docker0: port 1(veth160ed4a) entered disabled state kernel: [337359.301576] vethee2599d: renamed from eth0 kernel: [337359.339582] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.343059] docker0: port 2(veth3100aab) entered disabled state kernel: [337359.344987] device veth3100aab left promiscuous mode kernel: [337359.344997] docker0: port 2(veth3100aab) entered disabled state kernel: [337475.398358] [ cut here ] kernel: [337475.398370] WARNING: CPU: 3 PID: 23671 at /build/linux-lts-xenial-ep3zLI/linux-lts-xenial-4.4.0/net/ipv6/addrconf_core.c:159 in6_dev_f kernel: [337475.398372] Modules linked in: btrfs xor raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c veth ip6table_filter ip6_tab comment ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_fil filter bridge stp llc aufs tcp_diag inet_diag coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel ppdev aes_x86_64 lrw gf128mul input_leds serio_raw drm_kms_helper parport_pc drm lp fb_sys_fops syscopyarea sysfillrect sysimgblt vmw_vmci parport shpchp 8250_fintek i2c_piix4 kernel: [337475.398416] CPU: 3 PID: 23671 Comm: kworker/u8:0 Not tainted 4.4.0-83-generic #106~14.04.1-Ubuntu kernel: [337475.398418] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015 kernel: [337475.398423] Workqueue: netns cleanup_net kernel: [337475.398424] 8801b197fbb8 813ddefc kernel: [337475.398427] 81d7aa10 8801b197fbf0 8107ea36 88016f97d400 kernel: [337475.398428] 88016f97e800 0006 8801b197fcc0 kernel: [337475.398430] Call Trace: kernel: [337475.398435] [] dump_stack+0x63/0x87 kernel: [337475.398440] [] warn_slowpath_common+0x86/0xc0 kernel: [337475.398442] [] warn_slowpath_null+0x1a/0x20 kernel: [337475.398444] [] in6_dev_finish_destroy+0x6b/0xb0 kernel: [337475.398447] [] ip6_route_dev_notify+0x110/0x130 kernel: [337475.398450] [] notifier_call_chain+0x49/0x70 kernel: [337475.398452] [] raw_notifier_call_chain+0x16/0x20 kernel: [337475.398455] [] call_netdevice_notifiers_info+0x35/0x60 kernel: [337475.398458] [] netdev_run_todo+0x157/0x300 kernel: [337475.398461] [] ? rollback_registered_many+0x22e/0x2e0 kernel: [337475.398463] [] rtnl_unlock+0xe/0x10 kernel: [337475.398464] [] default_device_exit_batch+0x138/0x150 kernel: [337475.398468] [] ? __wake_up_sync+0x20/0x20 kernel: [337475.398470] [] ops_exit_list.isra.4+0x52/0x60 kernel: [337475.398472] [] cleanup_net+0x1b3/0x280 kernel: [337475.398476] [] process_one_work+0x150/0x3f0 kernel: [337475.398478] [] worker_thread+0x11a/0x470 kernel: [337475.398481] [] ? __schedule+0x359/0x980 kernel: [337475.398483] [] ? rescuer_thread+0x310/0x310 kernel: [337475.398484] [] kthread+0xd6/0xf0 kernel: [337475.398486] [] ? kthread_park+0x60/0x60 kernel: [337475.398488] [] ret_from_fork+0x3f/0x70 kernel: [337475.398490] [] ? kthread_park+0x60/0x60 kernel: [337475.398491] ---[ end trace 6d63274dc9a3dbec ]--- kernel: [337475.398492] [ cut here ] ``` ProblemType: Bug ApportVersion: 2.14.1-0ubuntu3.24 Architecture: amd64 Date: Fri Jul 7 13:59:36 2017 Dependencies: adduser 3.113+nmu3ubuntu3 apt-utils 1.0.1ubuntu2.17 base-passwd 3.5.33 busybox-initramfs 1:1.21.0-1ubuntu1 coreutils 8.21-1ubuntu5.4 cpio 2.11+dfsg-1ubuntu1.2 dbus 1.6.18-0ubuntu4.5 debconf 1.5.51ubuntu2 debconf-i18n 1.5.51ubuntu2
