[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
4.4.0-186 is already released. Both 4.4.0-185 and 4.4.0-186 contain the fix for this issue and work fine for me, no crashes observed. ** Changed in: linux (Ubuntu Xenial) Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Fix Released Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump installed I obtained the dmesgs and crash dumps. The backtrace appears somewhat similar to #202669 but that one only happened on bare hardware for us - this one is on KVM virtual instances. #202669 paniced in icmpv6_route_lookup and this one dies already in icmpv6_send. Between 2020-06-11 and 2020-06-15, on a set of 12 VMs running 4.4.0-184-generic, there were 85 crashes like this, on servers with noticeable IPv6 traffic. All of the 12 VMs with 4.4.0-184-generic crashed at least once. (There are more than 12 VMs experiencing this, this is just the set I had linux-crashdump on.) [57063.487084] BUG: unable to handle kernel NULL pointer dereference at 0018 [57063.487184] IP: [] icmp6_send+0x1fb/0x970 [57063.487218] PGD 0 [57063.487231] Oops: [#1] SMP [57063.488665] Call Trace: [57063.488679] [57063.488705] [] ? __netif_receive_skb+0x18/0x60 [57063.488739] [] ? task_tick_fair+0x4c8/0x8e0 [57063.488771] [] ? _raw_spin_unlock_bh+0x20/0x50 [57063.488802] [] icmpv6_send+0x21/0x30 [57063.488829] [] ip6_expire_frag_queue+0x115/0x1b0 [57063.488862] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.488897] [] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [57063.488937] [] call_timer_fn+0x37/0x140 [57063.488965] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.489002] [] run_timer_softirq+0x234/0x330 ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1883498/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
Is there any news on this Bug Report? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Confirmed Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump installed I obtained the dmesgs and crash dumps. The backtrace appears somewhat similar to #202669 but that one only happened on bare hardware for us - this one is on KVM virtual instances. #202669 paniced in icmpv6_route_lookup and this one dies already in icmpv6_send. Between 2020-06-11 and 2020-06-15, on a set of 12 VMs running 4.4.0-184-generic, there were 85 crashes like this, on servers with noticeable IPv6 traffic. All of the 12 VMs with 4.4.0-184-generic crashed at least once. (There are more than 12 VMs experiencing this, this is just the set I had linux-crashdump on.) [57063.487084] BUG: unable to handle kernel NULL pointer dereference at 0018 [57063.487184] IP: [] icmp6_send+0x1fb/0x970 [57063.487218] PGD 0 [57063.487231] Oops: [#1] SMP [57063.488665] Call Trace: [57063.488679] [57063.488705] [] ? __netif_receive_skb+0x18/0x60 [57063.488739] [] ? task_tick_fair+0x4c8/0x8e0 [57063.488771] [] ? _raw_spin_unlock_bh+0x20/0x50 [57063.488802] [] icmpv6_send+0x21/0x30 [57063.488829] [] ip6_expire_frag_queue+0x115/0x1b0 [57063.488862] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.488897] [] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [57063.488937] [] call_timer_fn+0x37/0x140 [57063.488965] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.489002] [] run_timer_softirq+0x234/0x330 ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1883498/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
I can confirm that since booting on 4.4.0-185 (~24h ago), we have not experienced any panics on our systems; while we had 4 panics in less than 24h on 4.4.0-184. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Confirmed Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump installed I obtained the dmesgs and crash dumps. The backtrace appears somewhat similar to #202669 but that one only happened on bare hardware for us - this one is on KVM virtual instances. #202669 paniced in icmpv6_route_lookup and this one dies already in icmpv6_send. Between 2020-06-11 and 2020-06-15, on a set of 12 VMs running 4.4.0-184-generic, there were 85 crashes like this, on servers with noticeable IPv6 traffic. All of the 12 VMs with 4.4.0-184-generic crashed at least once. (There are more than 12 VMs experiencing this, this is just the set I had linux-crashdump on.) [57063.487084] BUG: unable to handle kernel NULL pointer dereference at 0018 [57063.487184] IP: [] icmp6_send+0x1fb/0x970 [57063.487218] PGD 0 [57063.487231] Oops: [#1] SMP [57063.488665] Call Trace: [57063.488679] [57063.488705] [] ? __netif_receive_skb+0x18/0x60 [57063.488739] [] ? task_tick_fair+0x4c8/0x8e0 [57063.488771] [] ? _raw_spin_unlock_bh+0x20/0x50 [57063.488802] [] icmpv6_send+0x21/0x30 [57063.488829] [] ip6_expire_frag_queue+0x115/0x1b0 [57063.488862] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.488897] [] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [57063.488937] [] call_timer_fn+0x37/0x140 [57063.488965] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.489002] [] run_timer_softirq+0x234/0x330 ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1883498/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
i've made a upgrade to the latest kernel now on a server with unbound installed. $ uname -a Linux nsr4-cbn 4.4.0-185-generic #215-Ubuntu SMP Mon Jun 8 21:53:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux $ cat /etc/os-release VERSION="16.04.6 LTS (Xenial Xerus)" it seems 4.4.0-185 is out?! As far, no Kernel Panic -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Confirmed Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump installed I obtained the dmesgs and crash dumps. The backtrace appears somewhat similar to #202669 but that one only happened on bare hardware for us - this one is on KVM virtual instances. #202669 paniced in icmpv6_route_lookup and this one dies already in icmpv6_send. Between 2020-06-11 and 2020-06-15, on a set of 12 VMs running 4.4.0-184-generic, there were 85 crashes like this, on servers with noticeable IPv6 traffic. All of the 12 VMs with 4.4.0-184-generic crashed at least once. (There are more than 12 VMs experiencing this, this is just the set I had linux-crashdump on.) [57063.487084] BUG: unable to handle kernel NULL pointer dereference at 0018 [57063.487184] IP: [] icmp6_send+0x1fb/0x970 [57063.487218] PGD 0 [57063.487231] Oops: [#1] SMP [57063.488665] Call Trace: [57063.488679] [57063.488705] [] ? __netif_receive_skb+0x18/0x60 [57063.488739] [] ? task_tick_fair+0x4c8/0x8e0 [57063.488771] [] ? _raw_spin_unlock_bh+0x20/0x50 [57063.488802] [] icmpv6_send+0x21/0x30 [57063.488829] [] ip6_expire_frag_queue+0x115/0x1b0 [57063.488862] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.488897] [] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [57063.488937] [] call_timer_fn+0x37/0x140 [57063.488965] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.489002] [] run_timer_softirq+0x234/0x330 ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1883498/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
** Tags added: verification-done-xenial -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Confirmed Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump installed I obtained the dmesgs and crash dumps. The backtrace appears somewhat similar to #202669 but that one only happened on bare hardware for us - this one is on KVM virtual instances. #202669 paniced in icmpv6_route_lookup and this one dies already in icmpv6_send. Between 2020-06-11 and 2020-06-15, on a set of 12 VMs running 4.4.0-184-generic, there were 85 crashes like this, on servers with noticeable IPv6 traffic. All of the 12 VMs with 4.4.0-184-generic crashed at least once. (There are more than 12 VMs experiencing this, this is just the set I had linux-crashdump on.) [57063.487084] BUG: unable to handle kernel NULL pointer dereference at 0018 [57063.487184] IP: [] icmp6_send+0x1fb/0x970 [57063.487218] PGD 0 [57063.487231] Oops: [#1] SMP [57063.488665] Call Trace: [57063.488679] [57063.488705] [] ? __netif_receive_skb+0x18/0x60 [57063.488739] [] ? task_tick_fair+0x4c8/0x8e0 [57063.488771] [] ? _raw_spin_unlock_bh+0x20/0x50 [57063.488802] [] icmpv6_send+0x21/0x30 [57063.488829] [] ip6_expire_frag_queue+0x115/0x1b0 [57063.488862] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.488897] [] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [57063.488937] [] call_timer_fn+0x37/0x140 [57063.488965] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.489002] [] run_timer_softirq+0x234/0x330 ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1883498/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
4.4.0-185 from xenial-proposed seems stable still. Any chance of releasing it to mainline xenial soon, we have some other needs to run upgrades and deploying proposed packages is a bit of a hassle for a fleet of hundreds? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Confirmed Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump installed I obtained the dmesgs and crash dumps. The backtrace appears somewhat similar to #202669 but that one only happened on bare hardware for us - this one is on KVM virtual instances. #202669 paniced in icmpv6_route_lookup and this one dies already in icmpv6_send. Between 2020-06-11 and 2020-06-15, on a set of 12 VMs running 4.4.0-184-generic, there were 85 crashes like this, on servers with noticeable IPv6 traffic. All of the 12 VMs with 4.4.0-184-generic crashed at least once. (There are more than 12 VMs experiencing this, this is just the set I had linux-crashdump on.) [57063.487084] BUG: unable to handle kernel NULL pointer dereference at 0018 [57063.487184] IP: [] icmp6_send+0x1fb/0x970 [57063.487218] PGD 0 [57063.487231] Oops: [#1] SMP [57063.488665] Call Trace: [57063.488679] [57063.488705] [] ? __netif_receive_skb+0x18/0x60 [57063.488739] [] ? task_tick_fair+0x4c8/0x8e0 [57063.488771] [] ? _raw_spin_unlock_bh+0x20/0x50 [57063.488802] [] icmpv6_send+0x21/0x30 [57063.488829] [] ip6_expire_frag_queue+0x115/0x1b0 [57063.488862] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.488897] [] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [57063.488937] [] call_timer_fn+0x37/0x140 [57063.488965] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.489002] [] run_timer_softirq+0x234/0x330 ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1883498/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
I have deployed 4.4.0-185 from xenial-proposed to 6 VMs on Monday. No crashes yet, so it seems to me that 4.4.0-185 is good and fixes this issue. On 4.4.0-184 these ones crashed very frequently. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Confirmed Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump installed I obtained the dmesgs and crash dumps. The backtrace appears somewhat similar to #202669 but that one only happened on bare hardware for us - this one is on KVM virtual instances. #202669 paniced in icmpv6_route_lookup and this one dies already in icmpv6_send. Between 2020-06-11 and 2020-06-15, on a set of 12 VMs running 4.4.0-184-generic, there were 85 crashes like this, on servers with noticeable IPv6 traffic. All of the 12 VMs with 4.4.0-184-generic crashed at least once. (There are more than 12 VMs experiencing this, this is just the set I had linux-crashdump on.) [57063.487084] BUG: unable to handle kernel NULL pointer dereference at 0018 [57063.487184] IP: [] icmp6_send+0x1fb/0x970 [57063.487218] PGD 0 [57063.487231] Oops: [#1] SMP [57063.488665] Call Trace: [57063.488679] [57063.488705] [] ? __netif_receive_skb+0x18/0x60 [57063.488739] [] ? task_tick_fair+0x4c8/0x8e0 [57063.488771] [] ? _raw_spin_unlock_bh+0x20/0x50 [57063.488802] [] icmpv6_send+0x21/0x30 [57063.488829] [] ip6_expire_frag_queue+0x115/0x1b0 [57063.488862] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.488897] [] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [57063.488937] [] call_timer_fn+0x37/0x140 [57063.488965] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.489002] [] run_timer_softirq+0x234/0x330 ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1883498/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
I can deploy 4.4.0-185 from xenial-proposed for testing on Monday. The fix looks good, I was already staring at https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit?id=5d41ce29e3b91ef305f88d23f72b3359de329cec which came in 4.4.0-184. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Confirmed Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump installed I obtained the dmesgs and crash dumps. The backtrace appears somewhat similar to #202669 but that one only happened on bare hardware for us - this one is on KVM virtual instances. #202669 paniced in icmpv6_route_lookup and this one dies already in icmpv6_send. Between 2020-06-11 and 2020-06-15, on a set of 12 VMs running 4.4.0-184-generic, there were 85 crashes like this, on servers with noticeable IPv6 traffic. All of the 12 VMs with 4.4.0-184-generic crashed at least once. (There are more than 12 VMs experiencing this, this is just the set I had linux-crashdump on.) [57063.487084] BUG: unable to handle kernel NULL pointer dereference at 0018 [57063.487184] IP: [] icmp6_send+0x1fb/0x970 [57063.487218] PGD 0 [57063.487231] Oops: [#1] SMP [57063.488665] Call Trace: [57063.488679] [57063.488705] [] ? __netif_receive_skb+0x18/0x60 [57063.488739] [] ? task_tick_fair+0x4c8/0x8e0 [57063.488771] [] ? _raw_spin_unlock_bh+0x20/0x50 [57063.488802] [] icmpv6_send+0x21/0x30 [57063.488829] [] ip6_expire_frag_queue+0x115/0x1b0 [57063.488862] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.488897] [] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [57063.488937] [] call_timer_fn+0x37/0x140 [57063.488965] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.489002] [] run_timer_softirq+0x234/0x330 ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1883498/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
@dvaerum I installed it from the xenial-proposed archive, for testing purpose only of course. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Confirmed Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump installed I obtained the dmesgs and crash dumps. The backtrace appears somewhat similar to #202669 but that one only happened on bare hardware for us - this one is on KVM virtual instances. #202669 paniced in icmpv6_route_lookup and this one dies already in icmpv6_send. Between 2020-06-11 and 2020-06-15, on a set of 12 VMs running 4.4.0-184-generic, there were 85 crashes like this, on servers with noticeable IPv6 traffic. All of the 12 VMs with 4.4.0-184-generic crashed at least once. (There are more than 12 VMs experiencing this, this is just the set I had linux-crashdump on.) [57063.487084] BUG: unable to handle kernel NULL pointer dereference at 0018 [57063.487184] IP: [] icmp6_send+0x1fb/0x970 [57063.487218] PGD 0 [57063.487231] Oops: [#1] SMP [57063.488665] Call Trace: [57063.488679] [57063.488705] [] ? __netif_receive_skb+0x18/0x60 [57063.488739] [] ? task_tick_fair+0x4c8/0x8e0 [57063.488771] [] ? _raw_spin_unlock_bh+0x20/0x50 [57063.488802] [] icmpv6_send+0x21/0x30 [57063.488829] [] ip6_expire_frag_queue+0x115/0x1b0 [57063.488862] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.488897] [] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [57063.488937] [] call_timer_fn+0x37/0x140 [57063.488965] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.489002] [] run_timer_softirq+0x234/0x330 ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1883498/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
@blueh how did you install the 4.4.0-185 kernel? I don't seen to have it as an option -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Confirmed Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump installed I obtained the dmesgs and crash dumps. The backtrace appears somewhat similar to #202669 but that one only happened on bare hardware for us - this one is on KVM virtual instances. #202669 paniced in icmpv6_route_lookup and this one dies already in icmpv6_send. Between 2020-06-11 and 2020-06-15, on a set of 12 VMs running 4.4.0-184-generic, there were 85 crashes like this, on servers with noticeable IPv6 traffic. All of the 12 VMs with 4.4.0-184-generic crashed at least once. (There are more than 12 VMs experiencing this, this is just the set I had linux-crashdump on.) [57063.487084] BUG: unable to handle kernel NULL pointer dereference at 0018 [57063.487184] IP: [] icmp6_send+0x1fb/0x970 [57063.487218] PGD 0 [57063.487231] Oops: [#1] SMP [57063.488665] Call Trace: [57063.488679] [57063.488705] [] ? __netif_receive_skb+0x18/0x60 [57063.488739] [] ? task_tick_fair+0x4c8/0x8e0 [57063.488771] [] ? _raw_spin_unlock_bh+0x20/0x50 [57063.488802] [] icmpv6_send+0x21/0x30 [57063.488829] [] ip6_expire_frag_queue+0x115/0x1b0 [57063.488862] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.488897] [] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [57063.488937] [] call_timer_fn+0x37/0x140 [57063.488965] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.489002] [] run_timer_softirq+0x234/0x330 ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1883498/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
I have a server running with this (4.4.0-185) kernel right now, with Unbound installed, and it haven't panicked yet. This server hung seconds after boot with 4.4.0-184, so I believe it's a good candidate. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Confirmed Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump installed I obtained the dmesgs and crash dumps. The backtrace appears somewhat similar to #202669 but that one only happened on bare hardware for us - this one is on KVM virtual instances. #202669 paniced in icmpv6_route_lookup and this one dies already in icmpv6_send. Between 2020-06-11 and 2020-06-15, on a set of 12 VMs running 4.4.0-184-generic, there were 85 crashes like this, on servers with noticeable IPv6 traffic. All of the 12 VMs with 4.4.0-184-generic crashed at least once. (There are more than 12 VMs experiencing this, this is just the set I had linux-crashdump on.) [57063.487084] BUG: unable to handle kernel NULL pointer dereference at 0018 [57063.487184] IP: [] icmp6_send+0x1fb/0x970 [57063.487218] PGD 0 [57063.487231] Oops: [#1] SMP [57063.488665] Call Trace: [57063.488679] [57063.488705] [] ? __netif_receive_skb+0x18/0x60 [57063.488739] [] ? task_tick_fair+0x4c8/0x8e0 [57063.488771] [] ? _raw_spin_unlock_bh+0x20/0x50 [57063.488802] [] icmpv6_send+0x21/0x30 [57063.488829] [] ip6_expire_frag_queue+0x115/0x1b0 [57063.488862] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.488897] [] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [57063.488937] [] call_timer_fn+0x37/0x140 [57063.488965] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.489002] [] run_timer_softirq+0x234/0x330 ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1883498/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
It looks like this is fixed upstream with this change: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit?id=178af2f97dcaea27611f0420ec7b61c1a27d6776 Which is contained in the Ubuntu-4.4.0-185 kernel already. So Ubuntu-4.4.0-185 should be fixed. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Confirmed Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump installed I obtained the dmesgs and crash dumps. The backtrace appears somewhat similar to #202669 but that one only happened on bare hardware for us - this one is on KVM virtual instances. #202669 paniced in icmpv6_route_lookup and this one dies already in icmpv6_send. Between 2020-06-11 and 2020-06-15, on a set of 12 VMs running 4.4.0-184-generic, there were 85 crashes like this, on servers with noticeable IPv6 traffic. All of the 12 VMs with 4.4.0-184-generic crashed at least once. (There are more than 12 VMs experiencing this, this is just the set I had linux-crashdump on.) [57063.487084] BUG: unable to handle kernel NULL pointer dereference at 0018 [57063.487184] IP: [] icmp6_send+0x1fb/0x970 [57063.487218] PGD 0 [57063.487231] Oops: [#1] SMP [57063.488665] Call Trace: [57063.488679] [57063.488705] [] ? __netif_receive_skb+0x18/0x60 [57063.488739] [] ? task_tick_fair+0x4c8/0x8e0 [57063.488771] [] ? _raw_spin_unlock_bh+0x20/0x50 [57063.488802] [] icmpv6_send+0x21/0x30 [57063.488829] [] ip6_expire_frag_queue+0x115/0x1b0 [57063.488862] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.488897] [] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [57063.488937] [] call_timer_fn+0x37/0x140 [57063.488965] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.489002] [] run_timer_softirq+0x234/0x330 ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1883498/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
** Changed in: linux (Ubuntu Xenial) Assignee: (unassigned) => Kleber Sacilotto de Souza (kleber-souza) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Confirmed Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump installed I obtained the dmesgs and crash dumps. The backtrace appears somewhat similar to #202669 but that one only happened on bare hardware for us - this one is on KVM virtual instances. #202669 paniced in icmpv6_route_lookup and this one dies already in icmpv6_send. Between 2020-06-11 and 2020-06-15, on a set of 12 VMs running 4.4.0-184-generic, there were 85 crashes like this, on servers with noticeable IPv6 traffic. All of the 12 VMs with 4.4.0-184-generic crashed at least once. (There are more than 12 VMs experiencing this, this is just the set I had linux-crashdump on.) [57063.487084] BUG: unable to handle kernel NULL pointer dereference at 0018 [57063.487184] IP: [] icmp6_send+0x1fb/0x970 [57063.487218] PGD 0 [57063.487231] Oops: [#1] SMP [57063.488665] Call Trace: [57063.488679] [57063.488705] [] ? __netif_receive_skb+0x18/0x60 [57063.488739] [] ? task_tick_fair+0x4c8/0x8e0 [57063.488771] [] ? _raw_spin_unlock_bh+0x20/0x50 [57063.488802] [] icmpv6_send+0x21/0x30 [57063.488829] [] ip6_expire_frag_queue+0x115/0x1b0 [57063.488862] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.488897] [] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [57063.488937] [] call_timer_fn+0x37/0x140 [57063.488965] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.489002] [] run_timer_softirq+0x234/0x330 ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1883498/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
** Also affects: linux (Ubuntu Xenial) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Xenial) Status: New => Confirmed ** Changed in: linux (Ubuntu Xenial) Importance: Undecided => High ** Changed in: linux (Ubuntu) Status: Confirmed => Incomplete -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Confirmed Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump installed I obtained the dmesgs and crash dumps. The backtrace appears somewhat similar to #202669 but that one only happened on bare hardware for us - this one is on KVM virtual instances. #202669 paniced in icmpv6_route_lookup and this one dies already in icmpv6_send. Between 2020-06-11 and 2020-06-15, on a set of 12 VMs running 4.4.0-184-generic, there were 85 crashes like this, on servers with noticeable IPv6 traffic. All of the 12 VMs with 4.4.0-184-generic crashed at least once. (There are more than 12 VMs experiencing this, this is just the set I had linux-crashdump on.) [57063.487084] BUG: unable to handle kernel NULL pointer dereference at 0018 [57063.487184] IP: [] icmp6_send+0x1fb/0x970 [57063.487218] PGD 0 [57063.487231] Oops: [#1] SMP [57063.488665] Call Trace: [57063.488679] [57063.488705] [] ? __netif_receive_skb+0x18/0x60 [57063.488739] [] ? task_tick_fair+0x4c8/0x8e0 [57063.488771] [] ? _raw_spin_unlock_bh+0x20/0x50 [57063.488802] [] icmpv6_send+0x21/0x30 [57063.488829] [] ip6_expire_frag_queue+0x115/0x1b0 [57063.488862] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.488897] [] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [57063.488937] [] call_timer_fn+0x37/0x140 [57063.488965] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.489002] [] run_timer_softirq+0x234/0x330 ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1883498/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
I can confirm that we experienced the same problem on one VM after upgrade from 4.4.0-179-generic to 4.4.0-184-generic last weekend. Since the rollback to the last working kernel this VM is running stable for over 25h now. Ubound Version 1.5.8 is also installed and running on this VM Don't have any crashdump, but the traceback looks the same: [ 1963.770497] BUG: unable to handle kernel NULL pointer dereference at 0018 [ 1963.781264] IP: [] icmp6_send+0x1fb/0x970 [ 1963.782881] PGD 0 [ 1963.783503] Oops: [#1] SMP [ 1963.784479] Modules linked in: binfmt_misc ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables kvm_intel kvm irqbypass input_leds joydev serio_raw i2c_piix4 mac_hid 8250_fintek autofs4 qxl ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops psmouse drm pata_acpi floppy [ 1963.794748] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.4.0-184-generic #214-Ubuntu [ 1963.796182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 [ 1963.797696] task: 880236330d00 ti: 88023633c000 task.ti: 88023633c000 [ 1963.799081] RIP: 0010:[] [] icmp6_send+0x1fb/0x970 [ 1963.800628] RSP: 0018:88023fd03d00 EFLAGS: 00010246 [ 1963.801630] RAX: RBX: 8800bbad6700 RCX: 0020 [ 1963.802948] RDX: 0001 RSI: 0200 RDI: 880232c86a56 [ 1963.804281] RBP: 88023fd03e20 R08: R09: 880232c86a66 [ 1963.805625] R10: 0080 R11: R12: 880232c86a4e [ 1963.806951] R13: 81efb6c0 R14: 0001 R15: 0003 [ 1963.808399] FS: () GS:88023fd0() knlGS: [ 1963.809910] CS: 0010 DS: ES: CR0: 80050033 [ 1963.810987] CR2: 0018 CR3: 000234d7a000 CR4: 0670 [ 1963.812324] DR0: DR1: DR2: [ 1963.813662] DR3: DR6: fffe0ff0 DR7: 0400 [ 1963.815001] Stack: [ 1963.815395] 0001 23dc61b86a54e883 [ 1963.816889] 8802371b3a98 8802371b3ae0 371b3a80 880232c86a56 [ 1963.818372] 88020001 880232c86a66 [ 1963.819847] Call Trace: [ 1963.820317] [ 1963.820778] [] ? emulator_pio_in_emulated+0x1a0/0x1a0 [kvm] [ 1963.822192] [] ? notifier_call_chain+0x4c/0x70 [ 1963.823330] [] ? _raw_spin_unlock_bh+0x20/0x50 [ 1963.824475] [] icmpv6_send+0x21/0x30 [ 1963.825452] [] ip6_expire_frag_queue+0x115/0x1b0 [ 1963.826622] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [ 1963.827951] [] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [ 1963.829365] [] call_timer_fn+0x37/0x140 [ 1963.830428] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [ 1963.831826] [] run_timer_softirq+0x234/0x330 [ 1963.832958] [] __do_softirq+0x109/0x2b0 [ 1963.834002] [] irq_exit+0xa5/0xb0 [ 1963.834956] [] smp_apic_timer_interrupt+0x50/0x70 [ 1963.836164] [] apic_timer_interrupt+0xd4/0xe0 [ 1963.837292] [ 1963.837695] [] ? speculation_ctrl_update_tif+0x80/0x80 [ 1963.839394] [] ? native_safe_halt+0x12/0x20 [ 1963.840723] [] default_idle+0x1e/0xe0 [ 1963.841943] [] arch_cpu_idle+0x15/0x20 [ 1963.843171] [] default_idle_call+0x2a/0x40 [ 1963.844486] [] cpu_startup_entry+0x303/0x360 [ 1963.845822] [] start_secondary+0x177/0x1b0 [ 1963.847102] Code: 8b 5c 24 40 75 46 f6 c2 02 74 05 f6 c2 30 75 3c 48 8b 43 58 44 89 5c 24 34 89 54 24 40 44 89 44 24 48 4c 89 4c 24 60 48 83 e0 fe <48> 8b 78 18 e8 4c 0b 03 00 41 89 c2 4c 8b 4c 24 60 44 8b 44 24 [ 1963.852721] RIP [] icmp6_send+0x1fb/0x970 [ 1963.854022] RSP [ 1963.854896] CR2: 0018 [ 1963.857283] ---[ end trace e8d1fc7789e99b6a ]--- [ 1963.858369] Kernel panic - not syncing: Fatal exception in interrupt [ 1963.860271] Kernel Offset: disabled [ 1963.861149] ---[ end Kernel panic - not syncing: Fatal exception in interrupt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Confirmed Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump installed I obtained the dmesgs and crash dumps. The backtrace appears somewhat similar to #202669 but that one only happened on bare hardware for us - this one is on KVM virtual
[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
We had the same behaviour like Dennis! Kernel Panic after aprox. 4 our on kernel 4.4.0-184 Unbound installed as well with version 1.5.8 Unfortunately i have no crashdump by hand, but our panic was "same" as Dennis with ipv6 messages. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Confirmed Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump installed I obtained the dmesgs and crash dumps. The backtrace appears somewhat similar to #202669 but that one only happened on bare hardware for us - this one is on KVM virtual instances. #202669 paniced in icmpv6_route_lookup and this one dies already in icmpv6_send. Between 2020-06-11 and 2020-06-15, on a set of 12 VMs running 4.4.0-184-generic, there were 85 crashes like this, on servers with noticeable IPv6 traffic. All of the 12 VMs with 4.4.0-184-generic crashed at least once. (There are more than 12 VMs experiencing this, this is just the set I had linux-crashdump on.) [57063.487084] BUG: unable to handle kernel NULL pointer dereference at 0018 [57063.487184] IP: [] icmp6_send+0x1fb/0x970 [57063.487218] PGD 0 [57063.487231] Oops: [#1] SMP [57063.488665] Call Trace: [57063.488679] [57063.488705] [] ? __netif_receive_skb+0x18/0x60 [57063.488739] [] ? task_tick_fair+0x4c8/0x8e0 [57063.488771] [] ? _raw_spin_unlock_bh+0x20/0x50 [57063.488802] [] icmpv6_send+0x21/0x30 [57063.488829] [] ip6_expire_frag_queue+0x115/0x1b0 [57063.488862] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.488897] [] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [57063.488937] [] call_timer_fn+0x37/0x140 [57063.488965] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.489002] [] run_timer_softirq+0x234/0x330 ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1883498/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
We are having the same bug (I believe) after upgrading from kernel linux-image-4.4.0-178-generic to linux-image-4.4.0-184-generic. We have around 100 VMs there are affected. For now, we have rolled back to the previous kernel. I am not sure why but not all VMs are affected, from what I have found, it looks like unbound (DNS server) is triggering the kernel oops our clients environment. I can help test a new kernel if that could help/be useful. I also have a kernel dump from linux-crashdump, but I am not currently sure if I am allow to share it, but I will try to figure it out if needed. ### Our kernel crash [ 128.503474] BUG: unable to handle kernel NULL pointer dereference at 0018 [ 128.503608] IP: [] icmp6_send+0x1fb/0x970 [ 128.503673] PGD 8004275f2067 PUD 427495067 PMD 0 [ 128.503736] Oops: [#1] SMP [ 128.503800] Modules linked in: vmw_vsock_vmci_transport vsock zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO) vmw_balloon input_leds joydev serio_raw shpchp vmw_vmci i2c_piix4 mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd vmwgfx ttm drm_kms_helper psmouse syscopyarea sysfillrect vmxnet3 sysimgblt vmw_pvscsi fb_sys_fops pata_acpi drm ahci libahci fjes [ 128.504798] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P O 4.4.0-184-generic #214-Ubuntu [ 128.504990] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018 [ 128.505401] task: 81e13500 ti: 81e0 task.ti: 81e0 [ 128.505637] RIP: 0010:[] [] icmp6_send+0x1fb/0x970 [ 128.505892] RSP: 0018:88042d603d00 EFLAGS: 00010246 [ 128.506143] RAX: RBX: 880423804a00 RCX: 0020 [ 128.506409] RDX: 0001 RSI: 0200 RDI: 880427ce1856 [ 128.506686] RBP: 88042d603e20 R08: R09: 880427ce1866 [ 128.506962] R10: 0080 R11: R12: 880427ce184e [ 128.507246] R13: 81efb6c0 R14: 0001 R15: 0003 [ 128.507539] FS: () GS:88042d60() knlGS: [ 128.507842] CS: 0010 DS: ES: CR0: 80050033 [ 128.508176] CR2: 0018 CR3: 000427782000 CR4: 00360670 [ 128.508530] Stack: [ 128.508859] 0001 4a7e338b0c959fd7 [ 128.509212] 88042b139a38 88042b139a80 2b139a20 880427ce1856 [ 128.509577] 88040001 880427ce1866 [ 128.509945] Call Trace: [ 128.510314] [ 128.510324] [] ? _raw_spin_unlock_bh+0x20/0x50 [ 128.511089] [] icmpv6_send+0x21/0x30 [ 128.511483] [] ip6_expire_frag_queue+0x115/0x1b0 [ 128.511892] [] ? ip6_expire_frag_queue+0x1b0/0x1b0 [ 128.512301] [] ip6_frag_expire+0x1f/0x30 [ 128.512723] [] call_timer_fn+0x37/0x140 [ 128.513116] [] ? ip6_expire_frag_queue+0x1b0/0x1b0 [ 128.513509] [] run_timer_softirq+0x234/0x330 [ 128.513902] [] __do_softirq+0x109/0x2b0 [ 128.514291] [] irq_exit+0xa5/0xb0 [ 128.514673] [] smp_apic_timer_interrupt+0x50/0x70 [ 128.515045] [] apic_timer_interrupt+0xd4/0xe0 [ 128.515414] [ 128.515423] [] ? speculation_ctrl_update_tif+0x80/0x80 [ 128.516123] [] ? native_safe_halt+0x12/0x20 [ 128.516466] [] default_idle+0x1e/0xe0 [ 128.516802] [] arch_cpu_idle+0x15/0x20 [ 128.517124] [] default_idle_call+0x2a/0x40 [ 128.517441] [] cpu_startup_entry+0x303/0x360 [ 128.517757] [] rest_init+0x7c/0x80 [ 128.518055] [] start_kernel+0x483/0x4a4 [ 128.518367] [] ? early_idt_handler_array+0x120/0x120 [ 128.518665] [] x86_64_start_reservations+0x2a/0x2c [ 128.518952] [] x86_64_start_kernel+0x14a/0x16d [ 128.519234] Code: 8b 5c 24 40 75 46 f6 c2 02 74 05 f6 c2 30 75 3c 48 8b 43 58 44 89 5c 24 34 89 54 24 40 44 89 44 24 48 4c 89 4c 24 60 48 83 e0 fe <48> 8b 78 18 e8 4c 0b 03 00 41 89 c2 4c 8b 4c 24 60 44 8b 44 24 [ 128.520205] RIP [] icmp6_send+0x1fb/0x970 [ 128.520511] RSP [ 128.520818] CR2: 0018 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Confirmed Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump
[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
Paste error in original report; the related-but-not-quite-the-same bug was here: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1824687 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Confirmed Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump installed I obtained the dmesgs and crash dumps. The backtrace appears somewhat similar to #202669 but that one only happened on bare hardware for us - this one is on KVM virtual instances. #202669 paniced in icmpv6_route_lookup and this one dies already in icmpv6_send. Between 2020-06-11 and 2020-06-15, on a set of 12 VMs running 4.4.0-184-generic, there were 85 crashes like this, on servers with noticeable IPv6 traffic. All of the 12 VMs with 4.4.0-184-generic crashed at least once. (There are more than 12 VMs experiencing this, this is just the set I had linux-crashdump on.) [57063.487084] BUG: unable to handle kernel NULL pointer dereference at 0018 [57063.487184] IP: [] icmp6_send+0x1fb/0x970 [57063.487218] PGD 0 [57063.487231] Oops: [#1] SMP [57063.488665] Call Trace: [57063.488679] [57063.488705] [] ? __netif_receive_skb+0x18/0x60 [57063.488739] [] ? task_tick_fair+0x4c8/0x8e0 [57063.488771] [] ? _raw_spin_unlock_bh+0x20/0x50 [57063.488802] [] icmpv6_send+0x21/0x30 [57063.488829] [] ip6_expire_frag_queue+0x115/0x1b0 [57063.488862] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.488897] [] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [57063.488937] [] call_timer_fn+0x37/0x140 [57063.488965] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.489002] [] run_timer_softirq+0x234/0x330 ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1883498/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
Happened more than once on separate VMs. ** Changed in: linux (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Confirmed Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump installed I obtained the dmesgs and crash dumps. The backtrace appears somewhat similar to #202669 but that one only happened on bare hardware for us - this one is on KVM virtual instances. #202669 paniced in icmpv6_route_lookup and this one dies already in icmpv6_send. Between 2020-06-11 and 2020-06-15, on a set of 12 VMs running 4.4.0-184-generic, there were 85 crashes like this, on servers with noticeable IPv6 traffic. All of the 12 VMs with 4.4.0-184-generic crashed at least once. (There are more than 12 VMs experiencing this, this is just the set I had linux-crashdump on.) [57063.487084] BUG: unable to handle kernel NULL pointer dereference at 0018 [57063.487184] IP: [] icmp6_send+0x1fb/0x970 [57063.487218] PGD 0 [57063.487231] Oops: [#1] SMP [57063.488665] Call Trace: [57063.488679] [57063.488705] [] ? __netif_receive_skb+0x18/0x60 [57063.488739] [] ? task_tick_fair+0x4c8/0x8e0 [57063.488771] [] ? _raw_spin_unlock_bh+0x20/0x50 [57063.488802] [] icmpv6_send+0x21/0x30 [57063.488829] [] ip6_expire_frag_queue+0x115/0x1b0 [57063.488862] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.488897] [] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [57063.488937] [] call_timer_fn+0x37/0x140 [57063.488965] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.489002] [] run_timer_softirq+0x234/0x330 ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1883498/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1883498] Re: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic
** Attachment added: "6 dmesg of #1883498 crashes on different VMs" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1883498/+attachment/5383933/+files/1883498-dmesgs-6.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1883498 Title: Frequent Panic in ip6_expire_frag_queue->icmpv6_send on 4.4.0-184-generic Status in linux package in Ubuntu: Confirmed Bug description: I happened to do an upgrade on a number of servers last week. Some of them got 4.4.0-179-generic and the ones upgraded a bit later during the week got 4.4.0-184-generic as it was just released. The ones with 4.4.0-184-generic started getting stuck. With linux-crashdump installed I obtained the dmesgs and crash dumps. The backtrace appears somewhat similar to #202669 but that one only happened on bare hardware for us - this one is on KVM virtual instances. #202669 paniced in icmpv6_route_lookup and this one dies already in icmpv6_send. Between 2020-06-11 and 2020-06-15, on a set of 12 VMs running 4.4.0-184-generic, there were 85 crashes like this, on servers with noticeable IPv6 traffic. All of the 12 VMs with 4.4.0-184-generic crashed at least once. (There are more than 12 VMs experiencing this, this is just the set I had linux-crashdump on.) [57063.487084] BUG: unable to handle kernel NULL pointer dereference at 0018 [57063.487184] IP: [] icmp6_send+0x1fb/0x970 [57063.487218] PGD 0 [57063.487231] Oops: [#1] SMP [57063.488665] Call Trace: [57063.488679] [57063.488705] [] ? __netif_receive_skb+0x18/0x60 [57063.488739] [] ? task_tick_fair+0x4c8/0x8e0 [57063.488771] [] ? _raw_spin_unlock_bh+0x20/0x50 [57063.488802] [] icmpv6_send+0x21/0x30 [57063.488829] [] ip6_expire_frag_queue+0x115/0x1b0 [57063.488862] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.488897] [] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [57063.488937] [] call_timer_fn+0x37/0x140 [57063.488965] [] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [57063.489002] [] run_timer_softirq+0x234/0x330 ... To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1883498/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
