We had another server running 4.4.0-148-generic crash just now. It has a different role then the firewalls that we originally saw the crash with. After an automatic reboot, it got back up with 4.4.0-150-generic (which had been installed at an earlier stage but not rebooted into), and crashed twice in rapid succession. I'm including the traces printed to the serial console below:
Crash #1: [760425.631142] ------------[ cut here ]------------ [760425.655558] kernel BUG at /build/linux-JhELCR/linux-4.4.0/net/core/skbuff.c:1207! [760425.698342] invalid opcode: 0000 [#1] SMP [760425.722139] Modules linked in: cpuid xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 bridge stp llc ebtable_filter ebtables hpwdt nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp xt_multiport xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip_tables x_tables bonding intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul gpio_ich ghash_clmulni_intel aesni_intel ftdi_sio aes_x86_64 lrw cdc_ether gf128mul usbserial usbnet mii glue_helper input_leds hpilo joydev ablk_helper i7core_edac cryptd edac_core shpchp serio_raw 8250_fintek acpi_power_meter lpc_ich ipmi_ssif mac_hid ipmi_devintf ipmi_si ipmi_msghandler nf_nat_h323 nf_conntrack_h323 nf_nat_sip nf_conntrack_sip nf_nat nf_conntrack lp parport autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ixgbe dca amdkfd amd_iommu_v2 radeon i2c_algo_bit hid_generic ttm drm_kms_helper syscopyarea sysfillrect vxlan ip6_udp_tunnel sysimgblt fb_sys_fops udp_tunnel usbhid uas ptp hpsa pps_core psmouse drm usb_storage hid bnx2 mdio scsi_transport_sas fjes [760426.325859] CPU: 9 PID: 0 Comm: swapper/9 Tainted: G I 4.4.0-148-generic #174-Ubuntu [760426.377154] Hardware name: HP ProLiant DL360 G7, BIOS P68 05/21/2018 [760426.413661] task: ffff880c0461d940 ti: ffff880c04634000 task.ti: ffff880c04634000 [760426.456870] RIP: 0010:[<ffffffff81740913>] [<ffffffff81740913>] pskb_expand_head+0x243/0x250 [760426.504553] RSP: 0018:ffff880c1f903b80 EFLAGS: 00010202 [760426.535588] RAX: 0000000000000002 RBX: ffff880c0007ed00 RCX: 0000000002080020 [760426.577271] RDX: 0000000000000140 RSI: 0000000000000000 RDI: ffff880c0007ed00 [760426.618845] RBP: ffff880c1f903bb8 R08: 0000000000000001 R09: 000000000000000a [760426.660361] R10: ffff880bffcf6a00 R11: 0000000000000000 R12: ffff880c0007ed00 [760426.702862] R13: 0000000000000001 R14: 0000000000000011 R15: ffff880c0007ed00 [760426.743881] FS: 0000000000000000(0000) GS:ffff880c1f900000(0000) knlGS:0000000000000000 [760426.789899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [760426.822597] CR2: 00007ffccac51ff8 CR3: 0000000001e0a000 CR4: 0000000000020670 [760426.863053] Stack: [760426.874826] ffffffff81efb140 ffff880c1f903db0 ffff880c0007ed00 ffff880c0007ed00 [760426.917091] 0000000000000001 0000000000000011 0000000000000001 ffff880c1f903c00 [760426.960119] ffffffff81740e10 ffff880c002d29c0 0000000100000001 ffff880c0007ed00 [760427.003239] Call Trace: [760427.017831] <IRQ> [760427.029143] [<ffffffff81740e10>] __pskb_pull_tail+0x50/0x350 [760427.064432] [<ffffffff818395ca>] _decode_session6+0x26a/0x400 [760427.097198] [<ffffffff817ec969>] __xfrm_decode_session+0x39/0x50 [760427.132864] [<ffffffff81823c00>] icmpv6_route_lookup+0xf0/0x1c0 [760427.168344] [<ffffffff81824651>] icmp6_send+0x5e1/0x940 [760427.198021] [<ffffffff810b11c4>] ? check_preempt_curr+0x54/0x90 [760427.232489] [<ffffffff810b138d>] ? ttwu_do_activate.constprop.87+0x5d/0x70 [760427.272075] [<ffffffff810b1e39>] ? try_to_wake_up+0x49/0x400 [760427.304593] [<ffffffffc05e4260>] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [760427.345205] [<ffffffff8183d661>] icmpv6_send+0x21/0x30 [760427.375673] [<ffffffff8182b730>] ip6_expire_frag_queue+0xe0/0x120 [760427.411187] [<ffffffffc05e427f>] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [760427.452645] [<ffffffff810f3d27>] call_timer_fn+0x37/0x140 [760427.486666] [<ffffffffc05e4260>] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [760427.532271] [<ffffffff810f5634>] run_timer_softirq+0x234/0x330 [760427.570814] [<ffffffff8108a3a9>] __do_softirq+0x109/0x2b0 [760427.608307] [<ffffffff8108a6c5>] irq_exit+0xa5/0xb0 [760427.641799] [<ffffffff81866300>] smp_apic_timer_interrupt+0x50/0x70 [760427.682907] [<ffffffff81863a7c>] apic_timer_interrupt+0xcc/0xe0 [760427.721035] <EOI> [760427.732319] [<ffffffff816f0b5b>] ? cpuidle_enter_state+0x11b/0x2d0 [760427.777027] [<ffffffff816f0d47>] cpuidle_enter+0x17/0x20 [760427.812803] [<ffffffff810ca6c2>] call_cpuidle+0x32/0x60 [760427.847096] [<ffffffff816f0d29>] ? cpuidle_select+0x19/0x20 [760427.883583] [<ffffffff810ca986>] cpu_startup_entry+0x296/0x360 [760427.921417] [<ffffffff81052fd7>] start_secondary+0x177/0x1b0 [760427.958469] Code: 75 1a 41 8b 87 cc 00 00 00 49 03 87 d0 00 00 00 e9 e2 fe ff ff b8 f4 ff ff ff eb bc 4c 89 ef e8 84 9c ab ff b8 f4 ff ff ff eb ad <0f> 0b 90 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 [760428.073481] RIP [<ffffffff81740913>] pskb_expand_head+0x243/0x250 [760428.112534] RSP <ffff880c1f903b80> [760428.151392] ---[ end trace ae05fc32238b1676 ]--- [760428.182175] Kernel panic - not syncing: Fatal exception in interrupt [760428.224421] Kernel Offset: disabled [760428.248096] Rebooting in 120 seconds.. Crash #2: [39110.998327] kernel BUG at /build/linux-MNTIf1/linux-4.4.0/net/core/skbuff.c:1207! [39111.041191] invalid opcode: 0000 [#1] SMP [39111.064589] Modules linked in: ebtable_filter ebtables hpwdt nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp xt_multiport xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip_tables x_tables bonding intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 cdc_ether usbnet mii gpio_ich lrw hpilo gf128mul glue_helper joydev input_leds serio_raw ablk_helper ftdi_sio usbserial shpchp i7core_edac cryptd lpc_ich edac_core acpi_power_meter 8250_fintek mac_hid ipmi_ssif nf_nat_h323 nf_conntrack_h323 nf_nat_sip nf_conntrack_sip nf_nat nf_conntrack lp parport ipmi_si ipmi_devintf ipmi_msghandler autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm ixgbe drm_kms_helper syscopyarea dca sysfillrect sysimgblt vxlan fb_sys_fops hid_generic ip6_udp_tunnel udp_tunnel uas hpsa ptp usbhid pps_core hid drm usb_storage psmouse bnx2 mdio scsi_transport_sas fjes [39111.627243] CPU: 15 PID: 0 Comm: swapper/15 Tainted: G I 4.4.0-150-generic #176-Ubuntu [39111.679343] Hardware name: HP ProLiant DL360 G7, BIOS P68 05/21/2018 [39111.715980] task: ffff880c0461bfc0 ti: ffff880c04638000 task.ti: ffff880c04638000 [39111.757935] RIP: 0010:[<ffffffff81741913>] [<ffffffff81741913>] pskb_expand_head+0x243/0x250 [39111.805433] RSP: 0018:ffff880c1f9c3b80 EFLAGS: 00010202 [39111.835472] RAX: 0000000000000002 RBX: ffff880bffd31700 RCX: 0000000002080020 [39111.876070] RDX: 0000000000000140 RSI: 0000000000000000 RDI: ffff880bffd31700 [39111.916174] RBP: ffff880c1f9c3bb8 R08: 0000000000000001 R09: 000000000000000a [39111.956683] R10: ffff880bff532400 R11: 0000000000000000 R12: ffff880bffd31700 [39111.997285] R13: 0000000000000001 R14: 0000000000000011 R15: ffff880bffd31700 [39112.038416] FS: 0000000000000000(0000) GS:ffff880c1f9c0000(0000) knlGS:0000000000000000 [39112.085431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39112.118368] CR2: 00007ff5fab530a0 CR3: 0000000001e0a000 CR4: 0000000000020670 [39112.158872] Stack: [39112.170355] ffffffff81efb340 ffff880c1f9c3db0 ffff880bffd31700 ffff880bffd31700 [39112.211928] 0000000000000001 0000000000000011 0000000000000001 ffff880c1f9c3c00 [39112.253396] ffffffff81741e10 ffff880c00329380 0000000100000001 ffff880bffd31700 [39112.294511] Call Trace: [39112.307681] <IRQ> [39112.318514] [<ffffffff81741e10>] __pskb_pull_tail+0x50/0x350 [39112.351053] [<ffffffff8183a5ea>] _decode_session6+0x26a/0x400 [39112.384719] [<ffffffff817ed979>] __xfrm_decode_session+0x39/0x50 [39112.419150] [<ffffffff81824c40>] icmpv6_route_lookup+0xf0/0x1c0 [39112.453211] [<ffffffff81825681>] icmp6_send+0x5e1/0x940 [39112.481827] [<ffffffff810b1274>] ? check_preempt_curr+0x54/0x90 [39112.514398] [<ffffffff810b143d>] ? ttwu_do_activate.constprop.87+0x5d/0x70 [39112.554439] [<ffffffff810b1ee9>] ? try_to_wake_up+0x49/0x400 [39112.587377] [<ffffffffc05cd260>] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [39112.628369] [<ffffffff8183e681>] icmpv6_send+0x21/0x30 [39112.658438] [<ffffffff8182c760>] ip6_expire_frag_queue+0xe0/0x120 [39112.693975] [<ffffffffc05cd27f>] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [39112.735608] [<ffffffff810f3ea7>] call_timer_fn+0x37/0x140 [39112.766469] [<ffffffffc05cd260>] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [39112.810508] [<ffffffff810f57b4>] run_timer_softirq+0x234/0x330 [39112.849601] [<ffffffff8108a459>] __do_softirq+0x109/0x2b0 [39112.885588] [<ffffffff8108a775>] irq_exit+0xa5/0xb0 [39112.918595] [<ffffffff81867340>] smp_apic_timer_interrupt+0x50/0x70 [39112.959744] [<ffffffff81864abc>] apic_timer_interrupt+0xcc/0xe0 [39112.998577] <EOI> [39113.009734] [<ffffffff816f114b>] ? cpuidle_enter_state+0x11b/0x2d0 [39113.055103] [<ffffffff816f1337>] cpuidle_enter+0x17/0x20 [39113.090562] [<ffffffff810ca832>] call_cpuidle+0x32/0x60 [39113.124525] [<ffffffff816f1319>] ? cpuidle_select+0x19/0x20 [39113.158764] [<ffffffff810caafb>] cpu_startup_entry+0x29b/0x360 [39113.195332] [<ffffffff81052fd7>] start_secondary+0x177/0x1b0 [39113.232358] Code: 75 1a 41 8b 87 cc 00 00 00 49 03 87 d0 00 00 00 e9 e2 fe ff ff b8 f4 ff ff ff eb bc 4c 89 ef e8 54 8f ab ff b8 f4 ff ff ff eb ad <0f> 0b 90 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 [39113.346392] RIP [<ffffffff81741913>] pskb_expand_head+0x243/0x250 [39113.385395] RSP <ffff880c1f9c3b80> [39113.423817] ---[ end trace 5f54ccb09105271f ]--- [39113.453455] Kernel panic - not syncing: Fatal exception in interrupt [39113.494218] Kernel Offset: disabled [39113.517452] Rebooting in 120 seconds.. Crash #3: [ 405.191527] kernel BUG at /build/linux-MNTIf1/linux-4.4.0/net/core/skbuff.c:1207! [ 405.234393] invalid opcode: 0000 [#1] SMP [ 405.257924] Modules linked in: ebtable_filter ebtables hpwdt nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp xt_multiport xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip_tables x_tables bonding intel_powerclamp coretemp kvm_intel kvm gpio_ich ftdi_sio irqbypass joydev hpilo input_leds crct10dif_pclmul usbserial crc32_pclmul ghash_clmulni_intel aesni_intel cdc_ether shpchp aes_x86_64 usbnet lrw gf128mul mii glue_helper lpc_ich serio_raw ablk_helper i7core_edac cryptd acpi_power_meter edac_core 8250_fintek ipmi_ssif mac_hid nf_nat_h323 nf_conntrack_h323 nf_nat_sip nf_conntrack_sip nf_nat nf_conntrack lp parport ipmi_si ipmi_devintf ipmi_msghandler autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm ixgbe drm_kms_helper dca syscopyarea vxlan hid_generic sysfillrect ip6_udp_tunnel sysimgblt fb_sys_fops uas udp_tunnel ptp usbhid hpsa pps_core usb_storage psmouse drm hid bnx2 mdio scsi_transport_sas fjes [ 405.821444] CPU: 14 PID: 0 Comm: swapper/14 Tainted: G I 4.4.0-150-generic #176-Ubuntu [ 405.872473] Hardware name: HP ProLiant DL360 G7, BIOS P68 05/21/2018 [ 405.908454] task: ffff8806043d5940 ti: ffff8806043ec000 task.ti: ffff8806043ec000 [ 405.950727] RIP: 0010:[<ffffffff81741913>] [<ffffffff81741913>] pskb_expand_head+0x243/0x250 [ 405.997454] RSP: 0018:ffff880607bc3b80 EFLAGS: 00010202 [ 406.027625] RAX: 0000000000000002 RBX: ffff880603317600 RCX: 0000000002080020 [ 406.068879] RDX: 0000000000000140 RSI: 0000000000000000 RDI: ffff880603317600 [ 406.108894] RBP: ffff880607bc3bb8 R08: 0000000000000001 R09: 000000000000000a [ 406.149770] R10: ffff8805f8730780 R11: 0000000000000000 R12: ffff880603317600 [ 406.188408] R13: 0000000000000001 R14: 0000000000000011 R15: ffff880603317600 [ 406.226453] FS: 0000000000000000(0000) GS:ffff880607bc0000(0000) knlGS:0000000000000000 [ 406.272941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 406.305408] CR2: 00007ffd0b6fd5e8 CR3: 0000000001e0a000 CR4: 0000000000020670 [ 406.346469] Stack: [ 406.357952] ffffffff81efb340 ffff880607bc3db0 ffff880603317600 ffff880603317600 [ 406.400887] 0000000000000001 0000000000000011 0000000000000001 ffff880607bc3c00 [ 406.442902] ffffffff81741e10 ffff880600790240 0000000100000001 ffff880603317600 [ 406.485281] Call Trace: [ 406.499010] <IRQ> [ 406.510076] [<ffffffff81741e10>] __pskb_pull_tail+0x50/0x350 [ 406.542794] [<ffffffff8183a5ea>] _decode_session6+0x26a/0x400 [ 406.575744] [<ffffffff817ed979>] __xfrm_decode_session+0x39/0x50 [ 406.610309] [<ffffffff81824c40>] icmpv6_route_lookup+0xf0/0x1c0 [ 406.644299] [<ffffffff81825681>] icmp6_send+0x5e1/0x940 [ 406.674827] [<ffffffffc05f0260>] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [ 406.715847] [<ffffffff8183e681>] icmpv6_send+0x21/0x30 [ 406.745961] [<ffffffff8182c760>] ip6_expire_frag_queue+0xe0/0x120 [ 406.781422] [<ffffffffc05f027f>] nf_ct_frag6_expire+0x1f/0x30 [nf_defrag_ipv6] [ 406.823205] [<ffffffff810f3ea7>] call_timer_fn+0x37/0x140 [ 406.856020] [<ffffffffc05f0260>] ? nf_ct_net_exit+0x50/0x50 [nf_defrag_ipv6] [ 406.896513] [<ffffffff810f57b4>] run_timer_softirq+0x234/0x330 [ 406.929577] [<ffffffff8108a459>] __do_softirq+0x109/0x2b0 [ 406.960324] [<ffffffff8108a775>] irq_exit+0xa5/0xb0 [ 406.991931] [<ffffffff81867340>] smp_apic_timer_interrupt+0x50/0x70 [ 407.030828] [<ffffffff81864abc>] apic_timer_interrupt+0xcc/0xe0 [ 407.069200] <EOI> [ 407.080245] [<ffffffff816f114b>] ? cpuidle_enter_state+0x11b/0x2d0 [ 407.126744] [<ffffffff816f1337>] cpuidle_enter+0x17/0x20 [ 407.160670] [<ffffffff810ca832>] call_cpuidle+0x32/0x60 [ 407.194725] [<ffffffff816f1319>] ? cpuidle_select+0x19/0x20 [ 407.231690] [<ffffffff810caafb>] cpu_startup_entry+0x29b/0x360 [ 407.269708] [<ffffffff81052fd7>] start_secondary+0x177/0x1b0 [ 407.306947] Code: 75 1a 41 8b 87 cc 00 00 00 49 03 87 d0 00 00 00 e9 e2 fe ff ff b8 f4 ff ff ff eb bc 4c 89 ef e8 54 8f ab ff b8 f4 ff ff ff eb ad <0f> 0b 90 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 [ 407.420898] RIP [<ffffffff81741913>] pskb_expand_head+0x243/0x250 [ 407.458621] RSP <ffff880607bc3b80> [ 407.482590] ---[ end trace 3e762511d34f9a0d ]--- [ 407.512053] Kernel panic - not syncing: Fatal exception in interrupt [ 407.552302] Kernel Offset: disabled [ 407.575889] Rebooting in 120 seconds.. Downgraded to 4.4.0-143-generic, so far so good. Tore -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828345 Title: [4.4.0-144 regression] kernel BUG at [...]/linux-lts- xenial-4.4.0/net/core/skbuff.c:1207 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-meta-lts-xenial/+bug/1828345/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
