[Kernel-packages] [Bug 1377851] Re: Kernel panic skb_segment+0x5d7/0x980
No crashes so far with 3.13.0-38-generic, looks like it's fixed. ** Tags removed: verification-needed-trusty ** Tags added: verification-done-trusty -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1377851 Title: Kernel panic skb_segment+0x5d7/0x980 Status in “linux” package in Ubuntu: Fix Released Status in “linux” source package in Trusty: Fix Committed Status in “linux” source package in Utopic: Fix Released Bug description: On two Ubuntu 14.04 amd64 servers with tg3 NICs acting as a openvpn gateway we recently had a lot of trouble with kernel panics (linux-image-3.13.0-36-generic 3.13.0-36.63) The panics were kind of random happening sometimes already during the boot process and sometimes a couple of hours later. The boxes were running perfectly find for a couple of months before. We believe some kind of special packet triggered the bug. Potential related bugs: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331219 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1313591 Upgrading to linux-image-3.16.3-031603-generic (3.16.3-031603.201409171435) solved the problem for us. [ 6076.726520] BUG: unable to handle kernel NULL pointer dereference at 006c [ 6076.737716] IP: [81616787] skb_segment+0x5d7/0x980 [ 6076.745780] PGD 0 [ 6076.748641] Oops: [#1] SMP [ 6076.753268] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c cdc_ether usbnet mii mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase ipmi_si ipmi_devintf dell_rbu gpio_ich intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp dcdbas kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd 8021q garp stp mrp llc sb_edac edac_core shpchp joydev pl2303 usbserial lpc_ich wmi mei_me mei mac_hid acpi_power_meter ioatdma nf_conntrack dca lp parport raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 tg3 ahci hid_generic raid0 ptp usbhid multipath hid libahci pps_core linear [last unloaded: ipmi_si] [ 6076.850743] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.13.0-36-generic #63-Ubuntu [ 6076.861485] Hardware name: Dell Inc. PowerEdge R620/0PXXHP, BIOS 1.6.0 03/07/2013 [ 6076.872104] task: 880223841800 ti: 880223848000 task.ti: 880223848000 [ 6076.882722] RIP: 0010:[81616787] [81616787] skb_segment+0x5d7/0x980 [ 6076.894243] RSP: 0018:880227263790 EFLAGS: 00010246 [ 6076.901769] RAX: 0646 RBX: 88021f03f000 RCX: 8801ed4fff00 [ 6076.911893] RDX: 0646 RSI: 00c2 RDI: ea0007f6de00 [ 6076.971665] RBP: 880227263858 R08: fff6 R09: 0001 [ 6077.031463] R10: 88021f03e800 R11: 00010552 R12: 8801fdb1fc80 [ 6077.091313] R13: 0001 R14: R15: 0646 [ 6077.151619] FS: () GS:88022726() knlGS: [ 6077.262424] CS: 0010 DS: ES: CR0: 80050033 [ 6077.320083] CR2: 006c CR3: 01c0e000 CR4: 000407e0 [ 6077.379854] Stack: [ 6077.433555] 811a48f9 8802272772c0 fff6 000a [ 6077.546280] 00010552 006a 88021f03e800 00010020 [ 6077.658915] ffe4 00010012 001c055c 88021f03f000 [ 6077.771400] Call Trace: [ 6077.824494] IRQ [ 6077.827251] [811a48f9] ? __kmalloc_node_track_caller+0xb9/0x290 [ 6077.933211] [8168149d] tcp_gso_segment+0x10d/0x3f0 [ 6077.988641] [81691822] inet_gso_segment+0x132/0x360 [ 6078.043154] [810a5db2] ? enqueue_task_fair+0x422/0x6c0 [ 6078.097358] [81623ffc] skb_mac_gso_segment+0x9c/0x180 [ 6078.150464] [816a0fb4] gre_gso_segment+0x134/0x370 [ 6078.202321] [8109828d] ? ttwu_do_activate.constprop.74+0x5d/0x70 [ 6078.255348] [81691822] inet_gso_segment+0x132/0x360 [ 6078.306129] [8109a800] ? try_to_wake_up+0x240/0x2c0 [ 6078.355712] [81623ffc] skb_mac_gso_segment+0x9c/0x180 [ 6078.404660] [8162413d] __skb_gso_segment+0x5d/0xb0 [ 6078.452918] [8162444a] dev_hard_start_xmit+0x18a/0x560 [ 6078.501057] [8164360e] sch_direct_xmit+0xee/0x1c0 [ 6078.548821] [81624a50] __dev_queue_xmit+0x230/0x500 [ 6078.596793] [81624d30] dev_queue_xmit+0x10/0x20 [ 6078.644041] [8162be31] neigh_direct_output+0x11/0x20 [ 6078.691822] [8165d370] ip_finish_output+0x1b0/0x3b0 [ 6078.739211] [8165e8d8] ip_output+0x58/0x90 [ 6078.784448] [8165a84b] ip_forward_finish+0x8b/0x170 [ 6078.830211] [8165ac85] ip_forward+0x355/0x410 [
[Kernel-packages] [Bug 1377851] Re: Kernel panic skb_segment+0x5d7/0x980
We installed 3.13.0-38-generic from proposed during a maintenance today on one of the initially affected server. We'll provide an update soon. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1377851 Title: Kernel panic skb_segment+0x5d7/0x980 Status in “linux” package in Ubuntu: Fix Released Status in “linux” source package in Trusty: Fix Committed Status in “linux” source package in Utopic: Fix Released Bug description: On two Ubuntu 14.04 amd64 servers with tg3 NICs acting as a openvpn gateway we recently had a lot of trouble with kernel panics (linux-image-3.13.0-36-generic 3.13.0-36.63) The panics were kind of random happening sometimes already during the boot process and sometimes a couple of hours later. The boxes were running perfectly find for a couple of months before. We believe some kind of special packet triggered the bug. Potential related bugs: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331219 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1313591 Upgrading to linux-image-3.16.3-031603-generic (3.16.3-031603.201409171435) solved the problem for us. [ 6076.726520] BUG: unable to handle kernel NULL pointer dereference at 006c [ 6076.737716] IP: [81616787] skb_segment+0x5d7/0x980 [ 6076.745780] PGD 0 [ 6076.748641] Oops: [#1] SMP [ 6076.753268] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c cdc_ether usbnet mii mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase ipmi_si ipmi_devintf dell_rbu gpio_ich intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp dcdbas kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd 8021q garp stp mrp llc sb_edac edac_core shpchp joydev pl2303 usbserial lpc_ich wmi mei_me mei mac_hid acpi_power_meter ioatdma nf_conntrack dca lp parport raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 tg3 ahci hid_generic raid0 ptp usbhid multipath hid libahci pps_core linear [last unloaded: ipmi_si] [ 6076.850743] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.13.0-36-generic #63-Ubuntu [ 6076.861485] Hardware name: Dell Inc. PowerEdge R620/0PXXHP, BIOS 1.6.0 03/07/2013 [ 6076.872104] task: 880223841800 ti: 880223848000 task.ti: 880223848000 [ 6076.882722] RIP: 0010:[81616787] [81616787] skb_segment+0x5d7/0x980 [ 6076.894243] RSP: 0018:880227263790 EFLAGS: 00010246 [ 6076.901769] RAX: 0646 RBX: 88021f03f000 RCX: 8801ed4fff00 [ 6076.911893] RDX: 0646 RSI: 00c2 RDI: ea0007f6de00 [ 6076.971665] RBP: 880227263858 R08: fff6 R09: 0001 [ 6077.031463] R10: 88021f03e800 R11: 00010552 R12: 8801fdb1fc80 [ 6077.091313] R13: 0001 R14: R15: 0646 [ 6077.151619] FS: () GS:88022726() knlGS: [ 6077.262424] CS: 0010 DS: ES: CR0: 80050033 [ 6077.320083] CR2: 006c CR3: 01c0e000 CR4: 000407e0 [ 6077.379854] Stack: [ 6077.433555] 811a48f9 8802272772c0 fff6 000a [ 6077.546280] 00010552 006a 88021f03e800 00010020 [ 6077.658915] ffe4 00010012 001c055c 88021f03f000 [ 6077.771400] Call Trace: [ 6077.824494] IRQ [ 6077.827251] [811a48f9] ? __kmalloc_node_track_caller+0xb9/0x290 [ 6077.933211] [8168149d] tcp_gso_segment+0x10d/0x3f0 [ 6077.988641] [81691822] inet_gso_segment+0x132/0x360 [ 6078.043154] [810a5db2] ? enqueue_task_fair+0x422/0x6c0 [ 6078.097358] [81623ffc] skb_mac_gso_segment+0x9c/0x180 [ 6078.150464] [816a0fb4] gre_gso_segment+0x134/0x370 [ 6078.202321] [8109828d] ? ttwu_do_activate.constprop.74+0x5d/0x70 [ 6078.255348] [81691822] inet_gso_segment+0x132/0x360 [ 6078.306129] [8109a800] ? try_to_wake_up+0x240/0x2c0 [ 6078.355712] [81623ffc] skb_mac_gso_segment+0x9c/0x180 [ 6078.404660] [8162413d] __skb_gso_segment+0x5d/0xb0 [ 6078.452918] [8162444a] dev_hard_start_xmit+0x18a/0x560 [ 6078.501057] [8164360e] sch_direct_xmit+0xee/0x1c0 [ 6078.548821] [81624a50] __dev_queue_xmit+0x230/0x500 [ 6078.596793] [81624d30] dev_queue_xmit+0x10/0x20 [ 6078.644041] [8162be31] neigh_direct_output+0x11/0x20 [ 6078.691822] [8165d370] ip_finish_output+0x1b0/0x3b0 [ 6078.739211] [8165e8d8] ip_output+0x58/0x90 [ 6078.784448] [8165a84b] ip_forward_finish+0x8b/0x170 [ 6078.830211] [8165ac85] ip_forward+0x355/0x410 [
[Kernel-packages] [Bug 1377851] Re: Kernel panic skb_segment+0x5d7/0x980
Can't test with the 3.13 mainline as these are productive servers and we can't risk another outage at this time. As mentioned in the first post it in appears to be fixed in the 3.16 mainline kernel. ** Tags added: kernel-unable-to-test-upstream ** Changed in: linux (Ubuntu) Status: Incomplete = Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1377851 Title: Kernel panic skb_segment+0x5d7/0x980 Status in “linux” package in Ubuntu: Confirmed Bug description: On two Ubuntu 14.04 amd64 servers with tg3 NICs acting as a openvpn gateway we recently had a lot of trouble with kernel panics (linux-image-3.13.0-36-generic 3.13.0-36.63) The panics were kind of random happening sometimes already during the boot process and sometimes a couple of hours later. The boxes were running perfectly find for a couple of months before. We believe some kind of special packet triggered the bug. Potential related bugs: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331219 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1313591 Upgrading to linux-image-3.16.3-031603-generic (3.16.3-031603.201409171435) solved the problem for us. [ 6076.726520] BUG: unable to handle kernel NULL pointer dereference at 006c [ 6076.737716] IP: [81616787] skb_segment+0x5d7/0x980 [ 6076.745780] PGD 0 [ 6076.748641] Oops: [#1] SMP [ 6076.753268] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c cdc_ether usbnet mii mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase ipmi_si ipmi_devintf dell_rbu gpio_ich intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp dcdbas kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd 8021q garp stp mrp llc sb_edac edac_core shpchp joydev pl2303 usbserial lpc_ich wmi mei_me mei mac_hid acpi_power_meter ioatdma nf_conntrack dca lp parport raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 tg3 ahci hid_generic raid0 ptp usbhid multipath hid libahci pps_core linear [last unloaded: ipmi_si] [ 6076.850743] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.13.0-36-generic #63-Ubuntu [ 6076.861485] Hardware name: Dell Inc. PowerEdge R620/0PXXHP, BIOS 1.6.0 03/07/2013 [ 6076.872104] task: 880223841800 ti: 880223848000 task.ti: 880223848000 [ 6076.882722] RIP: 0010:[81616787] [81616787] skb_segment+0x5d7/0x980 [ 6076.894243] RSP: 0018:880227263790 EFLAGS: 00010246 [ 6076.901769] RAX: 0646 RBX: 88021f03f000 RCX: 8801ed4fff00 [ 6076.911893] RDX: 0646 RSI: 00c2 RDI: ea0007f6de00 [ 6076.971665] RBP: 880227263858 R08: fff6 R09: 0001 [ 6077.031463] R10: 88021f03e800 R11: 00010552 R12: 8801fdb1fc80 [ 6077.091313] R13: 0001 R14: R15: 0646 [ 6077.151619] FS: () GS:88022726() knlGS: [ 6077.262424] CS: 0010 DS: ES: CR0: 80050033 [ 6077.320083] CR2: 006c CR3: 01c0e000 CR4: 000407e0 [ 6077.379854] Stack: [ 6077.433555] 811a48f9 8802272772c0 fff6 000a [ 6077.546280] 00010552 006a 88021f03e800 00010020 [ 6077.658915] ffe4 00010012 001c055c 88021f03f000 [ 6077.771400] Call Trace: [ 6077.824494] IRQ [ 6077.827251] [811a48f9] ? __kmalloc_node_track_caller+0xb9/0x290 [ 6077.933211] [8168149d] tcp_gso_segment+0x10d/0x3f0 [ 6077.988641] [81691822] inet_gso_segment+0x132/0x360 [ 6078.043154] [810a5db2] ? enqueue_task_fair+0x422/0x6c0 [ 6078.097358] [81623ffc] skb_mac_gso_segment+0x9c/0x180 [ 6078.150464] [816a0fb4] gre_gso_segment+0x134/0x370 [ 6078.202321] [8109828d] ? ttwu_do_activate.constprop.74+0x5d/0x70 [ 6078.255348] [81691822] inet_gso_segment+0x132/0x360 [ 6078.306129] [8109a800] ? try_to_wake_up+0x240/0x2c0 [ 6078.355712] [81623ffc] skb_mac_gso_segment+0x9c/0x180 [ 6078.404660] [8162413d] __skb_gso_segment+0x5d/0xb0 [ 6078.452918] [8162444a] dev_hard_start_xmit+0x18a/0x560 [ 6078.501057] [8164360e] sch_direct_xmit+0xee/0x1c0 [ 6078.548821] [81624a50] __dev_queue_xmit+0x230/0x500 [ 6078.596793] [81624d30] dev_queue_xmit+0x10/0x20 [ 6078.644041] [8162be31] neigh_direct_output+0x11/0x20 [ 6078.691822] [8165d370] ip_finish_output+0x1b0/0x3b0 [ 6078.739211] [8165e8d8] ip_output+0x58/0x90 [ 6078.784448] [8165a84b] ip_forward_finish+0x8b/0x170 [ 6078.830211]
[Kernel-packages] [Bug 1377851] Re: Kernel panic skb_segment+0x5d7/0x980
backtrace already posted ** Changed in: linux (Ubuntu) Status: Incomplete = Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1377851 Title: Kernel panic skb_segment+0x5d7/0x980 Status in “linux” package in Ubuntu: Confirmed Bug description: On two Ubuntu 14.04 amd64 servers with tg3 NICs acting as a openvpn gateway we recently had a lot of trouble with kernel panics (linux-image-3.13.0-36-generic 3.13.0-36.63) The panics were kind of random happening sometimes already during the boot process and sometimes a couple of hours later. The boxes were running perfectly find for a couple of months before. We believe some kind of special packet triggered the bug. Potential related bugs: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331219 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1313591 Upgrading to linux-image-3.16.3-031603-generic (3.16.3-031603.201409171435) solved the problem for us. [ 6076.726520] BUG: unable to handle kernel NULL pointer dereference at 006c [ 6076.737716] IP: [81616787] skb_segment+0x5d7/0x980 [ 6076.745780] PGD 0 [ 6076.748641] Oops: [#1] SMP [ 6076.753268] Modules linked in: btrfs ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c cdc_ether usbnet mii mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase ipmi_si ipmi_devintf dell_rbu gpio_ich intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp dcdbas kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd 8021q garp stp mrp llc sb_edac edac_core shpchp joydev pl2303 usbserial lpc_ich wmi mei_me mei mac_hid acpi_power_meter ioatdma nf_conntrack dca lp parport raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 tg3 ahci hid_generic raid0 ptp usbhid multipath hid libahci pps_core linear [last unloaded: ipmi_si] [ 6076.850743] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.13.0-36-generic #63-Ubuntu [ 6076.861485] Hardware name: Dell Inc. PowerEdge R620/0PXXHP, BIOS 1.6.0 03/07/2013 [ 6076.872104] task: 880223841800 ti: 880223848000 task.ti: 880223848000 [ 6076.882722] RIP: 0010:[81616787] [81616787] skb_segment+0x5d7/0x980 [ 6076.894243] RSP: 0018:880227263790 EFLAGS: 00010246 [ 6076.901769] RAX: 0646 RBX: 88021f03f000 RCX: 8801ed4fff00 [ 6076.911893] RDX: 0646 RSI: 00c2 RDI: ea0007f6de00 [ 6076.971665] RBP: 880227263858 R08: fff6 R09: 0001 [ 6077.031463] R10: 88021f03e800 R11: 00010552 R12: 8801fdb1fc80 [ 6077.091313] R13: 0001 R14: R15: 0646 [ 6077.151619] FS: () GS:88022726() knlGS: [ 6077.262424] CS: 0010 DS: ES: CR0: 80050033 [ 6077.320083] CR2: 006c CR3: 01c0e000 CR4: 000407e0 [ 6077.379854] Stack: [ 6077.433555] 811a48f9 8802272772c0 fff6 000a [ 6077.546280] 00010552 006a 88021f03e800 00010020 [ 6077.658915] ffe4 00010012 001c055c 88021f03f000 [ 6077.771400] Call Trace: [ 6077.824494] IRQ [ 6077.827251] [811a48f9] ? __kmalloc_node_track_caller+0xb9/0x290 [ 6077.933211] [8168149d] tcp_gso_segment+0x10d/0x3f0 [ 6077.988641] [81691822] inet_gso_segment+0x132/0x360 [ 6078.043154] [810a5db2] ? enqueue_task_fair+0x422/0x6c0 [ 6078.097358] [81623ffc] skb_mac_gso_segment+0x9c/0x180 [ 6078.150464] [816a0fb4] gre_gso_segment+0x134/0x370 [ 6078.202321] [8109828d] ? ttwu_do_activate.constprop.74+0x5d/0x70 [ 6078.255348] [81691822] inet_gso_segment+0x132/0x360 [ 6078.306129] [8109a800] ? try_to_wake_up+0x240/0x2c0 [ 6078.355712] [81623ffc] skb_mac_gso_segment+0x9c/0x180 [ 6078.404660] [8162413d] __skb_gso_segment+0x5d/0xb0 [ 6078.452918] [8162444a] dev_hard_start_xmit+0x18a/0x560 [ 6078.501057] [8164360e] sch_direct_xmit+0xee/0x1c0 [ 6078.548821] [81624a50] __dev_queue_xmit+0x230/0x500 [ 6078.596793] [81624d30] dev_queue_xmit+0x10/0x20 [ 6078.644041] [8162be31] neigh_direct_output+0x11/0x20 [ 6078.691822] [8165d370] ip_finish_output+0x1b0/0x3b0 [ 6078.739211] [8165e8d8] ip_output+0x58/0x90 [ 6078.784448] [8165a84b] ip_forward_finish+0x8b/0x170 [ 6078.830211] [8165ac85] ip_forward+0x355/0x410 [ 6078.874484] [8165899d] ip_rcv_finish+0x7d/0x350 [ 6078.918046] [816592e8] ip_rcv+0x298/0x3d0 [ 6078.959829] [81622bb6]
