[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
This bug was fixed in the package linux - 3.13.0-109.156
---
linux (3.13.0-109.156) trusty; urgency=low
[ Thadeu Lima de Souza Cascardo ]
* Release Tracking Bug
- LP: #1662186
[ Luis Henriques ]
* Backport Dirty COW patch to prevent wineserver freeze (LP: #1658270)
- ARM: 7985/1: mm: implement pte_accessible for faulting mappings
- ARM: 8108/1: mm: Introduce {pte,pmd}_isset and {pte,pmd}_isclear
- ARM: 8037/1: mm: support big-endian page tables
- ARM: 8109/1: mm: Modify pte_write and pmd_write logic for LPAE
- arm64: mm: Route pmd thp functions through pte equivalents
- mm: fix huge zero page accounting in smaps report
- SAUCE: mm: Respect FOLL_FORCE/FOLL_COW for thp
* kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch] (LP: #1655683)
- SAUCE: openvswitch: gre: filter gre packets
* CVE-2016-7911
- block: fix use-after-free in sys_ioprio_get()
* CVE-2016-7910
- block: fix use-after-free in seq file
* Xen MSI setup code incorrectly re-uses cached pirq (LP: #1656381)
- SAUCE: xen: do not re-use pirq number cached in pci device msi msg data
-- Thadeu Lima de Souza Cascardo Tue, 07 Feb
2017 09:26:42 -0200
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Trusty:
Fix Released
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
This bug was fixed in the package linux - 3.13.0-109.156
---
linux (3.13.0-109.156) trusty; urgency=low
[ Thadeu Lima de Souza Cascardo ]
* Release Tracking Bug
- LP: #1662186
[ Luis Henriques ]
* Backport Dirty COW patch to prevent wineserver freeze (LP: #1658270)
- ARM: 7985/1: mm: implement pte_accessible for faulting mappings
- ARM: 8108/1: mm: Introduce {pte,pmd}_isset and {pte,pmd}_isclear
- ARM: 8037/1: mm: support big-endian page tables
- ARM: 8109/1: mm: Modify pte_write and pmd_write logic for LPAE
- arm64: mm: Route pmd thp functions through pte equivalents
- mm: fix huge zero page accounting in smaps report
- SAUCE: mm: Respect FOLL_FORCE/FOLL_COW for thp
* kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch] (LP: #1655683)
- SAUCE: openvswitch: gre: filter gre packets
* CVE-2016-7911
- block: fix use-after-free in sys_ioprio_get()
* CVE-2016-7910
- block: fix use-after-free in seq file
* Xen MSI setup code incorrectly re-uses cached pirq (LP: #1656381)
- SAUCE: xen: do not re-use pirq number cached in pci device msi msg data
-- Thadeu Lima de Souza Cascardo Tue, 07 Feb
2017 09:26:42 -0200
** Changed in: linux (Ubuntu Trusty)
Status: Fix Committed => Fix Released
** CVE added: http://www.cve.mitre.org/cgi-
bin/cvename.cgi?name=2016-7910
** CVE added: http://www.cve.mitre.org/cgi-
bin/cvename.cgi?name=2016-7911
** Changed in: linux (Ubuntu Trusty)
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Trusty:
Fix Released
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
Hello Andrew,
Can you help us to verify this?
Thank you!
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Trusty:
Fix Committed
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
/build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables veth
openvswitch gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich cdc_ether
x86_pkg_temp_thermal intel_powerclamp coretemp usbnet kvm_intel mii kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw
gf128mul glue_helper ablk_helper cryptd sb_edac edac_core lpc_ich wmi ipmi_si
bonding shpchp ioatdma lp mac_hid parport ahci libahci sfc igb e1000e mtd dca
i2c_algo_bit ptp pps_core megaraid_sas mdio
> [33100.133560] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.13.0-106-g
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
This bug is awaiting verification that the kernel in -proposed solves
the problem. Please test the kernel and update this bug with the
results. If the problem is solved, change the tag 'verification-needed-
trusty' to 'verification-done-trusty'. If the problem still exists,
change the tag 'verification-needed-trusty' to 'verification-failed-
trusty'.
If verification is not done by 5 working days from today, this fix will
be dropped from the source code, and this bug will be closed.
See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how
to enable and use -proposed. Thank you!
** Tags added: verification-needed-trusty
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Trusty:
Fix Committed
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
/build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_co
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
This patch should be released with 3.13.0-109.156
** Changed in: linux (Ubuntu Trusty)
Status: In Progress => Fix Committed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Trusty:
Fix Committed
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
/build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables veth
openvswitch gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich cdc_ether
x86_pkg_temp_thermal intel_powerclamp coretemp usbnet kvm_intel mii kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw
gf128mul glue_helper ablk_helper cryptd sb_edac edac_core lpc_ich wmi ipmi_si
bonding shpchp ioatdma lp mac_hid parport ahci libahci sfc igb e1000e mtd dca
i2c_algo_bit ptp pps_core megaraid_sas mdio
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
Thanks Tim for the clarification. I will not make any more status
changes.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Trusty:
In Progress
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
/build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables veth
openvswitch gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich cdc_ether
x86_pkg_temp_thermal intel_powerclamp coretemp usbnet kvm_intel mii kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw
gf128mul glue_helper ablk_helper cryptd sb_edac edac_core lpc_ich wmi ipmi_si
bonding shpchp ioatdma lp mac_hid parport ahci libahci sfc igb e1000e mtd dca
i2c_algo_bit ptp pps_core megaraid_sas mdio
> [33100.133560] CPU: 2 PID: 0 Comm: swapper/2 Not tainte
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
This patch has not been committed to the Trusty repository. It has been
reviewed on the kernel team mail list and should be committed soon.
** Changed in: linux (Ubuntu Trusty)
Status: Fix Committed => In Progress
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Trusty:
In Progress
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
/build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables veth
openvswitch gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich cdc_ether
x86_pkg_temp_thermal intel_powerclamp coretemp usbnet kvm_intel mii kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw
gf128mul glue_helper ablk_helper cryptd sb_edac edac_core lpc_ich wmi ipmi_si
bonding shpchp ioatdma lp mac_hid p
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
I have moved this to "fix committed", can someone verify that this patch
will be backported to the Trusty 3.13 series kernel, or do I need to
take any additional steps? I am not sure what my responsibility is in
moving this forward from here.
** Changed in: linux (Ubuntu Trusty)
Status: In Progress => Fix Committed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Trusty:
Fix Committed
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
/build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables veth
openvswitch gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich cdc_ether
x86_pkg_temp_thermal intel_powerclamp coretemp usbnet kvm_intel mii kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw
gf128mul
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
Ok scratch comment #13, must have been half asleep when I read comment
#12 , is there a status change I need to make for Trusty to move this
along? Thanks.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Trusty:
In Progress
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
/build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables veth
openvswitch gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich cdc_ether
x86_pkg_temp_thermal intel_powerclamp coretemp usbnet kvm_intel mii kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw
gf128mul glue_helper ablk_helper cryptd sb_edac edac_core lpc_ich wmi ipmi_si
bonding shpchp ioatdma lp mac_hid parport ahci libahci sfc igb e1000e mtd dca
i2c_algo_bit ptp pps_c
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
Hello Joseph, do you need me to test just kernel-image and headers? or
all debs in that dir, thanks for clarification.
-Andrew
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Trusty:
In Progress
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
/build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables veth
openvswitch gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich cdc_ether
x86_pkg_temp_thermal intel_powerclamp coretemp usbnet kvm_intel mii kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw
gf128mul glue_helper ablk_helper cryptd sb_edac edac_core lpc_ich wmi ipmi_si
bonding shpchp ioatdma lp mac_hid parport ahci libahci sfc igb e1000e mtd dca
i2c_algo_bit ptp pps_core megaraid_sas mdio
> [3
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
I see the patch was already submitted for SRU per comment #9, so no need
to test the kernel posted in comment #11.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Trusty:
In Progress
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
/build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables veth
openvswitch gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich cdc_ether
x86_pkg_temp_thermal intel_powerclamp coretemp usbnet kvm_intel mii kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw
gf128mul glue_helper ablk_helper cryptd sb_edac edac_core lpc_ich wmi ipmi_si
bonding shpchp ioatdma lp mac_hid parport ahci libahci sfc igb e1000e mtd dca
i2c_algo_bit ptp pps_core megaraid_sas mdio
> [33100.133560]
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
I built a Trusty test kernel with the patch. The test kernel can be downloaded
from:
http://kernel.ubuntu.com/~jsalisbury/lp1655683/
Can you test this kernel and see if it resolves this bug?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Trusty:
In Progress
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
/build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables veth
openvswitch gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich cdc_ether
x86_pkg_temp_thermal intel_powerclamp coretemp usbnet kvm_intel mii kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw
gf128mul glue_helper ablk_helper cryptd sb_edac edac_core lpc_ich wmi ipmi_si
bonding shpchp ioatdma lp mac_hid parport ahci libahci sfc igb
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
Between 2017-01-05 and about 2017-01-10 EST we had 17 recorded crashes
on the neutron node.
After patching as described above, we have had no crashes.
Not being particularly familiar with kernel internals, or the details of
the skb data structure, I was not able to confirm explicitly the cause
by sending the packets that trigger the crash. I am trusting that the
choice to drop the packets with the patch doesn't have any significant
side effects for the openvswitch module.
A clearer explanation with earlier patch(es) may be found here:
https://patchwork.ozlabs.org/patch/559944/
https://patchwork.ozlabs.org/patch/712373/
It also may be of note to others that gso and gro offloading to the NIC
are turned off on all of our interfaces.
I am not sure how timely the application of the corresponding patch in
the upstream stable kernel will be, so here is the patch I used.
attached is the patch for my test build, using the sources in comment #3
** Attachment added: "patch_1655683.txt"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+attachment/4803442/+files/patch_1655683.txt
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Trusty:
In Progress
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 h
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
https://lists.ubuntu.com/archives/kernel-team/2017-January/081806.html
** Changed in: linux (Ubuntu Trusty)
Status: Confirmed => In Progress
** Changed in: linux (Ubuntu Trusty)
Assignee: (unassigned) => Andrew Crawford (acrawford)
** Changed in: linux (Ubuntu)
Status: Confirmed => Fix Released
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Trusty:
In Progress
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
/build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables veth
openvswitch gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich cdc_ether
x86_pkg_temp_thermal intel_powerclamp coretemp usbnet kvm_intel mii kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw
gf128mul glue_help
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
** Changed in: linux (Ubuntu)
Importance: Undecided => High
** Also affects: linux (Ubuntu Trusty)
Importance: Undecided
Status: New
** Changed in: linux (Ubuntu Trusty)
Status: New => Incomplete
** Changed in: linux (Ubuntu Trusty)
Importance: Undecided => High
** Changed in: linux (Ubuntu Trusty)
Status: Incomplete => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Confirmed
Status in linux source package in Trusty:
Confirmed
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
/build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables veth
openvswitch gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich cdc_ether
x86_pkg_temp_thermal intel_powerclamp coretemp usbnet kvm_intel mii kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aes
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
The kernel running in the above apport report is the patched kernel FYI.
** Changed in: linux (Ubuntu)
Status: Incomplete => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Confirmed
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
/build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables veth
openvswitch gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich cdc_ether
x86_pkg_temp_thermal intel_powerclamp coretemp usbnet kvm_intel mii kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw
gf128mul glue_helper ablk_helper cryptd sb_edac edac_core lpc_ich wmi ipmi_si
bonding shpchp ioatdma lp mac_hid parport ahci libahci sfc igb e1000e mtd dca
i2c_algo_bit ptp pps_core megaraid_sas mdio
> [33100.133560] CPU: 2 PID: 0 Comm: swapper/2 N
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
Sorry, apport-collect is trying to open a browser from a server where
none is installed. I generated the report with apport-cli and have
attached here instead.
** Attachment added: "apport.linux-image-3.13.0-106-generic.b9qc1535.apport"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+attachment/4802917/+files/apport.linux-image-3.13.0-106-generic.b9qc1535.apport
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Incomplete
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
/build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables veth
openvswitch gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich cdc_ether
x86_pkg_temp_thermal intel_powerclamp coretemp usbnet kvm_intel mii kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw
gf128mul g
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
** Attachment added: "linux-image-3.13.0-106-generic-201701051502.crash"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+attachment/4802911/+files/linux-image-3.13.0-106-generic-201701051502.crash
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Incomplete
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
/build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables veth
openvswitch gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich cdc_ether
x86_pkg_temp_thermal intel_powerclamp coretemp usbnet kvm_intel mii kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw
gf128mul glue_helper ablk_helper cryptd sb_edac edac_core lpc_ich wmi ipmi_si
bonding shpchp ioatdma lp mac_hid parport ahci libahci sfc igb e1000e mtd dca
i2c_algo_bit ptp pps_
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
** Attachment added: "dump.201701051502"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+attachment/4802913/+files/dump.201701051502
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Incomplete
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
/build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables veth
openvswitch gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich cdc_ether
x86_pkg_temp_thermal intel_powerclamp coretemp usbnet kvm_intel mii kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw
gf128mul glue_helper ablk_helper cryptd sb_edac edac_core lpc_ich wmi ipmi_si
bonding shpchp ioatdma lp mac_hid parport ahci libahci sfc igb e1000e mtd dca
i2c_algo_bit ptp pps_core megaraid_sas mdio
> [33100.133560] CPU: 2 PID: 0 Comm: sw
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
** Attachment added: "dmesg.201701051502"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+attachment/4802912/+files/dmesg.201701051502
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Incomplete
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
/build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables veth
openvswitch gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich cdc_ether
x86_pkg_temp_thermal intel_powerclamp coretemp usbnet kvm_intel mii kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw
gf128mul glue_helper ablk_helper cryptd sb_edac edac_core lpc_ich wmi ipmi_si
bonding shpchp ioatdma lp mac_hid parport ahci libahci sfc igb e1000e mtd dca
i2c_algo_bit ptp pps_core megaraid_sas mdio
> [33100.133560] CPU: 2 PID: 0 Comm:
[Kernel-packages] [Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
I built a new kernel from the ubuntu kernel source package for linux-
image-3.13.0-106-generic. I am using the ubuntu shipped .config for
linux-image-3.13.0-106-generic. I am testing the patch linked below in
production.
https://patchwork.ozlabs.org/patch/712373/
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655683
Title:
kernel BUG at skbuff.h:1486 Insufficient linear data in skb
__skb_pull.part.7+0x4/0x6 [openvswitch]
Status in linux package in Ubuntu:
Incomplete
Bug description:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an &sk_buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an &sk_buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a
flap of the neutron services instead of a full panic that brings the
Neutron server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
/build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables veth
openvswitch gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich cdc_ether
x86_pkg_temp_thermal intel_powerclamp coretemp usbnet kvm_intel mii kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw
gf128mul glue_helper ablk_helper cryptd sb_edac edac_core lpc_ich wmi ipmi_si
bonding shpchp ioatdma lp mac_hid parport ahci lib
