[Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
Thanks Tim for the clarification. I will not make any more status changes. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655683 Title: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
I have moved this to "fix committed", can someone verify that this patch will be backported to the Trusty 3.13 series kernel, or do I need to take any additional steps? I am not sure what my responsibility is in moving this forward from here. ** Changed in: linux (Ubuntu Trusty) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655683 Title: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
Ok scratch comment #13, must have been half asleep when I read comment #12 , is there a status change I need to make for Trusty to move this along? Thanks. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655683 Title: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
Hello Joseph, do you need me to test just kernel-image and headers? or all debs in that dir, thanks for clarification. -Andrew -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655683 Title: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
Between 2017-01-05 and about 2017-01-10 EST we had 17 recorded crashes on the neutron node. After patching as described above, we have had no crashes. Not being particularly familiar with kernel internals, or the details of the skb data structure, I was not able to confirm explicitly the cause by sending the packets that trigger the crash. I am trusting that the choice to drop the packets with the patch doesn't have any significant side effects for the openvswitch module. A clearer explanation with earlier patch(es) may be found here: https://patchwork.ozlabs.org/patch/559944/ https://patchwork.ozlabs.org/patch/712373/ It also may be of note to others that gso and gro offloading to the NIC are turned off on all of our interfaces. I am not sure how timely the application of the corresponding patch in the upstream stable kernel will be, so here is the patch I used. attached is the patch for my test build, using the sources in comment #3 ** Attachment added: "patch_1655683.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+attachment/4803442/+files/patch_1655683.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655683 Title: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
The kernel running in the above apport report is the patched kernel FYI. ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655683 Title: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
Sorry, apport-collect is trying to open a browser from a server where none is installed. I generated the report with apport-cli and have attached here instead. ** Attachment added: "apport.linux-image-3.13.0-106-generic.b9qc1535.apport" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+attachment/4802917/+files/apport.linux-image-3.13.0-106-generic.b9qc1535.apport -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655683 Title: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
** Attachment added: "dump.201701051502" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+attachment/4802913/+files/dump.201701051502 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655683 Title: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
** Attachment added: "linux-image-3.13.0-106-generic-201701051502.crash" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+attachment/4802911/+files/linux-image-3.13.0-106-generic-201701051502.crash -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655683 Title: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
** Attachment added: "dmesg.201701051502" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+attachment/4802912/+files/dmesg.201701051502 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655683 Title: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655683] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
I built a new kernel from the ubuntu kernel source package for linux- image-3.13.0-106-generic. I am using the ubuntu shipped .config for linux-image-3.13.0-106-generic. I am testing the patch linked below in production. https://patchwork.ozlabs.org/patch/712373/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655683 Title: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655683/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655117] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
*** This bug is a duplicate of bug 1655683 *** https://bugs.launchpad.net/bugs/1655683 ** This bug has been marked a duplicate of bug 1655683 kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch] -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655117 Title: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1655117/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655683] [NEW] kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
Public bug reported:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an _buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an _buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a flap
of the neutron services instead of a full panic that brings the Neutron
server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
> /build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
> ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_conntrack_ipv4
> nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables veth
> openvswitch gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich cdc_ether
> x86_pkg_temp_thermal intel_powerclamp coretemp usbnet kvm_intel mii kvm
> crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw
> gf128mul glue_helper ablk_helper cryptd sb_edac edac_core lpc_ich wmi ipmi_si
> bonding shpchp ioatdma lp mac_hid parport ahci libahci sfc igb e1000e mtd dca
> i2c_algo_bit ptp pps_core megaraid_sas mdio
> [33100.133560] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.13.0-106-generic
> #153-Ubuntu
> [33100.133800] Hardware name: IBM System x3650 M4 : -[7915AC1]-/00Y8473, BIOS
> -[VVE136AUS-1.60]- 12/12/2013
> [33100.134096] task: 880469da4800 ti: 880469dae000 task.ti:
> 880469dae000
> [33100.134325] RIP: 0010:[] []
> __skb_pull.part.7+0x4/0x6 [openvswitch]
> [33100.134628] RSP: 0018:88046fd03bb0 EFLAGS: 00010297
> [33100.134792] RAX: 880035d73866 RBX: 880461efb600 RCX:
> 880035d73800
> [33100.135011] RDX: 0210 RSI: 0214 RDI:
> 88046fd03c98
> [33100.135231] RBP: 88046fd03bb0 R08:
[Bug 1655117] Re: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
Hi all, it looks like the patch I referenced above is indeed aimed at the openvswitch kernel module, should have looked more closely at the outset, so this bug really belongs with ubuntu-kernel, and I believe, specifically the pre-DKMS openvswitch kernel module. Looking into the ubuntu kernel source for /net/openvswitch/vport-gre.c https://github.com/Canonical-kernel/Ubuntu-kernel/blob/master/net/openvswitch/vport-gre.c The patch mentioned above at patchwork is not present. I am not familiar with the upstream kernel process. looking into it. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655117 Title: kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1655117/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655117] [NEW] kernel BUG at skbuff.h:1486 Insufficient linear data in skb __skb_pull.part.7+0x4/0x6 [openvswitch]
Public bug reported:
Since 2016-12-30 EST we have been experiencing repeated crashes of our
OpenStack Icehouse / Trusty Neutron node with a kernel BUG at skbuff.h
line 1486:
1471 /**
1472 * skb_peek - peek at the head of an _buff_head
1473 * @list_: list to peek at
1474 *
1475 * Peek an _buff. Unlike most other operations you _MUST_
1476 * be careful with this one. A peek leaves the buffer on the
1477 * list and someone else may run off with it. You must hold
1478 * the appropriate locks or have a private queue to do this.
1479 *
1480 * Returns %NULL for an empty list or a pointer to the head element.
1481 * The reference count is not incremented and the reference is therefore
1482 * volatile. Use with caution.
1483 */
1484 static inline struct sk_buff *skb_peek(const struct sk_buff_head *list_)
1485 {
1486 struct sk_buff *skb = list_->next;
1487
1488 if (skb == (struct sk_buff *)list_)
1489 skb = NULL;
1490 return skb;
1491 }
This generally results in a full panic crash of the Neutron node and
connectivity breaking for VMs within the cloud. However, after using
crash-dumptools to collect information on the crashes over the past
three days, the kernel loaded by kexec during the crashdump appears in
about 2 out of 3 crash instances to continue running, and we see a flap
of the neutron services instead of a full panic that brings the Neutron
server down and necessitates a hard reboot.
I believe that this is a manifestation of the openvswitch and issue
described on 2017-01-08 as:
"OVS can only process L2 packets. But OVS GRE receive handler
can accept IP-GRE packets. When such packet is processed by
OVS datapath it can trigger following assert failure due
to insufficient linear data in skb."
https://patchwork.ozlabs.org/patch/712373/
I have not tested the patch provided above yet.
Other information and a few sample dmesg outputs from the crash:
(multiple dumps available)
# lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
# apt-cache policy openvswitch
N: Unable to locate package openvswitch
root@neutron01:/var/crash# apt-cache policy openvswitch-common
openvswitch-common:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy openvswitch-switch
openvswitch-switch:
Installed: 2.0.2-0ubuntu0.14.04.3
Candidate: 2.0.2-0ubuntu0.14.04.3
Version table:
*** 2.0.2-0ubuntu0.14.04.3 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
2.0.1+git20140120-0ubuntu2 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
# apt-cache policy neutron-plugin-openvswitch-agent
neutron-plugin-openvswitch-agent:
Installed: 1:2014.1.5-0ubuntu7
Candidate: 1:2014.1.5-0ubuntu7
Version table:
*** 1:2014.1.5-0ubuntu7 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:2014.1.3-0ubuntu1.1 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64
Packages
1:2014.1-0ubuntu1 0
500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
example dmesg:
## dmesg.201701060019
> [33100.131019] [ cut here ]
> [33100.131176] kernel BUG at
> /build/linux-mi9H1O/linux-3.13.0/include/linux/skbuff.h:1486!
> [33100.131424] invalid opcode: [#1] SMP
> [33100.131560] Modules linked in: xt_nat xt_conntrack ip6table_filter
> ip6_tables iptable_filter xt_REDIRECT xt_tcpudp iptable_nat nf_conntrack_ipv4
> nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables veth
> openvswitch gre vxlan ip_tunnel libcrc32c ipmi_devintf gpio_ich cdc_ether
> x86_pkg_temp_thermal intel_powerclamp coretemp usbnet kvm_intel mii kvm
> crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw
> gf128mul glue_helper ablk_helper cryptd sb_edac edac_core lpc_ich wmi ipmi_si
> bonding shpchp ioatdma lp mac_hid parport ahci libahci sfc igb e1000e mtd dca
> i2c_algo_bit ptp pps_core megaraid_sas mdio
> [33100.133560] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.13.0-106-generic
> #153-Ubuntu
> [33100.133800] Hardware name: IBM System x3650 M4 : -[7915AC1]-/00Y8473, BIOS
> -[VVE136AUS-1.60]- 12/12/2013
> [33100.134096] task: 880469da4800 ti: 880469dae000 task.ti:
> 880469dae000
> [33100.134325] RIP: 0010:[] []
> __skb_pull.part.7+0x4/0x6 [openvswitch]
> [33100.134628] RSP: 0018:88046fd03bb0 EFLAGS: 00010297
> [33100.134792] RAX: 880035d73866 RBX: 880461efb600 RCX:
> 880035d73800
> [33100.135011] RDX: 0210 RSI: 0214 RDI:
> 88046fd03c98
> [33100.135231] RBP: 88046fd03bb0 R08:
[Bug 1312814] Re: red5-server has unmet dependencies
I did get it to install after manually installling libtomcat6-java, but To reproduce: ubuntu 14.04 LTS sudo apt-get update sudo apt-get upgrade sudo apt-get install red5-server red5-server: Depends: libtomcat6-java (= 6.0.20-7) but it is not going to be installed then: sudo apt-get install libtomcat6-java This successfully brings in all dependencies for libtomcat6-java and installs. Trying again after manually meeting the libtomcat6-java dependency: sudo apt-get install red5-server Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: libred5-java The following NEW packages will be installed: libred5-java red5-server 0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded. Need to get 0 B/1,023 kB of archives. After this operation, 1,372 kB of additional disk space will be used. Do you want to continue? [Y/n] Y Selecting previously unselected package libred5-java. (Reading database ... 263596 files and directories currently installed.) Preparing to unpack .../libred5-java_1.0~svn4374-3_all.deb ... Unpacking libred5-java (1.0~svn4374-3) ... Selecting previously unselected package red5-server. Preparing to unpack .../red5-server_1.0~svn4374-3_all.deb ... Unpacking red5-server (1.0~svn4374-3) ... Processing triggers for ureadahead (0.100.0-16) ... Setting up libred5-java (1.0~svn4374-3) ... Setting up red5-server (1.0~svn4374-3) ... Allowing use of questionable username. Adding system user `_red5' (UID 119) ... Adding new user `_red5' (UID 119) with group `nogroup' ... Not creating home directory `/usr/share/red5'. * Starting Flash streaming server red5-server [fail] invoke-rc.d: initscript red5-server, action start failed. dpkg: error processing package red5-server (--configure): subprocess installed post-installation script returned error exit status 1 Processing triggers for ureadahead (0.100.0-16) ... Errors were encountered while processing: red5-server E: Sub-process /usr/bin/dpkg returned an error code (1) /var/log/dpkg.log 2014-06-25 14:45:48 install libred5-java:all none 1.0~svn4374-3 2014-06-25 14:45:48 status half-installed libred5-java:all 1.0~svn4374-3 2014-06-25 14:45:48 status unpacked libred5-java:all 1.0~svn4374-3 2014-06-25 14:45:48 status unpacked libred5-java:all 1.0~svn4374-3 2014-06-25 14:45:48 install red5-server:all none 1.0~svn4374-3 2014-06-25 14:45:48 status half-installed red5-server:all 1.0~svn4374-3 2014-06-25 14:45:49 status triggers-pending ureadahead:amd64 0.100.0-16 2014-06-25 14:45:49 status half-installed red5-server:all 1.0~svn4374-3 2014-06-25 14:45:49 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:49 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:49 trigproc ureadahead:amd64 0.100.0-16 0.100.0-16 2014-06-25 14:45:49 status half-configured ureadahead:amd64 0.100.0-16 2014-06-25 14:45:49 status installed ureadahead:amd64 0.100.0-16 2014-06-25 14:45:50 startup packages configure 2014-06-25 14:45:50 configure libred5-java:all 1.0~svn4374-3 none 2014-06-25 14:45:50 status unpacked libred5-java:all 1.0~svn4374-3 2014-06-25 14:45:50 status half-configured libred5-java:all 1.0~svn4374-3 2014-06-25 14:45:50 status installed libred5-java:all 1.0~svn4374-3 2014-06-25 14:45:50 configure red5-server:all 1.0~svn4374-3 none 2014-06-25 14:45:50 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:50 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:50 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:50 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:50 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:50 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:50 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:50 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:50 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:50 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:50 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:51 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:51 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:51 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:51 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:51 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:51 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:51 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:51 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:51 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:51 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:51 status unpacked red5-server:all 1.0~svn4374-3 2014-06-25 14:45:51 status
[Bug 992075] Re: Commissioning status persists with cloud-init 0.6.3-0ubuntu1
I am also using cloud-init 0.6.3-0ubuntu1. At first I did see the bug at https://bugs.launchpad.net/ubuntu/+source/maas/+bug/981845 and as per my comment there, did have the network cables inadvertantly switched, no PXE boot. Switching the cables and re-booting the node had no effect on the persistent Commissioning status. Though the node did begin the boot process. Because the UI was unable to positively identify and enlist the node, and because my original setup (prior to dist-upgrade) was Orchestra using 11.10, I wanted to make sure it wasn't a problem with some of the manual reconfiguration necessary after the dist upgrade (switch DHCP and DNS providers etc.) However, after rectifying the cabling issue and installing a fresh MAAS server from the official 12.04 release (and with apt-get update apt- get upgrade) , I am still seeing this behavior, but only one node has this problem. I also had no luck with the daily ephemeral build. After watching the PXE boot process on the problematic node, at about 11 seconds into boot I receive a init: cloud-init-nonet main process (###) killed by TERM signal which strongly suggests that there is a good reason that the Commissioning process is never completed for this node Interestingly, the node continues to boot to a normal login prompt after about 3-4 minutes. if left alone, the node will drop a few errors on the console: * Starting App Armor profiles Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd landscape-client is not configured, please run landscape-config which makes some sense as the init process isn't completing. I am tempted to open a new bug, for the: init: cloud-init-nonet main process (###) killed by TERM signal boot error, but unless people are looking at the actual boot process of the node, the only direct indication that anything is wrong is the endless Commissioning state in the MAAS web UI. OK I have ruled out this as the problem: http://irclogs.ubuntu.com/2012/04/18/%23ubuntu-server.txt A hardware clock mismatch causing Oauth to fail. [01:31] DiabolicalGamer I'm attempting to setup a MaaS server on Ubuntu 12.04, but my nodes keep hanging at init: cloud-init-nonet main process (256) killed by TERM signal [01:31] DiabolicalGamer Can anyone help? [01:31] bigjools I can try [01:31] DiabolicalGamer Thanks :-) [01:32] bigjools having said that I am more familiar with the webapp side of things than cloud-init [01:32] bigjools smoser, any idea? ^ [01:33] DiabolicalGamer hmm, if I could login to the nodes themselves or access their logs that would really help [01:42] DiabolicalGamer I think I may have found the problem... [01:43] DiabolicalGamer http://pastebin.com/JPw9F5FN [01:43] DiabolicalGamer My apache error log is full of these and they appear whenever the cloud-init-nonet runs [01:44] DiabolicalGamer any ideas? [01:45] bigjools DiabolicalGamer: ah I know [01:45] bigjools DiabolicalGamer: the clock is wrong on the node [01:45] DiabolicalGamer lol [01:45] DiabolicalGamer is that all? [01:45] bigjools well either the node or the maas server [01:46] DiabolicalGamer *facepalm* [01:46] bigjools yeah, it breaks OAuth if they are too different [01:46] DiabolicalGamer it must be the nodes then because I configured the system clock when I installed ubuntu on the cloud controller [02:02] DiabolicalGamer is there a way to force the nodes to run ntp-update? [02:03] DiabolicalGamer *ntpdate [02:12] DiabolicalGamer OMG it worked! [02:12] DiabolicalGamer Thanks bigjools [02:12] bigjools DiabolicalGamer: yay! Since this seems tangental to this particular bug, I am leaving this here in the hope that someone may find it useful. I will post back if I file a new bug report or find an existing appropriate bug. Thanks -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to maas in Ubuntu. https://bugs.launchpad.net/bugs/992075 Title: Commissioning status persists with cloud-init 0.6.3-0ubuntu1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/maas/+bug/992075/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 992075] Re: Commissioning status persists with cloud-init 0.6.3-0ubuntu1
I am also using cloud-init 0.6.3-0ubuntu1. At first I did see the bug at https://bugs.launchpad.net/ubuntu/+source/maas/+bug/981845 and as per my comment there, did have the network cables inadvertantly switched, no PXE boot. Switching the cables and re-booting the node had no effect on the persistent Commissioning status. Though the node did begin the boot process. Because the UI was unable to positively identify and enlist the node, and because my original setup (prior to dist-upgrade) was Orchestra using 11.10, I wanted to make sure it wasn't a problem with some of the manual reconfiguration necessary after the dist upgrade (switch DHCP and DNS providers etc.) However, after rectifying the cabling issue and installing a fresh MAAS server from the official 12.04 release (and with apt-get update apt- get upgrade) , I am still seeing this behavior, but only one node has this problem. I also had no luck with the daily ephemeral build. After watching the PXE boot process on the problematic node, at about 11 seconds into boot I receive a init: cloud-init-nonet main process (###) killed by TERM signal which strongly suggests that there is a good reason that the Commissioning process is never completed for this node Interestingly, the node continues to boot to a normal login prompt after about 3-4 minutes. if left alone, the node will drop a few errors on the console: * Starting App Armor profiles Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd landscape-client is not configured, please run landscape-config which makes some sense as the init process isn't completing. I am tempted to open a new bug, for the: init: cloud-init-nonet main process (###) killed by TERM signal boot error, but unless people are looking at the actual boot process of the node, the only direct indication that anything is wrong is the endless Commissioning state in the MAAS web UI. OK I have ruled out this as the problem: http://irclogs.ubuntu.com/2012/04/18/%23ubuntu-server.txt A hardware clock mismatch causing Oauth to fail. [01:31] DiabolicalGamer I'm attempting to setup a MaaS server on Ubuntu 12.04, but my nodes keep hanging at init: cloud-init-nonet main process (256) killed by TERM signal [01:31] DiabolicalGamer Can anyone help? [01:31] bigjools I can try [01:31] DiabolicalGamer Thanks :-) [01:32] bigjools having said that I am more familiar with the webapp side of things than cloud-init [01:32] bigjools smoser, any idea? ^ [01:33] DiabolicalGamer hmm, if I could login to the nodes themselves or access their logs that would really help [01:42] DiabolicalGamer I think I may have found the problem... [01:43] DiabolicalGamer http://pastebin.com/JPw9F5FN [01:43] DiabolicalGamer My apache error log is full of these and they appear whenever the cloud-init-nonet runs [01:44] DiabolicalGamer any ideas? [01:45] bigjools DiabolicalGamer: ah I know [01:45] bigjools DiabolicalGamer: the clock is wrong on the node [01:45] DiabolicalGamer lol [01:45] DiabolicalGamer is that all? [01:45] bigjools well either the node or the maas server [01:46] DiabolicalGamer *facepalm* [01:46] bigjools yeah, it breaks OAuth if they are too different [01:46] DiabolicalGamer it must be the nodes then because I configured the system clock when I installed ubuntu on the cloud controller [02:02] DiabolicalGamer is there a way to force the nodes to run ntp-update? [02:03] DiabolicalGamer *ntpdate [02:12] DiabolicalGamer OMG it worked! [02:12] DiabolicalGamer Thanks bigjools [02:12] bigjools DiabolicalGamer: yay! Since this seems tangental to this particular bug, I am leaving this here in the hope that someone may find it useful. I will post back if I file a new bug report or find an existing appropriate bug. Thanks -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/992075 Title: Commissioning status persists with cloud-init 0.6.3-0ubuntu1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/maas/+bug/992075/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
