[Bug 1336555] Re: ovs-vswitchd segfault every 2 days
I think this is probably the cause of the SIGSEGV - I've seen this same trace reported twice upstream as well. This was generated in one of the Ubuntu QA openstack deployments. ** Attachment added: stacktrace.txt https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1336555/+attachment/4293360/+files/stacktrace.txt ** Changed in: openvswitch (Ubuntu) Importance: Undecided = High ** Summary changed: - ovs-vswitchd segfault every 2 days + ovs-vswitchd crashed with SIGSEGV in nl_attr_get_size() -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openvswitch in Ubuntu. https://bugs.launchpad.net/bugs/1336555 Title: ovs-vswitchd crashed with SIGSEGV in nl_attr_get_size() To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1336555/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1336555] Re: ovs-vswitchd segfault every 2 days
I'm still seeing the problem with new kernels. It breaks my ceph installation badly. [1368555.197934] [ cut here ] [1368555.197945] WARNING: CPU: 3 PID: 4400 at /build/buildd/linux-3.13.0/fs/proc/generic.c:511 remove_proc_entry+0x139/0x1b0() [1368555.197947] name 'fs/nfsfs' [1368555.197949] Modules linked in: gspca_ov534 nls_iso8859_1 usb_storage xt_TCPMSS xt_tcpmss arc4 ppp_mppe ppp_async crc_ccitt vhost_net vhost macvtap macvlan nf_conntrack_ipv6 nf_defrag_ipv6 xt_mac xt_physdev xt_multiport pci_stub vboxpci(OX) vboxnetadp(OX) vboxnetflt(OX) vboxdrv(OX) xt_conntrack ipt_REJECT veth ip6table_filter ip6_tables ebtable_nat ebtables xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables nbd ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi rfcomm rbd libceph openvswitch gre vxlan ip_tunnel binfmt_misc nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache dm_crypt xfs snd_usb_audio snd_usbmidi_lib fglrx(POX) kvm_amd kvm gspca_main videodev wacom serio_raw dm_multipath scsi_dh joydev edac_core edac_mce_amd k10temp snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel btusb snd_hda _codec snd_hwdep snd_pcm bluetooth snd_page_alloc snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore sp5100_tco amd_iommu_v2 i2c_piix4 asus_atk0110 mac_hid parport_pc ppdev lp parport btrfs libcrc32c raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 multipath linear dm_mirror dm_region_hash dm_log hid_generic hid_logitech_dj usbhid hid pata_acpi psmouse pata_atiixp ahci e1000e pata_marvell libahci firewire_ohci ptp firewire_core pps_core r8169 crc_itu_t mii floppy wmi [last unloaded: gspca_ov534] [1368555.198044] CPU: 3 PID: 4400 Comm: kworker/u12:1 Tainted: PW OX 3.13.0-37-generic #64-Ubuntu [1368555.198046] Hardware name: System manufacturer System Product Name/M4A79T Deluxe, BIOS 350305/06/2011 [1368555.198051] Workqueue: netns cleanup_net [1368555.198053] 0009 880119017c80 8171ed09 880119017cc8 [1368555.198057] 880119017cb8 8106773d 0005 [1368555.198060] a06e48a8 880323019ab0 0180 880119017d18 [1368555.198063] Call Trace: [1368555.198070] [8171ed09] dump_stack+0x45/0x56 [1368555.198074] [8106773d] warn_slowpath_common+0x7d/0xa0 [1368555.198079] [810677ac] warn_slowpath_fmt+0x4c/0x50 [1368555.198084] [81229839] remove_proc_entry+0x139/0x1b0 [1368555.198103] [a06c44e2] nfs_fs_proc_net_exit+0x62/0x70 [nfs] [1368555.198115] [a06ca5b2] nfs_net_exit+0x12/0x20 [nfs] [1368555.198118] [8161b409] ops_exit_list.isra.1+0x39/0x60 [1368555.198122] [8161bc90] cleanup_net+0x110/0x250 [1368555.198126] [810839c2] process_one_work+0x182/0x450 [1368555.198130] [810847b1] worker_thread+0x121/0x410 [1368555.198133] [81084690] ? rescuer_thread+0x430/0x430 [1368555.198137] [8108b492] kthread+0xd2/0xf0 [1368555.198140] [8108b3c0] ? kthread_create_on_node+0x1c0/0x1c0 [1368555.198143] [8172f77c] ret_from_fork+0x7c/0xb0 [1368555.198146] [8108b3c0] ? kthread_create_on_node+0x1c0/0x1c0 [1368555.198148] ---[ end trace 007872efa0f6c8f7 ]--- [1372797.021226] libceph: osd0 172.16.0.119:6800 socket closed (con state OPEN) [1373904.941772] libceph: osd2 172.16.0.119:6805 socket closed (con state OPEN) [1374022.647670] [ cut here ] [1374022.647691] WARNING: CPU: 1 PID: 17323 at /build/buildd/linux-3.13.0/fs/proc/generic.c:511 remove_proc_entry+0x139/0x1b0() [1374022.647696] name 'fs/nfsfs' [1374022.647700] Modules linked in: gspca_ov534 nls_iso8859_1 usb_storage xt_TCPMSS xt_tcpmss arc4 ppp_mppe ppp_async crc_ccitt vhost_net vhost macvtap macvlan nf_conntrack_ipv6 nf_defrag_ipv6 xt_mac xt_physdev xt_multiport pci_stub vboxpci(OX) vboxnetadp(OX) vboxnetflt(OX) vboxdrv(OX) xt_conntrack ipt_REJECT veth ip6table_filter ip6_tables ebtable_nat ebtables xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables nbd ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi rfcomm rbd libceph openvswitch gre vxlan ip_tunnel binfmt_misc nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache dm_crypt xfs snd_usb_audio snd_usbmidi_lib fglrx(POX) kvm_amd kvm gspca_main videodev wacom serio_raw dm_multipath scsi_dh joydev edac_core edac_mce_amd k10temp snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel btusb snd_hda _codec snd_hwdep snd_pcm bluetooth snd_page_alloc snd_seq_midi snd_seq_midi_event snd_rawmidi
[Bug 1336555] Re: ovs-vswitchd segfault every 2 days
After updating kernel in ubuntu 14.04 I found this again. But this time with kernel panic. ct 12 00:56:30 red-compute kernel: [ 2139.850867] ovs-vswitchd[19280]: segfault at 0 ip 00459340 sp 7fff2d88c3e8 error 4 in ovs-vswitchd[40+133000] Oct 12 00:56:31 red-compute ovs-vswitchd: ovs|00010|daemon(monitor)|ERR|7 crashes: pid 19280 died, killed (Segmentation fault), core dumped, restarting Oct 12 00:57:01 red-compute CRON[19736]: (root) CMD (if [ -x /usr/share/mdadm/checkarray ] [ $(date +%d) -le 7 ]; then /usr/share/mdadm/checkarray --cron --all --idle --quiet; fi) Oct 12 00:58:28 red-compute ovsdb-client: ovs|1|fatal_signal|WARN|terminating with signal 15 (Terminated) Oct 12 00:58:34 red-compute kernel: [ 2264.419340] init: neutron-plugin-openvswitch-agent main process (14411) killed by KILL signal Oct 12 00:58:56 red-compute kernel: [ 2286.159789] INFO: task jbd2/rbd1-8:10184 blocked for more than 120 seconds. Oct 12 00:58:56 red-compute kernel: [ 2286.159802] Tainted: PW OX 3.13.0-37-generic #64-Ubuntu Oct 12 00:58:56 red-compute kernel: [ 2286.159807] echo 0 /proc/sys/kernel/hung_task_timeout_secs disables this message. Oct 12 00:58:56 red-compute kernel: [ 2286.159812] jbd2/rbd1-8 D 88032fc54480 0 10184 2 0x Oct 12 00:58:56 red-compute kernel: [ 2286.159822] 88029baadbc8 0046 8802d1ac8000 88029baadfd8 Oct 12 00:58:56 red-compute kernel: [ 2286.159832] 00014480 00014480 8802d1ac8000 88032fc54d18 Oct 12 00:58:56 red-compute kernel: [ 2286.159840] 88032ffb1020 0002 811ee820 88029baadc40 Oct 12 00:58:56 red-compute kernel: [ 2286.159847] Call Trace: Oct 12 00:58:56 red-compute kernel: [ 2286.159864] [811ee820] ? generic_block_bmap+0x50/0x50 Oct 12 00:58:56 red-compute kernel: [ 2286.159875] [8172344d] io_schedule+0x9d/0x140 Oct 12 00:58:56 red-compute kernel: [ 2286.159884] [811ee82e] sleep_on_buffer+0xe/0x20 Oct 12 00:58:56 red-compute kernel: [ 2286.159892] [817238d2] __wait_on_bit+0x62/0x90 Oct 12 00:58:56 red-compute kernel: [ 2286.159899] [811ee820] ? generic_block_bmap+0x50/0x50 Oct 12 00:58:56 red-compute kernel: [ 2286.159907] [81723977] out_of_line_wait_on_bit+0x77/0x90 Oct 12 00:58:56 red-compute kernel: [ 2286.159916] [810ab010] ? autoremove_wake_function+0x40/0x40 Oct 12 00:58:56 red-compute kernel: [ 2286.159924] [811efb5a] __wait_on_buffer+0x2a/0x30 Oct 12 00:58:56 red-compute kernel: [ 2286.159933] [812888f0] jbd2_journal_commit_transaction+0xee0/0x1a70 Oct 12 00:58:56 red-compute kernel: [ 2286.159943] [810754ff] ? try_to_del_timer_sync+0x4f/0x70 Oct 12 00:58:56 red-compute kernel: [ 2286.159951] [8128d4ad] kjournald2+0xbd/0x250 Oct 12 00:58:56 red-compute kernel: [ 2286.159959] [810aafd0] ? prepare_to_wait_event+0x100/0x100 Oct 12 00:58:56 red-compute kernel: [ 2286.159967] [8128d3f0] ? commit_timeout+0x10/0x10 Oct 12 00:58:56 red-compute kernel: [ 2286.159974] [8108b492] kthread+0xd2/0xf0 Oct 12 00:58:56 red-compute kernel: [ 2286.159981] [8108b3c0] ? kthread_create_on_node+0x1c0/0x1c0 Oct 12 00:58:56 red-compute kernel: [ 2286.159988] [8172f77c] ret_from_fork+0x7c/0xb0 Oct 12 00:58:56 red-compute kernel: [ 2286.159995] [8108b3c0] ? kthread_create_on_node+0x1c0/0x1c0 Oct 12 00:58:56 red-compute kernel: [ 2286.160016] INFO: task init:10493 blocked for more than 120 seconds. Oct 12 00:58:56 red-compute kernel: [ 2286.160022] Tainted: PW OX 3.13.0-37-generic #64-Ubuntu Oct 12 00:58:56 red-compute kernel: [ 2286.160025] echo 0 /proc/sys/kernel/hung_task_timeout_secs disables this message. Oct 12 00:58:56 red-compute kernel: [ 2286.160028] initD 88032fc94480 0 10493 10443 0x Oct 12 00:58:56 red-compute kernel: [ 2286.160036] 8802eae2da28 0082 8802b37a3000 8802eae2dfd8 Oct 12 00:58:56 red-compute kernel: [ 2286.160043] 00014480 00014480 8802b37a3000 88032fc94d18 Oct 12 00:58:56 red-compute kernel: [ 2286.160050] 88032ffb7908 0002 812851c0 8802eae2daa0 Oct 12 00:58:56 red-compute kernel: [ 2286.160057] Call Trace: Oct 12 00:58:56 red-compute kernel: [ 2286.160066] [812851c0] ? start_this_handle+0x590/0x590 Oct 12 00:58:56 red-compute kernel: [ 2286.160074] [8172344d] io_schedule+0x9d/0x140 Oct 12 00:58:56 red-compute kernel: [ 2286.160081] [812851ce] sleep_on_shadow_bh+0xe/0x20 Oct 12 00:58:56 red-compute kernel: [ 2286.160088] [817238d2] __wait_on_bit+0x62/0x90 Oct 12 00:58:56 red-compute kernel: [ 2286.160095] [812851c0] ? start_this_handle+0x590/0x590 Oct 12 00:58:56 red-compute kernel: [ 2286.160103] [81723977] out_of_line_wait_on_bit+0x77/0x90 Oct 12 00:58:56 red-compute kernel: [
[Bug 1336555] Re: ovs-vswitchd segfault every 2 days
ovs-vsctl (Open vSwitch) 2.0.2 Compiled Aug 15 2014 14:31:02 I did: apport-collect 1336555 Package openvswitch not installed and no hook available, ignoring But no luck. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openvswitch in Ubuntu. https://bugs.launchpad.net/bugs/1336555 Title: ovs-vswitchd segfault every 2 days To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1336555/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1336555] Re: ovs-vswitchd segfault every 2 days
** Description changed: - Hi I find that every 2 days or so I loose part of my cluster. + Hi I find that every 2 days or so I lose part of my cluster. It seems that openvswitch is crashing... The only message left on syslog is as follows: syslog:Jul 1 22:52:32 blue-compute kernel: [530482.190688] ovs- vswitchd[1935]: segfault at 0 ip 00459110 sp 7fff85804758 error 4 in ovs-vswitchd[40+133000] And this is the last message. I'm unable to reboot gracefully. I have to reset. (This can be because ceph not giving up also). And I can see a lot of traffic going around in the network. There so much traffic that some lowend routers/switches fail. Can be because another problem (machines stalled because the ovs fault and others trying to connect. Maybe it fails because much traffic). But I tell this for completeness. Now some info: Linux version 3.13.0-30-generic (buildd@allspice) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #54-Ubu ntu SMP Mon Jun 9 22:45:01 UTC 2014 (Ubuntu 3.13.0-30.54-generic 3.13.11.2) - vendor_id : AuthenticAMD cpu family: 16 model : 4 model name: AMD Phenom(tm) II X4 810 Processor - Ubuntu 14.04 LTS (server). - ovs-vsctl --version ovs-vsctl (Open vSwitch) 2.0.1 Compiled Feb 23 2014 14:42:32 I can attach full logs but I think there's nothing useful because only one line referring the problem. - NOTE: restarting ovs does not solve the problem. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openvswitch in Ubuntu. https://bugs.launchpad.net/bugs/1336555 Title: ovs-vswitchd segfault every 2 days To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1336555/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1336555] Re: ovs-vswitchd segfault every 2 days
Hi, I have the same problem. There is just one difference: I'm using ubuntu 12.04 with linux kernel 3.13. When the crash happens I can no longer access virtual machines. If I restart neutron-plugin-openvswitch-switch the communication to vms is restored. ProblemType: Crash Architecture: amd64 CrashCounter: 1 Date: Wed Aug 6 21:19:18 2014 DistroRelease: Ubuntu 12.04 ExecutablePath: /usr/sbin/ovs-vswitchd ExecutableTimestamp: 1397658071 ProcCmdline: ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor ProcCwd: / ProcEnviron: TERM=linux PATH=(custom, no user) ProcMaps: 0040-00535000 r-xp 08:01 174237 /usr/sbin/ovs-vswitchd ... ff60-ff601000 r-xp 00:00 0 [vsyscall] ProcStatus: Name: ovs-vswitchd State: S (sleeping) Tgid: 1413 Ngid: 0 Pid: 1413 PPid: 1412 TracerPid: 0 Uid: 0 0 0 0 Gid: 0 0 0 0 FDSize:64 Groups: VmPeak: 236840 kB VmSize: 171984 kB VmLck: 171976 kB VmPin:0 kB VmHWM:32396 kB VmRSS:32396 kB VmData: 148604 kB VmStk: 252 kB VmExe: 1236 kB VmLib: 5176 kB VmPTE: 124 kB VmSwap: 0 kB Threads: 3 SigQ: 6/7775 SigPnd: ShdPnd: SigBlk: SigIgn:1000 SigCgt:000180016003 CapInh: CapPrm:001f CapEff:001f CapBnd:001f Seccomp: 0 Cpus_allowed: 3 Cpus_allowed_list: 0-1 Mems_allowed: ,0001 Mems_allowed_list: 0 voluntary_ctxt_switches: 324391 nonvoluntary_ctxt_switches:45330 Signal: 11 Uname: Linux 3.13.0-32-generic x86_64 UserGroups: -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openvswitch in Ubuntu. https://bugs.launchpad.net/bugs/1336555 Title: ovs-vswitchd segfault every 2 days To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1336555/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1336555] Re: ovs-vswitchd segfault every 2 days
Hi Nooope, I had to restart the computer and I think I don't have it. It didn't hapened for a while. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openvswitch in Ubuntu. https://bugs.launchpad.net/bugs/1336555 Title: ovs-vswitchd segfault every 2 days To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1336555/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1336555] Re: ovs-vswitchd segfault every 2 days
Do you have anything in /var/crash? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openvswitch in Ubuntu. https://bugs.launchpad.net/bugs/1336555 Title: ovs-vswitchd segfault every 2 days To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1336555/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1336555] Re: ovs-vswitchd segfault every 2 days
Hi, Is there a way I can get a core from this process? I suppose that leaving ulimit -c unlimited is not enough. And I don't know where it will be left. But I can try. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openvswitch in Ubuntu. https://bugs.launchpad.net/bugs/1336555 Title: ovs-vswitchd segfault every 2 days To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1336555/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1336555] Re: ovs-vswitchd segfault every 2 days
To make it happen more ofter, is needed a lot of transfer connections through the network node. I have GB eth with two interfaces on each host to make it happen. But only 4-5 vmachines on the cloud. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openvswitch in Ubuntu. https://bugs.launchpad.net/bugs/1336555 Title: ovs-vswitchd segfault every 2 days To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1336555/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1336555] Re: ovs-vswitchd segfault every 2 days
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: openvswitch (Ubuntu) Status: New = Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openvswitch in Ubuntu. https://bugs.launchpad.net/bugs/1336555 Title: ovs-vswitchd segfault every 2 days To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1336555/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1336555] Re: ovs-vswitchd segfault every 2 days
Same here: Ubuntu 14.04 64bit openvswitch 2.0.1+git20140120-0ubuntu2 ovs-vsctl (Open vSwitch) 2.0.1 Compiled Feb 23 2014 14:42:32 Running network node for OpenStack -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openvswitch in Ubuntu. https://bugs.launchpad.net/bugs/1336555 Title: ovs-vswitchd segfault every 2 days To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1336555/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
