I think I can reproduce that bug. Not 100% sure but today I think I had it several times.
Interesting is that it happens when a VM on kvm boots. Certainly strange is that several hosts crash simultaneously. My setup is a lab setup and can be accessed if it helps in troubleshooting. <http://www.nuvula.ch/> Marco Kuendig / CEO / Founder ma...@nuvula.ch <mailto:ma...@nuvula.ch> / +41 78 751 99 71 Marco's Google Hangout <https://plus.google.com/hangouts/_/nuvula.ch/marco> Nuvula AG - Hybrid Clouds Weierbachstrasse 7b 8193 Eglisau Switzerland http://www.nuvula.ch <http://www.nuvula.ch/> > On 31 Mar 2015, at 23:12, Joe Stringer <joestrin...@nicira.com> wrote: > > James, I believe you were involved last time this bug came up, I wonder if > you ever got to the bottom of this? > > --- > > This looks the same as a bug reported in October: > > http://openvswitch.org/pipermail/discuss/2014-October/015429.html > <http://openvswitch.org/pipermail/discuss/2014-October/015429.html> > > Ben's assessment was that there is no logical issue in the code, so perhaps > there was weird code generation caused by GCC. > > > On 31 March 2015 at 13:05, Marco Kuendig <ma...@nuvula.ch > <mailto:ma...@nuvula.ch>> wrote: > Reading symbols from /usr/sbin/ovs-vswitchd...Reading symbols from > /usr/lib/debug//usr/sbin/ovs-vswitchd...done. > done. > [New LWP 32725] > [New LWP 32732] > [New LWP 32726] > [New LWP 32730] > [New LWP 32727] > [New LWP 32728] > [New LWP 32729] > [New LWP 32731] > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > Core was generated by `ovs-vswitchd unix:/var/run/openvswitch/db.sock > -vconsole:emer -vsyslog:err -vfi'. > Program terminated with signal SIGSEGV, Segmentation fault. > #0 nl_attr_get_size (nla=nla@entry=0x0) at ../lib/netlink.c:506 > 506 ../lib/netlink.c: No such file or directory. > (gdb) bt > #0 nl_attr_get_size (nla=nla@entry=0x0) at ../lib/netlink.c:506 > #1 0x0000000000460473 in format_generic_odp_key (a=a@entry=0x0, > ds=ds@entry=0x7fff0408f3b0) at ../lib/odp-util.c:767 > #2 0x0000000000460cd2 in format_odp_key_attr (a=a@entry=0xc485a4, > ma=ma@entry=0x0, ds=ds@entry=0x7fff0408f3b0, verbose=verbose@entry=true) > at ../lib/odp-util.c:1332 > #3 0x00000000004609d7 in odp_flow_format (key=<optimized out>, key_len=40, > mask=0x0, mask_len=0, ds=0x7fff0408f3b0, verbose=true) at > ../lib/odp-util.c:1402 > #4 0x0000000000460fc4 in format_odp_key_attr (a=a@entry=0xc48580, > ma=ma@entry=0x0, ds=ds@entry=0x7fff0408f3b0, verbose=verbose@entry=true) at > ../lib/odp-util.c:987 > #5 0x00000000004609d7 in odp_flow_format (key=key@entry=0xc48520, > key_len=key_len@entry=140, mask=mask@entry=0x0, mask_len=mask_len@entry=0, > ds=ds@entry=0x7fff0408f3b0, verbose=verbose@entry=true) at > ../lib/odp-util.c:1402 > #6 0x00000000004450f3 in log_flow_message (error=error@entry=2, > operation=operation@entry=0x4d0e73 "flow_del", key=0xc48520, key_len=140, > mask=mask@entry=0x0, > mask_len=mask_len@entry=0, stats=0x0, actions=actions@entry=0x0, > actions_len=actions_len@entry=0, dpif=<optimized out>) at ../lib/dpif.c:1354 > #7 0x00000000004453c9 in log_flow_del_message (dpif=dpif@entry=0xc489c0, > del=del@entry=0x7fff0408f460, error=error@entry=2) at ../lib/dpif.c:1397 > #8 0x0000000000445433 in log_flow_del_message (error=2, del=0x7fff0408f460, > dpif=0xc489c0) at ../lib/dpif.c:1396 > #9 dpif_flow_del__ (dpif=0xc489c0, del=del@entry=0x7fff0408f460) at > ../lib/dpif.c:945 > #10 0x00000000004455ca in dpif_flow_del (dpif=<optimized out>, key=<optimized > out>, key_len=<optimized out>, stats=stats@entry=0x7fff0408f490) at > ../lib/dpif.c:965 > #11 0x000000000041b423 in subfacet_uninstall (subfacet=0xbd76a0) at > ../ofproto/ofproto-dpif.c:4686 > #12 0x0000000000420f18 in facet_remove (facet=facet@entry=0xbd72a0) at > ../ofproto/ofproto-dpif.c:4014 > #13 0x0000000000422f52 in facet_revalidate (facet=facet@entry=0xbd72a0) at > ../ofproto/ofproto-dpif.c:4321 > #14 0x0000000000424b5a in facet_lookup_valid (flow=0x7f3e700020a8, > ofproto=0xc52600) at ../ofproto/ofproto-dpif.c:4203 > #15 handle_flow_miss (n_ops=<synthetic pointer>, ops=0x7fff0408fb60, > miss=0x7f3e70002090) at ../ofproto/ofproto-dpif.c:3339 > #16 handle_flow_misses (fmb=fmb@entry=0x7f3e700008e0, backer=<optimized out>) > at ../ofproto/ofproto-dpif.c:3410 > #17 0x0000000000425196 in handle_upcalls (backer=<optimized out>) at > ../ofproto/ofproto-dpif.c:3565 > #18 dpif_backer_run_fast (backer=<optimized out>) at > ../ofproto/ofproto-dpif.c:1007 > #19 type_run_fast (type=<optimized out>) at ../ofproto/ofproto-dpif.c:1024 > #20 0x00000000004122cf in ofproto_type_run_fast (datapath_type=<optimized > out>, datapath_type@entry=0xc4ef20 "system") at ../ofproto/ofproto.c:1326 > #21 0x00000000004081a5 in bridge_run_fast () at ../vswitchd/bridge.c:2318 > #22 0x00000000004059c5 in main (argc=<optimized out>, argv=<optimized out>) > at ../vswitchd/ovs-vswitchd.c:119 > (gdb) > > > > <http://www.nuvula.ch/> > Marco Kuendig / CEO / Founder > ma...@nuvula.ch <mailto:ma...@nuvula.ch> / +41 78 751 99 71 > <tel:%2B41%2078%20751%2099%2071> > Marco's Google Hangout <https://plus.google.com/hangouts/_/nuvula.ch/marco> > Nuvula AG - Hybrid Clouds > Weierbachstrasse 7b 8193 Eglisau Switzerland > http://www.nuvula.ch <http://www.nuvula.ch/> >> On 31 Mar 2015, at 22:04, Joe Stringer <joestrin...@nicira.com >> <mailto:joestrin...@nicira.com>> wrote: >> >> Great, we're moving. Looks like the gdb version of this is working below. Do >> you get the gdb prompt from there? the command 'bt' should provide the >> backtrace we're after. >> >> On 31 March 2015 at 12:52, Marco Kuendig <ma...@nuvula.ch >> <mailto:ma...@nuvula.ch>> wrote: >> that brought us a step forward. thank Sab. >> >> Important to know is: >> >> I got 4 kvm servers, meshed with openvswitch. I use vxlan for tunnelling. >> >> Sometimes when I restart a domain in kvm, 3 or 4 hosts crash at the same >> time. >> >> I have STP enabled to avoid loops. >> >> >> this is the output now: >> >> root@nuv-vir-kvm-server-1 ~ # gdb /usr/sbin/ovs-vswitchd >> /var/crash/ovs/CoreDump >> GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1 >> Copyright (C) 2014 Free Software Foundation, Inc. >> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html >> <http://gnu.org/licenses/gpl.html>> >> This is free software: you are free to change and redistribute it. >> There is NO WARRANTY, to the extent permitted by law. Type "show copying" >> and "show warranty" for details. >> This GDB was configured as "x86_64-linux-gnu". >> Type "show configuration" for configuration details. >> For bug reporting instructions, please see: >> <http://www.gnu.org/software/gdb/bugs/ >> <http://www.gnu.org/software/gdb/bugs/>>. >> Find the GDB manual and other documentation resources online at: >> <http://www.gnu.org/software/gdb/documentation/ >> <http://www.gnu.org/software/gdb/documentation/>>. >> For help, type "help". >> Type "apropos word" to search for commands related to "word"... >> Reading symbols from /usr/sbin/ovs-vswitchd...Reading symbols from >> /usr/lib/debug//usr/sbin/ovs-vswitchd...done. >> done. >> [New LWP 32725] >> [New LWP 32732] >> [New LWP 32726] >> [New LWP 32730] >> [New LWP 32727] >> [New LWP 32728] >> [New LWP 32729] >> [New LWP 32731] >> [Thread debugging using libthread_db enabled] >> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". >> Core was generated by `ovs-vswitchd unix:/var/run/openvswitch/db.sock >> -vconsole:emer -vsyslog:err -vfi'. >> Program terminated with signal SIGSEGV, Segmentation fault. >> #0 nl_attr_get_size (nla=nla@entry=0x0) at ../lib/netlink.c:506 >> 506 ../lib/netlink.c: No such file or directory. >> >> >> root@nuv-vir-kvm-server-1 ~ # crash /usr/sbin/ovs-vswitchd >> /var/crash/ovs/CoreDump >> >> crash 7.0.3 >> Copyright (C) 2002-2013 Red Hat, Inc. >> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation >> Copyright (C) 1999-2006 Hewlett-Packard Co >> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited >> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. >> Copyright (C) 2005, 2011 NEC Corporation >> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. >> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. >> This program is free software, covered by the GNU General Public License, >> and you are welcome to change it and/or distribute copies of it under >> certain conditions. Enter "help copying" to see the conditions. >> This program has absolutely no warranty. Enter "help warranty" for details. >> >> >> crash: /usr/sbin/ovs-vswitchd: no debugging data available >> >> root@nuv-vir-kvm-server-1 ~ # ll /var/crash/ovs/ >> Architecture Date ExecutableTimestamp ProcCwd >> ProcStatus UserGroups >> CoreDump DistroRelease ProblemType ProcEnviron >> Signal >> CrashCounter ExecutablePath ProcCmdline ProcMaps >> Uname >> >> <http://www.nuvula.ch/> >> Marco Kuendig / CEO / Founder >> ma...@nuvula.ch <mailto:ma...@nuvula.ch> / +41 78 751 99 71 >> <tel:%2B41%2078%20751%2099%2071> >> Marco's Google Hangout <https://plus.google.com/hangouts/_/nuvula.ch/marco> >> Nuvula AG - Hybrid Clouds >> Weierbachstrasse 7b 8193 Eglisau Switzerland >> http://www.nuvula.ch <http://www.nuvula.ch/> >> >>> On 31 Mar 2015, at 21:45, Sabyasachi Sengupta >>> <sabyasachi.sengu...@alcatel-lucent.com >>> <mailto:sabyasachi.sengu...@alcatel-lucent.com>> wrote: >>> >>> >>> Typically Ubuntu does not unpack the crashes. Can you try apport-unpack? >>> # apport-unpack /var/crash/<name> <crash-dir> >>> >>> On Tue, 31 Mar 2015, Marco Kuendig wrote: >>> >>>> thanks Joe and Ben >>>> have done: >>>> 1. installed dgb symbols for kernel....doesn't help >>>> 2. installed debug symbols for openvswitch >>>> no change, gdb and crash still don't work for me. I'm not a dev, need >>>> more help to get that backtrace done. >>>> here some output: >>>> root@nuv-vir-kvm-server-1 ~ # crash >>>> /usr/lib/debug/boot/vmlinux-3.13.0-48-generic >>>> /var/crash/_usr_sbin_ovs-vswitchd.0.crash >>>> crash 7.0.3 >>>> Copyright (C) 2002-2013 Red Hat, Inc. >>>> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation >>>> Copyright (C) 1999-2006 Hewlett-Packard Co >>>> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited >>>> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. >>>> Copyright (C) 2005, 2011 NEC Corporation >>>> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. >>>> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. >>>> This program is free software, covered by the GNU General Public License, >>>> and you are welcome to change it and/or distribute copies of it under >>>> certain conditions. Enter "help copying" to see the conditions. >>>> This program has absolutely no warranty. Enter "help warranty" for >>>> details. >>>> crash: /var/crash/_usr_sbin_ovs-vswitchd.0.crash: not a supported file >>>> format >>>> Usage: >>>> >>>> crash [OPTION]... NAMELIST MEMORY-IMAGE (dumpfile form) >>>> crash [OPTION]... [NAMELIST] (live system form) >>>> Enter "crash -h" for details. >>>> root@nuv-vir-kvm-server-1 ~ # gdb /usr/sbin/ovs-vswitchd >>>> /var/crash/_usr_sbin_ovs-vswitchd.0.crash >>>> GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1 >>>> Copyright (C) 2014 Free Software Foundation, Inc. >>>> License GPLv3+: GNU GPL version 3 or later >>>> <http://gnu.org/licenses/gpl.html <http://gnu.org/licenses/gpl.html>> >>>> This is free software: you are free to change and redistribute it. >>>> There is NO WARRANTY, to the extent permitted by law. Type "show >>>> copying" >>>> and "show warranty" for details. >>>> This GDB was configured as "x86_64-linux-gnu". >>>> Type "show configuration" for configuration details. >>>> For bug reporting instructions, please see: >>>> <http://www.gnu.org/software/gdb/bugs/ >>>> <http://www.gnu.org/software/gdb/bugs/>>. >>>> Find the GDB manual and other documentation resources online at: >>>> <http://www.gnu.org/software/gdb/documentation/ >>>> <http://www.gnu.org/software/gdb/documentation/>>. >>>> For help, type "help". >>>> Type "apropos word" to search for commands related to "word"... >>>> Reading symbols from /usr/sbin/ovs-vswitchd...Reading symbols from >>>> /usr/lib/debug//usr/sbin/ovs-vswitchd...done. >>>> done. >>>> "/var/crash/_usr_sbin_ovs-vswitchd.0.crash" is not a core dump: File >>>> format not recognized >>>> (gdb) q >>>> root@nuv-vir-kvm-server-1 ~ # >>>> Nuvula AG >>>> Marco Kuendig / CEO / Founder ma...@nuvula.ch <mailto:ma...@nuvula.ch> / >>>> +41 78 751 99 71 <tel:%2B41%2078%20751%2099%2071> >>>> Marco's Google Hangout >>>> Nuvula AG - Hybrid Clouds Weierbachstrasse 7b 8193 Eglisau Switzerland >>>> http://www.nuvula.ch <http://www.nuvula.ch/> >>>> >>>> On 31 Mar 2015, at 19:00, Joe Stringer >>>> <joestrin...@nicira.com <mailto:joestrin...@nicira.com>> wrote: >>>> For the 'File format not recognized' problem, you might have better >>>> luck with the 'crash' utility. >>>> $ crash <binary> <crashdump> >>>> On 31 March 2015 at 08:16, Marco Kuendig <ma...@nuvula.ch >>>> <mailto:ma...@nuvula.ch>> wrote: >>>> Have tried this: >>>> http://openvswitch.org/pipermail/discuss/2015-February/016582.html >>>> <http://openvswitch.org/pipermail/discuss/2015-February/016582.html> >>>> this is the output, so doesn't seem to be correct: >>>> root@nuv-vir-kvm-server-2 ~ # gdb /usr/sbin/ovs-vswitchd >>>> /var/crash/_usr_sbin_ovs-vswitchd.0.crash >>>> GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1 >>>> Copyright (C) 2014 Free Software Foundation, Inc. >>>> License GPLv3+: GNU GPL version 3 or later >>>> <http://gnu.org/licenses/gpl.html <http://gnu.org/licenses/gpl.html>> >>>> This is free software: you are free to change and >>>> redistribute it. >>>> There is NO WARRANTY, to the extent permitted by law. Type >>>> "show copying" >>>> and "show warranty" for details. >>>> This GDB was configured as "x86_64-linux-gnu". >>>> Type "show configuration" for configuration details. >>>> For bug reporting instructions, please see: >>>> <http://www.gnu.org/software/gdb/bugs/ >>>> <http://www.gnu.org/software/gdb/bugs/>>. >>>> Find the GDB manual and other documentation resources online >>>> at: >>>> <http://www.gnu.org/software/gdb/documentation/ >>>> <http://www.gnu.org/software/gdb/documentation/>>. >>>> For help, type "help". >>>> Type "apropos word" to search for commands related to >>>> "word"... >>>> Reading symbols from /usr/sbin/ovs-vswitchd...(no debugging >>>> symbols found)...done. >>>> "/var/crash/_usr_sbin_ovs-vswitchd.0.crash" is not a core >>>> dump: File format not recognized >>>> (gdb) bt >>>> No stack. >>>> (gdb) quit >>>> any more hints please ? >>>> thanks >>>> marco >>>> Nuvula AG >>>> Marco Kuendig / CEO / Founder ma...@nuvula.ch <mailto:ma...@nuvula.ch> / >>>> +41 78 751 99 71 <tel:%2B41%2078%20751%2099%2071> >>>> Marco's Google Hangout >>>> Nuvula AG - Hybrid Clouds Weierbachstrasse 7b 8193 Eglisau Switzerland >>>> http://www.nuvula.ch <http://www.nuvula.ch/> >>>> >>>> On 31 Mar 2015, at 17:00, Ben Pfaff >>>> <b...@nicira.com <mailto:b...@nicira.com>> wrote: >>>> Can you get a backtrace for these? >>>> On Tue, Mar 31, 2015 at 7:09 AM, Marco Kuendig >>>> <ma...@nuvula.ch <mailto:ma...@nuvula.ch>> wrote: >>>> Folks, >>>> any chance of having somebody look at these crash >>>> files ? >>>> I have several servers that are loosing network >>>> connectivity because of this. >>>> Downloads: >>>> https://drive.google.com/file/d/0Bx_w1Tf2B5VSRU9yUmRpTDJLVEU/view?usp=sharing >>>> >>>> <https://drive.google.com/file/d/0Bx_w1Tf2B5VSRU9yUmRpTDJLVEU/view?usp=sharing> >>>> Thanks for any hint or fix >>>> marco >>>> Nuvula AG >>>> Marco Kuendig / CEO / Founder ma...@nuvula.ch <mailto:ma...@nuvula.ch> / >>>> +41 78 751 99 71 <tel:%2B41%2078%20751%2099%2071> >>>> Marco's Google Hangout >>>> Nuvula AG - Hybrid Clouds Weierbachstrasse 7b 8193 Eglisau Switzerland >>>> http://www.nuvula.ch <http://www.nuvula.ch/> >>>> _______________________________________________ >>>> discuss mailing list >>>> discuss@openvswitch.org <mailto:discuss@openvswitch.org> >>>> http://openvswitch.org/mailman/listinfo/discuss >>>> <http://openvswitch.org/mailman/listinfo/discuss> >>>> -- >>>> "I don't normally do acked-by's. I think it's my way >>>> of avoiding >>>> getting blamed when it all blows up." Andrew Morton >>>> _______________________________________________ >>>> discuss mailing list >>>> discuss@openvswitch.org <mailto:discuss@openvswitch.org> >>>> http://openvswitch.org/mailman/listinfo/discuss >>>> <http://openvswitch.org/mailman/listinfo/discuss> >> > >
_______________________________________________ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
