I was able to take a backtrace, after attaching gdb to the process. Please see attached.
> -----Original Message----- > From: discuss [mailto:discuss-boun...@openvswitch.org] On Behalf Of Eli > Britstein > Sent: Thursday, 31 March, 2016 10:11 AM > To: Flavio Leitner > Cc: b...@openvswitch.org > Subject: Re: [ovs-discuss] ovs-vswitchd aborted > > Update: > > I changed log levels and now it has some more info in ovs-vswitchd.log. > It shows a debug message in which I can see it received my packet: > > 2016-03-31T08:00:10.087Z|00274|poll_loop|DBG|wakeup due to [POLLIN] on > fd 36 (FIFO pipe:[2229598]) at lib/dpif-netdev.c:2625 > 2016-03-31T08:00:10.124Z|00004|dpif_netdev(pmd60)|DBG|ovs-netdev: > miss upcall: > skb_priority(0),skb_mark(0),recirc_id(0),dp_hash(0),in_port(2),eth(src=3c:97 > :0e:a4:4f:d0,dst=46:61:ec:71:a3:30),eth_type(0x0800),ipv4(src=10.200.201.22 > 0,dst=255.255.255.255,proto=17,tos=0,ttl=128,frag=no),udp(src=60772,dst=2 > 008) > udp,vlan_tci=0x0000,dl_src=3c:97:0e:a4:4f:d0,dl_dst=46:61:ec:71:a3:30,nw_s > rc=10.200.201.220,nw_dst=255.255.255.255,nw_tos=0,nw_ecn=0,nw_ttl=128 > ,tp_src=60772,tp_dst=2008 udp_csum:fbe2 > 2016-03- > 31T08:00:10.124Z|00005|ofproto_dpif_xlate(pmd60)|DBG|/proc/sys/net/co > re/netdev_max_backlog: using 1000 max_backlog > > > > > -----Original Message----- > > From: discuss [mailto:discuss-boun...@openvswitch.org] On Behalf Of > > Eli Britstein > > Sent: Thursday, 31 March, 2016 9:45 AM > > To: Flavio Leitner > > Cc: b...@openvswitch.org > > Subject: Re: [ovs-discuss] ovs-vswitchd aborted > > > > [eli@saturn94 ~]$ ll /proc/sys/kernel/core_* -rw-r--r--. 1 root root 0 > > Mar 21 > > 11:41 /proc/sys/kernel/core_pattern -rw-r--r--. 1 root root 0 Mar 21 > > 11:41 /proc/sys/kernel/core_pipe_limit -rw-r--r--. 1 root root 0 Mar > > 21 11:41 /proc/sys/kernel/core_uses_pid > > > > [eli@saturn94 ~]$ cat /proc/sys/kernel/core_pattern > > |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %e > > > > [eli@saturn94 ~]$ /usr/libexec/abrt-hook-ccpp > > Usage: /usr/libexec/abrt-hook-ccpp SIGNO CORE_SIZE_LIMIT PID UID GID > > TIME BINARY_NAME [HOSTNAME] > > > > [eli@saturn94 ~]$ cat /proc/sys/kernel/core_pipe_limit > > 4 > > [eli@saturn94 ~]$ cat /proc/sys/kernel/core_uses_pid > > 1 > > > > [eli@saturn94 ~]$ coredumpctl list ovs-vswitchd No coredumps found. > > [eli@saturn94 ~]$ coredumpctl list > > No coredumps found. > > > > What's next? > > > > Thanks, > > Eli > > > > > -----Original Message----- > > > From: Flavio Leitner [mailto:f...@sysclose.org] > > > Sent: Thursday, 31 March, 2016 12:20 AM > > > To: Eli Britstein > > > Cc: b...@openvswitch.org > > > Subject: Re: [ovs-discuss] ovs-vswitchd aborted > > > > > > On Wed, Mar 30, 2016 at 01:52:04PM +0000, Eli Britstein wrote: > > > > Hi > > > > > > > > I want to test performance and latency with dpdkr ports. > > > > In order to do so, I configured 2 such ports, and wrote a test > > > > program that > > > lookup for the relevant rings, and streams traffic into the rings. I > > > run this program as a secondary DPDK process. > > > > Once I invoke the program, and I enqueuer mbufs to the rings, ovs- > > > vswitchd is "killed by SIGABRT (core dumped)". > > > > I first thought my packet might be corrupted, so I captured a > > > > valid packet > > > and initialize my packets with the bytes of it. > > > > I have tried to invoke it with a log file, but there is no message > > > > there when > > > the problem occurs. Also, no message in "dmesg -w" when the problem > > > occurs. > > > > I also tried to invoke it after "ulimit -c unlimited", but I could > > > > not find any > > > core dump. > > > > > > You might want to look at /proc/sys/kernel/core_* and coredumpctl(1). > > > fbl > > > > > > > > > > ---------------------------------------------------------------------- > > ------------------------ > > --------------------------------------------------- > > This email and any files transmitted and/or attachments with it are > > confidential and proprietary information of Toga Networks Ltd., and > > intended solely for the use of the individual or entity to whom they > > are addressed. > > If you have received this email in error please notify the system manager. > > This message contains confidential information of Toga Networks Ltd., > > and is intended only for the individual named. If you are not the > > named addressee you should not disseminate, distribute or copy this > > e-mail. Please notify the sender immediately by e-mail if you have > > received this e-mail by mistake and delete this e-mail from your > > system. If you are not the intended recipient you are notified that > > disclosing, copying, distributing or taking any action in reliance on the > contents of this information is strictly prohibited. > > ---------------------------------------------------------------------- > > ------------------------ > > -------------------------------------------------- > > > > _______________________________________________ > > discuss mailing list > > discuss@openvswitch.org > > http://openvswitch.org/mailman/listinfo/discuss > > ---------------------------------------------------------------------- > > ------------------------ > > --------------------------------------------------- > > This email and any files transmitted and/or attachments with it are > > confidential and proprietary information of Toga Networks Ltd., and > > intended solely for the use of the individual or entity to whom they > > are addressed. > > If you have received this email in error please notify the system manager. > > This message contains confidential information of Toga Networks Ltd., > > and is intended only for the individual named. If you are not the > > named addressee you should not disseminate, distribute or copy this > > e-mail. Please notify the sender immediately by e-mail if you have > > received this e-mail by mistake and delete this e-mail from your > > system. If you are not the intended recipient you are notified that > > disclosing, copying, distributing or taking any action in reliance on the > contents of this information is strictly prohibited. > > ---------------------------------------------------------------------- > > ------------------------ > > -------------------------------------------------- > > ---------------------------------------------------------------------------------------------- > --------------------------------------------------- > This email and any files transmitted and/or attachments with it are > confidential and proprietary information of Toga Networks Ltd., and > intended solely for the use of the individual or entity to whom they are > addressed. > If you have received this email in error please notify the system manager. > This message contains confidential information of Toga Networks Ltd., and is > intended only for the individual named. If you are not the named addressee > you should not disseminate, distribute or copy this e-mail. Please notify the > sender immediately by e-mail if you have received this e-mail by mistake and > delete this e-mail from your system. If you are not the intended recipient > you are notified that disclosing, copying, distributing or taking any action > in > reliance on the contents of this information is strictly prohibited. > ---------------------------------------------------------------------------------------------- > -------------------------------------------------- > > ---------------------------------------------------------------------------------------------- > --------------------------------------------------- > This email and any files transmitted and/or attachments with it are > confidential and proprietary information of Toga Networks Ltd., and > intended solely for the use of the individual or entity to whom they are > addressed. > If you have received this email in error please notify the system manager. > This message contains confidential information of Toga Networks Ltd., and is > intended only for the individual named. If you are not the named addressee > you should not disseminate, distribute or copy this e-mail. Please notify the > sender immediately by e-mail if you have received this e-mail by mistake and > delete this e-mail from your system. If you are not the intended recipient > you are notified that disclosing, copying, distributing or taking any action > in > reliance on the contents of this information is strictly prohibited. > ---------------------------------------------------------------------------------------------- > -------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------- This email and any files transmitted and/or attachments with it are confidential and proprietary information of Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. ------------------------------------------------------------------------------------------------------------------------------------------------
[root@saturn94 openvswitch-2.5.0]# gdb -p 40400 GNU gdb (GDB) Fedora 7.8.1-30.fc21 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word". Attaching to process 40400 Reading symbols from /home/eli/ovs/openvswitch-2.5.0/vswitchd/ovs-vswitchd...done. Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [New LWP 40473] [New LWP 40471] [New LWP 40470] [New LWP 40469] [New LWP 40468] [New LWP 40467] [New LWP 40466] [New LWP 40465] [New LWP 40464] [New LWP 40463] [New LWP 40462] [New LWP 40461] [New LWP 40460] [New LWP 40459] [New LWP 40458] [New LWP 40457] [New LWP 40456] [New LWP 40455] [New LWP 40454] [New LWP 40453] [New LWP 40452] [New LWP 40451] [New LWP 40450] [New LWP 40449] [New LWP 40448] [New LWP 40447] [New LWP 40446] [New LWP 40445] [New LWP 40444] [New LWP 40407] [New LWP 40403] [New LWP 40402] [New LWP 40401] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 0x00007f7869669c8d in poll () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install glibc-2.20-8.fc21.x86_64 (gdb) cont Continuing. Program received signal SIGABRT, Aborted. [Switching to Thread 0x7f77dffff700 (LWP 40473)] 0x00007f78695a98d7 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install libgcc-4.9.2-6.fc21.x86_64 (gdb) bt #0 0x00007f78695a98d7 in raise () from /lib64/libc.so.6 #1 0x00007f78695ab53a in abort () from /lib64/libc.so.6 #2 0x00007f78695ecda3 in __libc_message () from /lib64/libc.so.6 #3 0x00007f78695f89f5 in free () from /lib64/libc.so.6 #4 0x00000000005b6350 in dp_packet_uninit (b=<optimized out>) at lib/dp-packet.c:115 #5 0x000000000068187d in dp_packet_delete (b=0x7f76d9521400) at lib/dp-packet.h:160 #6 netdev_dpdk_send__ (may_steal=<optimized out>, cnt=1, pkts=<optimized out>, qid=0, dev=0x7f76da7b2b40) at lib/netdev-dpdk.c:1324 #7 netdev_dpdk_ring_send (netdev_=0x7f76da7b2b40, qid=<optimized out>, pkts=<optimized out>, cnt=1, may_steal=<optimized out>) at lib/netdev-dpdk.c:2195 #8 0x00000000005e2ff0 in netdev_send (netdev=<optimized out>, qid=<optimized out>, buffers=buffers@entry=0x7f77dfffe970, cnt=cnt@entry=1, may_steal=may_steal@entry=true) at lib/netdev.c:742 #9 0x00000000005bcdec in dp_execute_cb (aux_=aux_@entry=0x7f77dfff7580, packets=packets@entry=0x7f77dfffe970, cnt=cnt@entry=1, a=a@entry=0x7f77dfff7638, may_steal=true) at lib/dpif-netdev.c:3655 #10 0x00000000005edbc3 in odp_execute_actions (dp=dp@entry=0x7f77dfff7580, packets=packets@entry=0x7f77dfffe970, cnt=cnt@entry=1, steal=steal@entry=true, actions=<optimized out>, actions_len=<optimized out>, dp_execute_action=0x5bcc50 <dp_execute_cb>) at lib/odp-execute.c:535 #11 0x00000000005bc15d in dp_netdev_execute_actions (actions_len=<optimized out>, actions=<optimized out>, may_steal=true, cnt=1, packets=0x7f77dfffe970, pmd=0x21024f10) at lib/dpif-netdev.c:3802 #12 fast_path_processing (pmd=pmd@entry=0x21024f10, packets=packets@entry=0x7f77dfffe970, cnt=32, keys=keys@entry=0x7f77dfffa9e0, batches=batches@entry=0x7f77dfff86d0, n_batches=n_batches@entry=0x7f77dfffe918) at lib/dpif-netdev.c:3460 #13 0x00000000005bc5b7 in dp_netdev_input__ (pmd=pmd@entry=0x21024f10, packets=packets@entry=0x7f77dfffe970, cnt=<optimized out>, md_is_valid=md_is_valid@entry=false, port_no=2) at lib/dpif-netdev.c:3543 #14 0x00000000005bc68c in dp_netdev_input (port_no=<optimized out>, cnt=<optimized out>, packets=0x7f77dfffe970, pmd=0x21024f10) at lib/dpif-netdev.c:3560 #15 dp_netdev_process_rxq_port (pmd=pmd@entry=0x21024f10, rxq=<optimized out>, port=<optimized out>, port=<optimized out>) at lib/dpif-netdev.c:2564 #16 0x00000000005bca11 in pmd_thread_main (f_=0x21024f10) at lib/dpif-netdev.c:2688 #17 0x0000000000630734 in ovsthread_wrapper (aux_=<optimized out>) at lib/ovs-thread.c:340 #18 0x00007f7869e4952a in start_thread () from /lib64/libpthread.so.0 #19 0x00007f786967522d in clone () from /lib64/libc.so.6
_______________________________________________ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
