Hi OvS
I'm getting the same exception each time (dp_packet_delete) while 3 iperf3
traffic running between 6 qemu hosts. Also, when I start ovs with valgrind
(--leak-check=yes), I cannot see the problem because of slowness. Any
suggestion would be appreciated. Thanks in advance.
- Volkan
2016-10-08T13:46:47Z|01582|connmgr|INFO|Dropped 1994 log messages in last 6
seconds (most recently, 1 seconds ago) due to excessive rate
2016-10-08T13:46:47Z|01583|connmgr|INFO|e20<->tcp:10.10.10.200:6653: sending
OFPMMFC_INVALID_METER error reply to OFPT_FLOW_MOD message
2016-10-08T13:46:48Z|01584|connmgr|INFO|e20<->tcp:10.10.10.200:6653: 9
flow_mods 10 s ago (9 adds)
2016-10-08T13:46:51Z|01585|connmgr|INFO|Dropped 103375 log messages in last 12
seconds (most recently, 0 seconds ago) due to excessive rate
2016-10-08T13:46:51Z|01586|connmgr|INFO|e20<->tcp:10.10.10.200:6653: dropping
packet-in due to queue overflow
2016-10-08T13:46:53Z|01587|connmgr|INFO|Dropped 1591 log messages in last 6
seconds (most recently, 1 seconds ago) due to excessive rate
2016-10-08T13:46:53Z|01588|connmgr|INFO|e20<->tcp:10.10.10.200:6653: sending
OFPMMFC_INVALID_METER error reply to OFPT_FLOW_MOD message
2016-10-08T13:46:59Z|01589|connmgr|INFO|Dropped 1448 log messages in last 6
seconds (most recently, 1 seconds ago) due to excessive rate
2016-10-08T13:46:59Z|01590|connmgr|INFO|e20<->tcp:10.10.10.200:6653: sending
OFPMMFC_INVALID_METER error reply to OFPT_FLOW_MOD message
2016-10-08T13:47:00Z|01591|connmgr|INFO|c02<->tcp:10.10.10.200:6653: 1
flow_mods 29 s ago (1 deletes)
2016-10-08T13:47:00Z|01592|connmgr|INFO|a21<->tcp:10.10.10.200:6653: 1
flow_mods 24 s ago (1 deletes)
2016-10-08T13:47:00Z|01593|connmgr|INFO|a01<->tcp:10.10.10.200:6653: 1
flow_mods 24 s ago (1 deletes)
2016-10-08T13:47:03Z|01594|connmgr|INFO|Dropped 105195 log messages in last 12
seconds (most recently, 0 seconds ago) due to excessive rate
2016-10-08T13:47:03Z|01595|connmgr|INFO|e20<->tcp:10.10.10.200:6653: dropping
packet-in due to queue overflow
2016-10-08T13:47:05Z|01596|connmgr|INFO|Dropped 1301 log messages in last 6
seconds (most recently, 1 seconds ago) due to excessive rate
2016-10-08T13:47:05Z|01597|connmgr|INFO|e20<->tcp:10.10.10.200:6653: sending
OFPMMFC_INVALID_METER error reply to OFPT_FLOW_MOD message
2016-10-08T13:47:11Z|01598|connmgr|INFO|e00<->tcp:10.10.10.200:6653: 13318
flow_mods in the 33 s starting 34 s ago (13316 adds, 2 deletes)
2016-10-08T13:47:11Z|01599|connmgr|INFO|a20<->tcp:10.10.10.200:6653: 13334
flow_mods in the last 32 s (13330 adds, 4 deletes)
2016-10-08T13:47:11Z|01600|connmgr|INFO|a00<->tcp:10.10.10.200:6653: 13334
flow_mods in the last 32 s (13330 adds, 4 deletes)
2016-10-08T13:47:11Z|01601|connmgr|INFO|c01<->tcp:10.10.10.200:6653: 13335
flow_mods in the last 59 s (13330 adds, 5 deletes)
2016-10-08T13:47:11Z|01602|connmgr|INFO|e21<->tcp:10.10.10.200:6653: 2
flow_mods in the 25 s starting 59 s ago (2 deletes)
2016-10-08T13:47:11Z|01603|connmgr|INFO|Dropped 6345 log messages in last 6
seconds (most recently, 1 seconds ago) due to excessive rate
2016-10-08T13:47:11Z|01604|connmgr|INFO|e00<->tcp:10.10.10.200:6653: sending
OFPMMFC_BAD_BURST error reply to OFPT_METER_MOD message
*** Error in `/usr/local/sbin/ovs-vswitchd': double free or corruption (out):
0x00000000216a8f90 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x77725)[0x7ffff6f48725]
/lib/x86_64-linux-gnu/libc.so.6(+0x7ff4a)[0x7ffff6f50f4a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7ffff6f54abc]
/usr/local/sbin/ovs-vswitchd[0x59f1c6]
/usr/local/sbin/ovs-vswitchd[0x5a3783]
/usr/local/sbin/ovs-vswitchd[0x5ca5e3]
/usr/local/sbin/ovs-vswitchd[0x59ce26]
/usr/local/sbin/ovs-vswitchd[0x58c0f4]
/usr/local/sbin/ovs-vswitchd[0x5912f9]
/usr/local/sbin/ovs-vswitchd[0x40da7d]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7ffff6ef1830]
/usr/local/sbin/ovs-vswitchd[0x40eab9]
======= Memory map: ========
00400000-007cc000 r-xp 00000000 08:02 14813834
/usr/local/sbin/ovs-vswitchd
009cc000-009cd000 r--p 003cc000 08:02 14813834
/usr/local/sbin/ovs-vswitchd
009cd000-00a07000 rw-p 003cd000 08:02 14813834
/usr/local/sbin/ovs-vswitchd
00a07000-21a19000 rw-p 00000000 00:00 0 [heap]
7fff5c000000-7fff5c021000 rw-p 00000000 00:00 0
7fff5c021000-7fff60000000 ---p 00000000 00:00 0
7fff64000000-7fff6c000000 rw-s 00000000 00:34 17314572
/mnt/huge_2mb/qemu_back_mem._objects_mem.xGh4Vu (deleted)
7fff6c000000-7fffa9a00000 rw-s 00000000 00:26 16715628
/dev/hugepages/rtemap_1
7fffab000000-7fffb3000000 rw-s 00000000 00:34 17305550
/mnt/huge_2mb/qemu_back_mem._objects_mem.hVwHyv (deleted)
7fffb3000000-7fffbb000000 rw-s 00000000 00:34 17279527
/mnt/huge_2mb/qemu_back_mem._objects_mem.cUDgVs (deleted)
7fffbb000000-7fffc3000000 rw-s 00000000 00:34 17265061
/mnt/huge_2mb/qemu_back_mem._objects_mem.Bp5DMd (deleted)
7fffc3000000-7fffcb000000 rw-s 00000000 00:34 17240200
/mnt/huge_2mb/qemu_back_mem._objects_mem.1WndGC (deleted)
7fffcb000000-7fffd3000000 rw-s 00000000 00:34 17198478
/mnt/huge_2mb/qemu_back_mem._objects_mem.9Hfncw (deleted)
7fffd3000000-7fffd5600000 rw-s 00000000 00:26 16715627
/dev/hugepages/rtemap_0
7fffd8000000-7fffd8022000 rw-p 00000000 00:00 0
7fffd8022000-7fffdc000000 ---p 00000000 00:00 0
7fffdc000000-7fffdc463000 rw-p 00000000 00:00 0
7fffdc463000-7fffe0000000 ---p 00000000 00:00 0
7fffe0000000-7fffe0021000 rw-p 00000000 00:00 0
7fffe0021000-7fffe4000000 ---p 00000000 00:00 0
7fffe63ea000-7fffe6400000 r-xp 00000000 08:02 5509547
/lib/x86_64-linux-gnu/libgcc_s.so.1
7fffe6400000-7fffe65ff000 ---p 00016000 08:02 5509547
/lib/x86_64-linux-gnu/libgcc_s.so.1
7fffe65ff000-7fffe6600000 rw-p 00015000 08:02 5509547
/lib/x86_64-linux-gnu/libgcc_s.so.1
7fffe6600000-7fffe6800000 rw-s 00000000 00:34 17314572
/mnt/huge_2mb/qemu_back_mem._objects_mem.xGh4Vu (deleted)
7fffe6800000-7fffe6a00000 rw-s 00000000 00:34 17305550
/mnt/huge_2mb/qemu_back_mem._objects_mem.hVwHyv (deleted)
7fffe6a00000-7fffe6c00000 rw-s 00000000 00:34 17279527
/mnt/huge_2mb/qemu_back_mem._objects_mem.cUDgVs (deleted)
7fffe6c00000-7fffe6e00000 rw-s 00000000 00:34 17265061
/mnt/huge_2mb/qemu_back_mem._objects_mem.Bp5DMd (deleted)
7fffe6ffe000-7fffe6fff000 ---p 00000000 00:00 0
7fffe6fff000-7fffe77ff000 rw-p 00000000 00:00 0
7fffe77ff000-7fffe7800000 ---p 00000000 00:00 0
7fffe7800000-7fffe8000000 rw-p 00000000 00:00 0
7fffe8000000-7fffe8021000 rw-p 00000000 00:00 0
7fffe8021000-7fffec000000 ---p 00000000 00:00 0
7fffec000000-7fffec021000 rw-p 00000000 00:00 0
7fffec021000-7ffff0000000 ---p 00000000 00:00 0
7ffff0000000-7ffff0021000 rw-p 00000000 00:00 0
7ffff0021000-7ffff4000000 ---p 00000000 00:00 0
7ffff4200000-7ffff4400000 rw-s 00000000 00:34 17240200
/mnt/huge_2mb/qemu_back_mem._objects_mem.1WndGC (deleted)
7ffff4400000-7ffff4600000 rw-s 00000000 00:34 17198478
/mnt/huge_2mb/qemu_back_mem._objects_mem.9Hfncw (deleted)
7ffff46cc000-7ffff46cd000 ---p 00000000 00:00 0
7ffff46cd000-7ffff4ecd000 rw-p 00000000 00:00 0
7ffff4ecd000-7ffff4ece000 ---p 00000000 00:00 0
7ffff4ece000-7ffff56ce000 rw-p 00000000 00:00 0
7ffff56ce000-7ffff56cf000 ---p 00000000 00:00 0
7ffff56cf000-7ffff5ecf000 rw-p 00000000 00:00 0
7ffff5ecf000-7ffff5ed0000 ---p 00000000 00:00 0
7ffff5ed0000-7ffff66d0000 rw-p 00000000 00:00 0
7ffff66d0000-7ffff66d1000 ---p 00000000 00:00 0
7ffff66d1000-7ffff6ed1000 rw-p 00000000 00:00 0
7ffff6ed1000-7ffff7091000 r-xp 00000000 08:02 5509509
/lib/x86_64-linux-gnu/libc-2.23.so
7ffff7091000-7ffff7290000 ---p 001c0000 08:02 5509509
/lib/x86_64-linux-gnu/libc-2.23.so
7ffff7290000-7ffff7294000 r--p 001bf000 08:02 5509509
/lib/x86_64-linux-gnu/libc-2.23.so
7ffff7294000-7ffff7296000 rw-p 001c3000 08:02 5509509
/lib/x86_64-linux-gnu/libc-2.23.so
7ffff7296000-7ffff729a000 rw-p 00000000 00:00 0
7ffff729a000-7ffff72a4000 r-xp 00000000 08:02 14688949
/usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
7ffff72a4000-7ffff74a3000 ---p 0000a000 08:02 14688949
/usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
7ffff74a3000-7ffff74a4000 r--p 00009000 08:02 14688949
/usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
7ffff74a4000-7ffff74a5000 rw-p 0000a000 08:02 14688949
/usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
7ffff74a5000-7ffff75ad000 r-xp 00000000 08:02 5509579
/lib/x86_64-linux-gnu/libm-2.23.so
7ffff75ad000-7ffff77ac000 ---p 00108000 08:02 5509579
/lib/x86_64-linux-gnu/libm-2.23.so
7ffff77ac000-7ffff77ad000 r--p 00107000 08:02 5509579
/lib/x86_64-linux-gnu/libm-2.23.so
7ffff77ad000-7ffff77ae000 rw-p 00108000 08:02 5509579
/lib/x86_64-linux-gnu/libm-2.23.so
7ffff77ae000-7ffff77b5000 r-xp 00000000 08:02 5509663
/lib/x86_64-linux-gnu/librt-2.23.so
7ffff77b5000-7ffff79b4000 ---p 00007000 08:02 5509663
/lib/x86_64-linux-gnu/librt-2.23.so
7ffff79b4000-7ffff79b5000 r--p 00006000 08:02 5509663
/lib/x86_64-linux-gnu/librt-2.23.so
7ffff79b5000-7ffff79b6000 rw-p 00007000 08:02 5509663
/lib/x86_64-linux-gnu/librt-2.23.so
7ffff79b6000-7ffff79ce000 r-xp 00000000 08:02 5509655
/lib/x86_64-linux-gnu/libpthread-2.23.so
7ffff79ce000-7ffff7bcd000 ---p 00018000 08:02 5509655
/lib/x86_64-linux-gnu/libpthread-2.23.so
7ffff7bcd000-7ffff7bce000 r--p 00017000 08:02 5509655
/lib/x86_64-linux-gnu/libpthread-2.23.so
7ffff7bce000-7ffff7bcf000 rw-p 00018000 08:02 5509655
/lib/x86_64-linux-gnu/libpthread-2.23.so
7ffff7bcf000-7ffff7bd3000 rw-p 00000000 00:00 0
7ffff7bd3000-7ffff7bd6000 r-xp 00000000 08:02 5509533
/lib/x86_64-linux-gnu/libdl-2.23.so
7ffff7bd6000-7ffff7dd5000 ---p 00003000 08:02 5509533
/lib/x86_64-linux-gnu/libdl-2.23.so
7ffff7dd5000-7ffff7dd6000 r--p 00002000 08:02 5509533
/lib/x86_64-linux-gnu/libdl-2.23.so
7ffff7dd6000-7ffff7dd7000 rw-p 00003000 08:02 5509533
/lib/x86_64-linux-gnu/libdl-2.23.so
7ffff7dd7000-7ffff7dfd000 r-xp 00000000 08:02 5509481
/lib/x86_64-linux-gnu/ld-2.23.so
7ffff7f63000-7ffff7f9e000 rw-s 00000000 00:13 703
/run/.rte_hugepage_info
7ffff7f9e000-7ffff7fd7000 rw-s 00000000 00:13 702
/run/.rte_config
7ffff7fd7000-7ffff7fdc000 rw-p 00000000 00:00 0
7ffff7ff5000-7ffff7ff8000 rw-p 00000000 00:00 0
7ffff7ff8000-7ffff7ffa000 r--p 00000000 00:00 0 [vvar]
7ffff7ffa000-7ffff7ffc000 r-xp 00000000 00:00 0 [vdso]
7ffff7ffc000-7ffff7ffd000 r--p 00025000 08:02 5509481
/lib/x86_64-linux-gnu/ld-2.23.so
7ffff7ffd000-7ffff7ffe000 rw-p 00026000 08:02 5509481
/lib/x86_64-linux-gnu/ld-2.23.so
7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0
7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0 [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
Thread 1 "ovs-vswitchd" received signal SIGABRT, Aborted.
0x00007ffff6f06418 in __GI_raise (sig=sig@entry=6) at
../sysdeps/unix/sysv/linux/raise.c:54
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) i threads
Id Target Id Frame
* 1 Thread 0x7ffff7fd7b00 (LWP 7259) "ovs-vswitchd" 0x00007ffff6f06418 in
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
2 Thread 0x7ffff6ed0700 (LWP 7263) "eal-intr-thread" 0x00007ffff6fd8153 in
epoll_wait () at ../sysdeps/unix/syscall-template.S:84
3 Thread 0x7ffff66cf700 (LWP 7264) "dpdk_watchdog1" 0x00007ffff6f9c8dd in
nanosleep () at ../sysdeps/unix/syscall-template.S:84
4 Thread 0x7ffff5ece700 (LWP 7265) "vhost_thread3" 0x00007ffff6fcdd13 in
select () at ../sysdeps/unix/syscall-template.S:84
5 Thread 0x7ffff56cd700 (LWP 7266) "urcu2" 0x00007ffff6fcbe8d in poll ()
at ../sysdeps/unix/syscall-template.S:84
36 Thread 0x7fffe77fe700 (LWP 7460) "pmd34" 0x000000000048af27 in
rte_vhost_dequeue_burst ()
127 Thread 0x7ffff4ecc700 (LWP 8261) "handler125" 0x00007ffff6fcbe8d in poll
() at ../sysdeps/unix/syscall-template.S:84
128 Thread 0x7fffe7fff700 (LWP 8262) "revalidator126" 0x00007ffff6fcbe8d in
poll () at ../sysdeps/unix/syscall-template.S:84
(gdb) where
#0 0x00007ffff6f06418 in __GI_raise (sig=sig@entry=6) at
../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007ffff6f0801a in __GI_abort () at abort.c:89
#2 0x00007ffff6f4872a in __libc_message (do_abort=do_abort@entry=2,
fmt=fmt@entry=0x7ffff70616b0 "*** Error in `%s': %s: 0x%s ***\n")
at ../sysdeps/posix/libc_fatal.c:175
#3 0x00007ffff6f50f4a in malloc_printerr (ar_ptr=<optimized out>,
ptr=<optimized out>, str=0x7ffff70617c0 "double free or corruption (out)",
action=3) at malloc.c:5007
#4 _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at
malloc.c:3868
#5 0x00007ffff6f54abc in __GI___libc_free (mem=<optimized out>) at
malloc.c:2969
#6 0x000000000059f1c6 in dp_packet_delete (b=0x211fc2a0) at
./lib/dp-packet.h:165
#7 handle_packet_out (ofconn=ofconn@entry=0x2178f140, oh=oh@entry=0x219a5db0)
at ofproto/ofproto.c:3496
#8 0x00000000005a3783 in handle_openflow__ (msg=0x21291040, ofconn=0x2178f140)
at ofproto/ofproto.c:7344
#9 handle_openflow (ofconn=0x2178f140, ofp_msg=0x21291040) at
ofproto/ofproto.c:7516
#10 0x00000000005ca5e3 in ofconn_run (handle_openflow=0x5a2f70
<handle_openflow>, ofconn=0x2178f140) at ofproto/connmgr.c:1378
#11 connmgr_run (mgr=0x2181cf50, handle_openflow=handle_openflow@entry=0x5a2f70
<handle_openflow>) at ofproto/connmgr.c:320
#12 0x000000000059ce26 in ofproto_run (p=0x21832330) at ofproto/ofproto.c:1820
#13 0x000000000058c0f4 in bridge_run__ () at vswitchd/bridge.c:2880
#14 0x00000000005912f9 in bridge_run () at vswitchd/bridge.c:2937
#15 0x000000000040da7d in main (argc=3, argv=0x7fffffffe518) at
vswitchd/ovs-vswitchd.c:112
_______________________________________________
discuss mailing list
discuss@openvswitch.org
http://openvswitch.org/mailman/listinfo/discuss
