Hi, all:
Three memory errors occured after processing "ovs-appctl exit".
Our OVS configure below:
[root@hs237 ~]# ovs-vsctl show
750fc78f-3f6a-4cae-a429-4fde1560d749
Bridge br_bond
datapath_type: netdev
Port patch-to-br0
Interface patch-to-br0
type: patch
options: {peer=patch-to-br_bond}
Port br_bond
Interface br_bond
type: internal
Port dpdk-bond0
Interface dpdk-nic2
type: dpdk
options: {dpdk-devargs="0000:1a:00.1", n_rxq="1",
n_txq_descs="32"}
Interface dpdk-nic1
type: dpdk
options: {dpdk-devargs="0000:1a:00.0", n_rxq="1",
n_txq_descs="32"}
Bridge br0
datapath_type: netdev
Port br0
Interface br0
type: internal
Port vm-20ac9c031ea5
Interface vm-20ac9c031ea5
type: dpdkvhostuserclient
options:
{vhost-server-path="/var/run/openvswitch/vm-20ac9c031ea5.sock"}
Port vm-20ac9c031ea6
Interface vm-20ac9c031ea6
type: dpdkvhostuserclient
options:
{vhost-server-path="/var/run/openvswitch/vm-20ac9c031ea6.sock"}
Port patch-to-br_bond
Interface patch-to-br_bond
type: patch
options: {peer=patch-to-br0}
ovs_version: "2.14.1"
[root@hs237 ~]#
Three "heap-use-after-free" errors:
1:
=================================================================
==2479103==ERROR: AddressSanitizer: heap-use-after-free on address
0x61a0000084b0 at pc 0x000005404e58 bp 0x7fffee8f4e60 sp 0x7fffee8f4e50
READ of size 8 at 0x61a0000084b0 thread T5 (urcu2)
#0 0x5404e57 in rule_destroy_cb ofproto/ofproto.c:2959
#1 0x5779998 (/usr/sbin/ovs-vswitchd+0x5779998) ovsrcu_call_postponed:348
#2 0x5779c33 (/usr/sbin/ovs-vswitchd+0x5779c33) ovsrcu_postpone_thread:364
#3 0x5780c24 (/usr/sbin/ovs-vswitchd+0x5780c24)
#4 0x7ffff5bcd149 in start_thread (/lib64/libpthread.so.0+0x8149)
#5 0x7ffff4c22f22 in clone (/lib64/libc.so.6+0xfcf22)
0x61a0000084b0 is located 48 bytes inside of 1392-byte region
[0x61a000008480,0x61a0000089f0)
freed by thread T5 (urcu2) here:
#0 0x7ffff6ef57e0 in __interceptor_free (/lib64/libasan.so.5+0xef7e0)
#1 0x5779998 (/usr/sbin/ovs-vswitchd+0x5779998) ovsrcu_call_postponed:348
#2 0x5779c33 (/usr/sbin/ovs-vswitchd+0x5779c33) ovsrcu_postpone_thread:364
#3 0x5780c24 (/usr/sbin/ovs-vswitchd+0x5780c24)
#4 0x7ffff5bcd149 in start_thread (/lib64/libpthread.so.0+0x8149)
previously allocated by thread T0 here:
#0 0x7ffff6ef5db0 in calloc (/lib64/libasan.so.5+0xefdb0)
#1 0x58373d2 (/usr/sbin/ovs-vswitchd+0x58373d2)
#2 0x54399bd (/usr/sbin/ovs-vswitchd+0x54399bd)
#3 0x5415478 in ofproto_create ofproto/ofproto.c:498
#4 0x53dabd8 in bridge_reconfigure vswitchd/bridge.c:903
#5 0x53e79de in bridge_run vswitchd/bridge.c:3355
#6 0x90d074 (/usr/sbin/ovs-vswitchd+0x90d074)
#7 0x7ffff4b497b2 in __libc_start_main (/lib64/libc.so.6+0x237b2)
Thread T5 (urcu2) created by T4 (dpdk_watchdog1) here:
#0 0x7ffff6e58ea3 in __interceptor_pthread_create
(/lib64/libasan.so.5+0x52ea3)
#1 0x5780ef8 (/usr/sbin/ovs-vswitchd+0x5780ef8)
#2 0x5779be7 (/usr/sbin/ovs-vswitchd+0x5779be7)
#3 0x583ec6f (/usr/sbin/ovs-vswitchd+0x583ec6f)
#4 0x58e80a0 (/usr/sbin/ovs-vswitchd+0x58e80a0)
#5 0x5780c24 (/usr/sbin/ovs-vswitchd+0x5780c24)
#6 0x7ffff5bcd149 in start_thread (/lib64/libpthread.so.0+0x8149)
Thread T4 (dpdk_watchdog1) created by T0 here:
#0 0x7ffff6e58ea3 in __interceptor_pthread_create
(/lib64/libasan.so.5+0x52ea3)
#1 0x5780ef8 (/usr/sbin/ovs-vswitchd+0x5780ef8)
#2 0x58d9640 (/usr/sbin/ovs-vswitchd+0x58d9640)
#3 0x563a33a (/usr/sbin/ovs-vswitchd+0x563a33a)
#4 0x58eb0cd (/usr/sbin/ovs-vswitchd+0x58eb0cd)
#5 0x58ccfb4 (/usr/sbin/ovs-vswitchd+0x58ccfb4)
#6 0x58ce1b0 (/usr/sbin/ovs-vswitchd+0x58ce1b0)
#7 0x53e74aa in bridge_run vswitchd/bridge.c:3317
#8 0x90d074 (/usr/sbin/ovs-vswitchd+0x90d074)
#9 0x7ffff4b497b2 in __libc_start_main (/lib64/libc.so.6+0x237b2)
SUMMARY: AddressSanitizer: heap-use-after-free ofproto/ofproto.c:2959 in
rule_destroy_cb
Shadow bytes around the buggy address:
0x0c347fff9040: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c347fff9050: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c347fff9060: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c347fff9070: fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa fa
0x0c347fff9080: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c347fff9090: fd fd fd fd fd fd[fd]fd fd fd fd fd fd fd fd fd
0x0c347fff90a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c347fff90b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c347fff90c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c347fff90d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c347fff90e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
2:
=================================================================
==2505462==ERROR: AddressSanitizer: heap-use-after-free on address
0x61a000008ba8 at pc 0x00000545b342 bp 0x7fffe0fc1910 sp 0x7fffe0fc1900
READ of size 8 at 0x61a000008ba8 thread T31 (revalidator28)
#0 0x545b341 in ofproto_dpif_credit_table_stats ofproto/ofproto-dpif.c:4347
#1 0x54f51e4 in xlate_push_stats_entry
ofproto/ofproto-dpif-xlate-cache.c:135
#2 0x54f5c56 in xlate_push_stats ofproto/ofproto-dpif-xlate-cache.c:218
#3 0x54a182e in revalidate_ukey ofproto/ofproto-dpif-upcall.c:2301
#4 0x54ac476 in revalidate ofproto/ofproto-dpif-upcall.c:2713
#5 0x54af9f6 in udpif_revalidator ofproto/ofproto-dpif-upcall.c:938
#6 0x5780c24 (/usr/sbin/ovs-vswitchd+0x5780c24)
#7 0x7ffff5bcd149 in start_thread (/lib64/libpthread.so.0+0x8149)
#8 0x7ffff4c22f22 in clone (/lib64/libc.so.6+0xfcf22)
0x61a000008ba8 is located 296 bytes inside of 1392-byte region
[0x61a000008a80,0x61a000008ff0)
freed by thread T5 (urcu2) here:
#0 0x7ffff6ef57e0 in __interceptor_free (/lib64/libasan.so.5+0xef7e0)
#1 0x5779998 (/usr/sbin/ovs-vswitchd+0x5779998)
#2 0x5779c33 (/usr/sbin/ovs-vswitchd+0x5779c33)
#3 0x5780c24 (/usr/sbin/ovs-vswitchd+0x5780c24)
#4 0x7ffff5bcd149 in start_thread (/lib64/libpthread.so.0+0x8149)
previously allocated by thread T0 here:
#0 0x7ffff6ef5db0 in calloc (/lib64/libasan.so.5+0xefdb0)
#1 0x58373d2 (/usr/sbin/ovs-vswitchd+0x58373d2)
#2 0x54399bd (/usr/sbin/ovs-vswitchd+0x54399bd)
#3 0x5415478 in ofproto_create ofproto/ofproto.c:498
#4 0x53dabd8 in bridge_reconfigure vswitchd/bridge.c:903
#5 0x53e79de in bridge_run vswitchd/bridge.c:3355
#6 0x90d074 (/usr/sbin/ovs-vswitchd+0x90d074)
#7 0x7ffff4b497b2 in __libc_start_main (/lib64/libc.so.6+0x237b2)
Thread T31 (revalidator28) created by T0 here:
#0 0x7ffff6e58ea3 in __interceptor_pthread_create
(/lib64/libasan.so.5+0x52ea3)
#1 0x5780ef8 (/usr/sbin/ovs-vswitchd+0x5780ef8)
#2 0x54a29e9 in udpif_start_threads ofproto/ofproto-dpif-upcall.c:595
#3 0x54a29e9 in udpif_start_threads ofproto/ofproto-dpif-upcall.c:560
#4 0x546ec25 in open_dpif_backer ofproto/ofproto-dpif.c:832
#5 0x5415c9e in ofproto_create ofproto/ofproto.c:550
#6 0x53dabd8 in bridge_reconfigure vswitchd/bridge.c:903
#7 0x53e79de in bridge_run vswitchd/bridge.c:3355
#8 0x90d074 (/usr/sbin/ovs-vswitchd+0x90d074)
#9 0x7ffff4b497b2 in __libc_start_main (/lib64/libc.so.6+0x237b2)
Thread T5 (urcu2) created by T4 (dpdk_watchdog1) here:
#0 0x7ffff6e58ea3 in __interceptor_pthread_create
(/lib64/libasan.so.5+0x52ea3)
#1 0x5780ef8 (/usr/sbin/ovs-vswitchd+0x5780ef8)
#2 0x5779be7 (/usr/sbin/ovs-vswitchd+0x5779be7)
#3 0x583ec6f (/usr/sbin/ovs-vswitchd+0x583ec6f)
#4 0x58e80a0 (/usr/sbin/ovs-vswitchd+0x58e80a0)
#5 0x5780c24 (/usr/sbin/ovs-vswitchd+0x5780c24)
#6 0x7ffff5bcd149 in start_thread (/lib64/libpthread.so.0+0x8149)
Thread T4 (dpdk_watchdog1) created by T0 here:
#0 0x7ffff6e58ea3 in __interceptor_pthread_create
(/lib64/libasan.so.5+0x52ea3)
#1 0x5780ef8 (/usr/sbin/ovs-vswitchd+0x5780ef8)
#2 0x58d9640 (/usr/sbin/ovs-vswitchd+0x58d9640)
#3 0x563a33a (/usr/sbin/ovs-vswitchd+0x563a33a)
#4 0x58eb0cd (/usr/sbin/ovs-vswitchd+0x58eb0cd)
#5 0x58ccfb4 (/usr/sbin/ovs-vswitchd+0x58ccfb4)
#6 0x58ce1b0 (/usr/sbin/ovs-vswitchd+0x58ce1b0)
#7 0x53e74aa in bridge_run vswitchd/bridge.c:3317
#8 0x90d074 (/usr/sbin/ovs-vswitchd+0x90d074)
#9 0x7ffff4b497b2 in __libc_start_main (/lib64/libc.so.6+0x237b2)
SUMMARY: AddressSanitizer: heap-use-after-free ofproto/ofproto-dpif.c:4347 in
ofproto_dpif_credit_table_stats
Shadow bytes around the buggy address:
0x0c347fff9120: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c347fff9130: fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa fa
0x0c347fff9140: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c347fff9150: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c347fff9160: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c347fff9170: fd fd fd fd fd[fd]fd fd fd fd fd fd fd fd fd fd
0x0c347fff9180: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c347fff9190: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c347fff91a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c347fff91b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c347fff91c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==2505462==ABORTING
3:
=================================================================
==2508866==ERROR: AddressSanitizer: heap-use-after-free on address
0x61a000008ba8 at pc 0x00000545b342 bp 0x7fffffffb690 sp 0x7fffffffb680
READ of size 8 at 0x61a000008ba8 thread T0
#0 0x545b341 in ofproto_dpif_credit_table_stats ofproto/ofproto-dpif.c:4347
#1 0x54f51e4 in xlate_push_stats_entry
ofproto/ofproto-dpif-xlate-cache.c:135
#2 0x54f5c56 in xlate_push_stats ofproto/ofproto-dpif-xlate-cache.c:218
#3 0x549f4f2 in push_dp_ops ofproto/ofproto-dpif-upcall.c:2414
#4 0x549fa6e in push_ukey_ops ofproto/ofproto-dpif-upcall.c:2446
#5 0x549ff34 (/usr/sbin/ovs-vswitchd+0x549ff34)
#6 0x556d071 (/usr/sbin/ovs-vswitchd+0x556d071)
#7 0x5579aca (/usr/sbin/ovs-vswitchd+0x5579aca)
#8 0x557bd56 (/usr/sbin/ovs-vswitchd+0x557bd56)
#9 0x557cb07 (/usr/sbin/ovs-vswitchd+0x557cb07)
#10 0x55a0d7f (/usr/sbin/ovs-vswitchd+0x55a0d7f)
#11 0x5455ea2 (/usr/sbin/ovs-vswitchd+0x5455ea2)
#12 0x540cbdb in ofport_destroy ofproto/ofproto.c:2630
#13 0x542b3eb in ofproto_destroy ofproto/ofproto.c:1728
#14 0x53e6eb5 in bridge_exit vswitchd/bridge.c:569
#15 0x90d20f (/usr/sbin/ovs-vswitchd+0x90d20f)
#16 0x7ffff4b497b2 in __libc_start_main (/lib64/libc.so.6+0x237b2)
#17 0x1356d5d (/usr/sbin/ovs-vswitchd+0x1356d5d)
0x61a000008ba8 is located 296 bytes inside of 1392-byte region
[0x61a000008a80,0x61a000008ff0)
freed by thread T5 (urcu2) here:
#0 0x7ffff6ef57e0 in __interceptor_free (/lib64/libasan.so.5+0xef7e0)
#1 0x5779998 (/usr/sbin/ovs-vswitchd+0x5779998)
#2 0x5779c33 (/usr/sbin/ovs-vswitchd+0x5779c33)
#3 0x5780c24 (/usr/sbin/ovs-vswitchd+0x5780c24)
#4 0x7ffff5bcd149 in start_thread (/lib64/libpthread.so.0+0x8149)
previously allocated by thread T0 here:
#0 0x7ffff6ef5db0 in calloc (/lib64/libasan.so.5+0xefdb0)
#1 0x58373d2 (/usr/sbin/ovs-vswitchd+0x58373d2)
#2 0x54399bd (/usr/sbin/ovs-vswitchd+0x54399bd)
#3 0x5415478 in ofproto_create ofproto/ofproto.c:498
#4 0x53dabd8 in bridge_reconfigure vswitchd/bridge.c:903
#5 0x53e79de in bridge_run vswitchd/bridge.c:3355
#6 0x90d074 (/usr/sbin/ovs-vswitchd+0x90d074)
#7 0x7ffff4b497b2 in __libc_start_main (/lib64/libc.so.6+0x237b2)
Thread T5 (urcu2) created by T4 (dpdk_watchdog1) here:
#0 0x7ffff6e58ea3 in __interceptor_pthread_create
(/lib64/libasan.so.5+0x52ea3)
#1 0x5780ef8 (/usr/sbin/ovs-vswitchd+0x5780ef8)
#2 0x5779be7 (/usr/sbin/ovs-vswitchd+0x5779be7)
#3 0x583ec6f (/usr/sbin/ovs-vswitchd+0x583ec6f)
#4 0x58e80a0 (/usr/sbin/ovs-vswitchd+0x58e80a0)
#5 0x5780c24 (/usr/sbin/ovs-vswitchd+0x5780c24)
#6 0x7ffff5bcd149 in start_thread (/lib64/libpthread.so.0+0x8149)
Thread T4 (dpdk_watchdog1) created by T0 here:
#0 0x7ffff6e58ea3 in __interceptor_pthread_create
(/lib64/libasan.so.5+0x52ea3)
#1 0x5780ef8 (/usr/sbin/ovs-vswitchd+0x5780ef8)
#2 0x58d9640 (/usr/sbin/ovs-vswitchd+0x58d9640)
#3 0x563a33a (/usr/sbin/ovs-vswitchd+0x563a33a)
#4 0x58eb0cd (/usr/sbin/ovs-vswitchd+0x58eb0cd)
#5 0x58ccfb4 (/usr/sbin/ovs-vswitchd+0x58ccfb4)
#6 0x58ce1b0 (/usr/sbin/ovs-vswitchd+0x58ce1b0)
#7 0x53e74aa in bridge_run vswitchd/bridge.c:3317
#8 0x90d074 (/usr/sbin/ovs-vswitchd+0x90d074)
#9 0x7ffff4b497b2 in __libc_start_main (/lib64/libc.so.6+0x237b2)
SUMMARY: AddressSanitizer: heap-use-after-free ofproto/ofproto-dpif.c:4347 in
ofproto_dpif_credit_table_stats
Shadow bytes around the buggy address:
0x0c347fff9120: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c347fff9130: fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa fa
0x0c347fff9140: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c347fff9150: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c347fff9160: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c347fff9170: fd fd fd fd fd[fd]fd fd fd fd fd fd fd fd fd fd
0x0c347fff9180: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c347fff9190: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c347fff91a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c347fff91b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c347fff91c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==2508866==ABORTING
And we found that, if use --cleanup, the error can not be reproduced.
After checking the code, we found that udpif_flush can not be called if the
second parameter of function bridge_destroy is false, and the revalidator
threads are running when the bridge is destroyed.
So if the "--cleanup" must be used?
[email protected]
_______________________________________________
discuss mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
