On 9/22/21 17:17, David Marchand wrote: > Hello, > > That is more a FYI, in case nobody noticed. > > I did not analyse it. > GHA ASan build caught it 3 times over past days.
Yeah, it's kind of a "well-known" issue. And there are several issues here. One that ofproto itself needs RCu/refcounting and another that meters inside the ofproto needs something like that too. Some discussions around: https://patchwork.ozlabs.org/project/openvswitch/patch/[email protected]/ https://patchwork.ozlabs.org/project/openvswitch/patch/[email protected]/ And my previous report: https://mail.openvswitch.org/pipermail/ovs-dev/2021-February/380582.html Current state is 'stale'. Mainly because of me being overloaded by other stuff and not pushing on solution or reviewing in time. But we definitely need to get back to this issue at some point and drag the solution to the finish line already. I have no ETA on this though. If someone wants to take these issues and work with authors of previous patches to create a good solution, feel free to do that. > > Last example: > https://github.com/ovsrobot/ovs/runs/3632261562?check_suite_focus=true > > It happens when OVS is terminated: > > ==478==ERROR: AddressSanitizer: heap-use-after-free on address > 0x614000000e50 at pc 0x000000535c47 bp 0x7fbd37bfc990 sp > 0x7fbd37bfc988 > READ of size 8 at 0x614000000e50 thread T4 (urcu2) > #0 0x535c46 in free_meter_id > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../ofproto/ofproto-dpif.c:6747:37 > #1 0x72e3e0 in ovsrcu_call_postponed > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../lib/ovs-rcu.c:346:13 > #2 0x72e831 in ovsrcu_postpone_thread > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../lib/ovs-rcu.c:362:14 > #3 0x732f3c in ovsthread_wrapper > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../lib/ovs-thread.c:383:12 > #4 0x7fbd3e7d36da in start_thread > (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da) > #5 0x7fbd3dd5271e in clone (/lib/x86_64-linux-gnu/libc.so.6+0x12171e) > > 0x614000000e50 is located 16 bytes inside of 400-byte region > [0x614000000e40,0x614000000fd0) > freed by thread T0 here: > #0 0x4963cd in free > (/home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/vswitchd/ovs-vswitchd+0x4963cd) > #1 0x517998 in destruct > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../ofproto/ofproto-dpif.c:1822:5 > #2 0x4f08a0 in ofproto_destroy > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../ofproto/ofproto.c:1718:5 > #3 0x4c71a4 in bridge_destroy > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../vswitchd/bridge.c:3606:9 > #4 0x4c6f0a in bridge_exit > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../vswitchd/bridge.c:553:9 > #5 0x4e105a in main > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../vswitchd/ovs-vswitchd.c:143:5 > #6 0x7fbd3dc52bf6 in __libc_start_main > (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6) > > previously allocated by thread T0 here: > #0 0x49664d in malloc > (/home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/vswitchd/ovs-vswitchd+0x49664d) > #1 0x7a5c14 in xmalloc__ > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../lib/util.c:137:15 > #2 0x528e7d in open_dpif_backer > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../ofproto/ofproto-dpif.c:774:14 > #3 0x516f7b in construct > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../ofproto/ofproto-dpif.c:1634:13 > #4 0x4ec6d0 in ofproto_create > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../ofproto/ofproto.c:549:13 > #5 0x4c7f67 in bridge_reconfigure > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../vswitchd/bridge.c:882:21 > #6 0x4c7495 in bridge_run > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../vswitchd/bridge.c:3331:9 > #7 0x4e0f71 in main > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../vswitchd/ovs-vswitchd.c:127:9 > #8 0x7fbd3dc52bf6 in __libc_start_main > (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6) > > Thread T4 (urcu2) created by T2 (ct_clean1) here: > #0 0x480dda in pthread_create > (/home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/vswitchd/ovs-vswitchd+0x480dda) > #1 0x732b57 in ovs_thread_create > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../lib/ovs-thread.c:447:13 > #2 0x72dd7a in ovsrcu_quiesced > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../lib/ovs-rcu.c:121:13 > #3 0x793b73 in time_poll > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../lib/timeval.c:321:17 > #4 0x75e577 in poll_block > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../lib/poll-loop.c:364:14 > #5 0x89861c in clean_thread_main > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../lib/conntrack.c:1616:9 > #6 0x732f3c in ovsthread_wrapper > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../lib/ovs-thread.c:383:12 > #7 0x7fbd3e7d36da in start_thread > (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da) > > Thread T2 (ct_clean1) created by T0 here: > #0 0x480dda in pthread_create > (/home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/vswitchd/ovs-vswitchd+0x480dda) > #1 0x732b57 in ovs_thread_create > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../lib/ovs-thread.c:447:13 > #2 0x89842d in conntrack_init > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../lib/conntrack.c:323:24 > #3 0x5d385a in create_dp_netdev > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../lib/dpif-netdev.c:1748:21 > #4 0x5cc22e in dpif_netdev_open > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../lib/dpif-netdev.c:1806:26 > #5 0x5f7721 in do_open > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../lib/dpif.c:347:13 > #6 0x5f7ad8 in dpif_create_and_open > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../lib/dpif.c:415:13 > #7 0x528e97 in open_dpif_backer > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../ofproto/ofproto-dpif.c:776:13 > #8 0x516f7b in construct > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../ofproto/ofproto-dpif.c:1634:13 > #9 0x4ec6d0 in ofproto_create > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../ofproto/ofproto.c:549:13 > #10 0x4c7f67 in bridge_reconfigure > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../vswitchd/bridge.c:882:21 > #11 0x4c7495 in bridge_run > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../vswitchd/bridge.c:3331:9 > #12 0x4e0f71 in main > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../vswitchd/ovs-vswitchd.c:127:9 > #13 0x7fbd3dc52bf6 in __libc_start_main > (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6) > > SUMMARY: AddressSanitizer: heap-use-after-free > /home/runner/work/ovs/ovs/openvswitch-2.16.90/_build/sub/../../ofproto/ofproto-dpif.c:6747:37 > in free_meter_id > Shadow bytes around the buggy address: > 0x0c287fff8170: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa > 0x0c287fff8180: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd > 0x0c287fff8190: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c287fff81a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c287fff81b0: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa > =>0x0c287fff81c0: fa fa fa fa fa fa fa fa fd fd[fd]fd fd fd fd fd > 0x0c287fff81d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c287fff81e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c287fff81f0: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa > 0x0c287fff8200: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00 > 0x0c287fff8210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > Shadow byte legend (one shadow byte represents 8 application bytes): > Addressable: 00 > Partially addressable: 01 02 03 04 05 06 07 > Heap left redzone: fa > Freed heap region: fd > Stack left redzone: f1 > Stack mid redzone: f2 > Stack right redzone: f3 > Stack after return: f5 > Stack use after scope: f8 > Global redzone: f9 > Global init order: f6 > Poisoned by user: f7 > Container overflow: fc > Array cookie: ac > Intra object redzone: bb > ASan internal: fe > Left alloca redzone: ca > Right alloca redzone: cb > Shadow gap: cc > ==478==ABORTING > > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
