[vpp-dev] Unable to run VPP with ASAN enabled
Hi experts,
Here is an ASAN error when run vpp with ASAN enabled.
>
> [root@localhost vpp]# make
> VPP_EXTRA_CMAKE_ARGS=-DVPP_ENABLE_SANITIZE_ADDR=ON build& debug
> make[1]: Entering directory `/home/dev/code/vpp/build-root'
> Arch for platform 'vpp' is native
> Finding source for external
> Makefile fragment found in
> /home/dev/code/vpp/build-data/packages/external.mk
> Source found in /home/dev/code/vpp/build
> Arch for platform 'vpp' is native
> Finding source for vpp
> Makefile fragment found in
> /home/dev/code/vpp/build-data/packages/vpp.mk
> Source found in /home/dev/code/vpp/src
> Configuring external: nothing to do
> Building external: nothing to do
> Installing external: nothing to do
> Configuring vpp: nothing to do
> Building vpp: nothing to do
> Installing vpp: nothing to do
> make[1]: Leaving directory `/home/dev/code/vpp/build-root'
> WARNING: STARTUP_CONF not defined or file doesn't exist.
> Running with minimal startup config: unix { interactive cli-listen
> /run/vpp/cli.sock gid 0 } dpdk { no-pci } \n
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-119.el7
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
>
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
>
Re: [vpp-dev]: Unable to run VPP with ASAN enabled
Hi Rajith, > The problem seems to be due to external libraries that we have linked with > VPP. These external libraries have not been compiled with ASAN. > I could see that when those external libraries were suppressed through the > MyASAN.supp file, VPP started running with ASAN enabled. This is surprising because the default policy of ASan is to allow access to any memory by default - IOW ASan will not detect access memory errors *unless* this memory address was *explicitly* marked as unaccessible. ASan interpose malloc() etc. symbols so libc allocations are automatically marked as accessible/unaccessible by the ASan runtime, and VPP does the same for its own memory allocator. Of course there might still be false positive but in my experience they are usually rare. Best ben -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19486): https://lists.fd.io/g/vpp-dev/message/19486 Mute This Topic: https://lists.fd.io/mt/83071228/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev]: Unable to run VPP with ASAN enabled
Hi Ben, The problem seems to be due to external libraries that we have linked with VPP. These external libraries have not been compiled with ASAN. I could see that when those external libraries were suppressed through the MyASAN.supp file, VPP started running with ASAN enabled. Thanks, Rajith On Wed, May 26, 2021 at 2:25 PM Benoit Ganne (bganne) wrote: > Hi Rajith, > > > I was able to proceed further after setting LD_PRELOAD to the asan > > library. After this i get SIGSEGV crash in asan. These dont seem to be > > related to our code, as without ASAN they have been perfectly working. > > I suspect the opposite - ASan detects errors we do not detect in > release or debug mode, esp. out-of-bound access and use-after-free. Look > carefully at /home/supervisor/libvpp/src/vpp/rtbrick/rtb_vpp_ifp.c:287 > > Best > ben > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19484): https://lists.fd.io/g/vpp-dev/message/19484 Mute This Topic: https://lists.fd.io/mt/83071228/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev]: Unable to run VPP with ASAN enabled
Hi Rajith, > I was able to proceed further after setting LD_PRELOAD to the asan > library. After this i get SIGSEGV crash in asan. These dont seem to be > related to our code, as without ASAN they have been perfectly working. I suspect the opposite - ASan detects errors we do not detect in release or debug mode, esp. out-of-bound access and use-after-free. Look carefully at /home/supervisor/libvpp/src/vpp/rtbrick/rtb_vpp_ifp.c:287 Best ben -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19471): https://lists.fd.io/g/vpp-dev/message/19471 Mute This Topic: https://lists.fd.io/mt/83071228/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev]: Unable to run VPP with ASAN enabled
in qb_loop_run_level (level=level@entry=0x611000b0) at /development/rtbrick-infrastructure/code/qb/lib/loop.c:43 #33 0x766bf54c in qb_loop_run (lp=) at /development/rtbrick-infrastructure/code/qb/lib/loop.c:210 #34 0x766cb19a in lib_qb_service_start_event_loop () at /development/rtbrick-infrastructure/code/qb/lib/wrapper/lib_qb_service.c:257 #35 0x502d in main (argc=3, argv=) at /development/rtbrick-infrastructure/code/bd/src/bd/bd_main.c:136 Thanks, Rajith On Tue, May 25, 2021 at 3:10 PM Benoit Ganne (bganne) wrote: > How are you starting VPP? If this is through 'make test' then chances are > the culprit is the interaction of asan, clang and python [1]. > The easy way to fix is to rebuild with gcc instead of clang, eg. > ~# make rebuild VPP_EXTRA_CMAKE_ARGS=-DVPP_ENABLE_SANITIZE_ADDR=ON CC=gcc-9 > > Best > ben > > [1] > https://gerrit.fd.io/r/c/vpp/+/27268/3/src/vpp-api/python/vpp_papi/vpp_ffi.py.in#1 > > > -Original Message- > > From: vpp-dev@lists.fd.io On Behalf Of Rajith PR > via > > lists.fd.io > > Sent: mardi 25 mai 2021 09:51 > > To: vpp-dev > > Subject: [vpp-dev]: Unable to run VPP with ASAN enabled > > > > Hi All, > > > > I am not able to run VPP with ASAN. Though we have been using VPP for > > sometime this is the first time we enabled ASAN in the build. > > I have followed the steps as mentioned in the sanitizer doc, can someone > > please let me know what is missed here. > > > > Run Time Error(Missing symbol): > > > > /usr/local/lib/librtbvpp.so:/home/supervisor/libvpp/build-root/install- > > vpp_debug-native/vpp/lib/libvppinfra.so.1.0.1: undefined symbol: > > __asan_option_detect_stack_use_after_return > > > > > > VPP Version : 20.09 > > > > > > Build Command : make rebuild VPP_EXTRA_CMAKE_ARGS=- > > DVPP_ENABLE_SANITIZE_ADDR=ON > > > > > > Build Summary : > > > > VPP version : 1.0.1-3032~g4b28254fc-dirty > > VPP library version : 1.0.1 > > GIT toplevel dir: /home/supervisor/libvpp > > Build type : debug > > C flags : -fsanitize=address -DCLIB_SANITIZE_ADDR -Wno- > > address-of-packed-member -g -fPIC -Werror -Wall -march=corei7 - > > mtune=corei7-avx -O0 -DCLIB_DEBUG -fstack-protector -DFORTIFY_SOURCE=2 - > > fno-common > > Linker flags (apps) : -fsanitize=address > > Linker flags (libs) : -fsanitize=address > > Host processor : x86_64 > > Target processor: x86_64 > > Prefix path : > > /opt/vpp/external/x86_64;/home/supervisor/libvpp/build-root/install- > > vpp_debug-native/external > > Install prefix : /home/supervisor/libvpp/build-root/install- > > vpp_debug-native/vpp > > > > > > > > Reference : https://fd.io/docs/vpp/master/troubleshooting/sanitizer.html > > > > > > Thanks, > > Rajith > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19462): https://lists.fd.io/g/vpp-dev/message/19462 Mute This Topic: https://lists.fd.io/mt/83071228/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev]: Unable to run VPP with ASAN enabled
How are you starting VPP? If this is through 'make test' then chances are the culprit is the interaction of asan, clang and python [1]. The easy way to fix is to rebuild with gcc instead of clang, eg. ~# make rebuild VPP_EXTRA_CMAKE_ARGS=-DVPP_ENABLE_SANITIZE_ADDR=ON CC=gcc-9 Best ben [1] https://gerrit.fd.io/r/c/vpp/+/27268/3/src/vpp-api/python/vpp_papi/vpp_ffi.py.in#1 > -Original Message- > From: vpp-dev@lists.fd.io On Behalf Of Rajith PR via > lists.fd.io > Sent: mardi 25 mai 2021 09:51 > To: vpp-dev > Subject: [vpp-dev]: Unable to run VPP with ASAN enabled > > Hi All, > > I am not able to run VPP with ASAN. Though we have been using VPP for > sometime this is the first time we enabled ASAN in the build. > I have followed the steps as mentioned in the sanitizer doc, can someone > please let me know what is missed here. > > Run Time Error(Missing symbol): > > /usr/local/lib/librtbvpp.so:/home/supervisor/libvpp/build-root/install- > vpp_debug-native/vpp/lib/libvppinfra.so.1.0.1: undefined symbol: > __asan_option_detect_stack_use_after_return > > > VPP Version : 20.09 > > > Build Command : make rebuild VPP_EXTRA_CMAKE_ARGS=- > DVPP_ENABLE_SANITIZE_ADDR=ON > > > Build Summary : > > VPP version : 1.0.1-3032~g4b28254fc-dirty > VPP library version : 1.0.1 > GIT toplevel dir: /home/supervisor/libvpp > Build type : debug > C flags : -fsanitize=address -DCLIB_SANITIZE_ADDR -Wno- > address-of-packed-member -g -fPIC -Werror -Wall -march=corei7 - > mtune=corei7-avx -O0 -DCLIB_DEBUG -fstack-protector -DFORTIFY_SOURCE=2 - > fno-common > Linker flags (apps) : -fsanitize=address > Linker flags (libs) : -fsanitize=address > Host processor : x86_64 > Target processor: x86_64 > Prefix path : > /opt/vpp/external/x86_64;/home/supervisor/libvpp/build-root/install- > vpp_debug-native/external > Install prefix : /home/supervisor/libvpp/build-root/install- > vpp_debug-native/vpp > > > > Reference : https://fd.io/docs/vpp/master/troubleshooting/sanitizer.html > > > Thanks, > Rajith -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19452): https://lists.fd.io/g/vpp-dev/message/19452 Mute This Topic: https://lists.fd.io/mt/83071228/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev]: Unable to run VPP with ASAN enabled
Hi All, I am not able to run VPP with ASAN. Though we have been using VPP for sometime this is the first time we enabled ASAN in the build. I have followed the steps as mentioned in the sanitizer doc, can someone please let me know what is missed here. *Run Time Error(Missing symbol):* /usr/local/lib/librtbvpp.so:/home/supervisor/libvpp/build-root/install-vpp_debug-native/vpp/lib/libvppinfra.so.1.0.1: undefined symbol: __asan_option_detect_stack_use_after_return *VPP Version* : 20.09 *Build Command* : make rebuild VPP_EXTRA_CMAKE_ARGS= -DVPP_ENABLE_SANITIZE_ADDR=ON *Build Summary *: VPP version : 1.0.1-3032~g4b28254fc-dirty VPP library version : 1.0.1 GIT toplevel dir: /home/supervisor/libvpp Build type : debug C flags : -fsanitize=address -DCLIB_SANITIZE_ADDR -Wno-address-of-packed-member -g -fPIC -Werror -Wall -march=corei7 -mtune=corei7-avx -O0 -DCLIB_DEBUG -fstack-protector -DFORTIFY_SOURCE=2 -fno-common Linker flags (apps) : -fsanitize=address Linker flags (libs) : -fsanitize=address Host processor : x86_64 Target processor: x86_64 Prefix path : /opt/vpp/external/x86_64;/home/supervisor/libvpp/build-root/install-vpp_debug-native/external Install prefix : /home/supervisor/libvpp/build-root/install-vpp_debug-native/vpp *Reference *: https://fd.io/docs/vpp/master/troubleshooting/sanitizer.html Thanks, Rajith -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19450): https://lists.fd.io/g/vpp-dev/message/19450 Mute This Topic: https://lists.fd.io/mt/83071228/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
