Re: [SUSPECTED SPAM] [vpp-dev] VPP crashes on LX2160A platform
Hello, The current build (22.10, cross-compiled via SolidRun toolchain) crashes without dependency to optimization level, and, with debug enabled. shows the following: Thread 1 "vpp_main" received signal SIGABRT, Aborted. __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0xf6d7caac in __GI_abort () at abort.c:79 #2 0x00406fe4 in os_panic () at /work/build/vpp/src/vpp/vnet/main.c:416 #3 0xf6fa6514 in debugger () at /work/build/vpp/src/vppinfra/error.c:84 #4 0xf6fa6874 in _clib_error (how_to_die=2, function_name=0xf7173978 <__FUNCTION__.32141> "vlib_buffer_validate_alloc_free", line_number=333, fmt=0xf7173438 "%s %U buffer 0x%x") at /work/build/vpp/src/vppinfra/error.c:143 #5 0xf70c1218 in vlib_buffer_validate_alloc_free (vm=0xb6d5c740, buffers=0xb4bac810, n_buffers=1, expected_state=VLIB_BUFFER_KNOWN_ALLOCATED) at /work/build/vpp/src/vlib/buffer.c:332 #6 0xf716afc4 in vlib_buffer_pool_put (vm=0xb6d5c740, buffer_pool_index=0 '\000', buffers=0xb4bac810, n_buffers=1) at /work/build/vpp/src/vlib/buffer_funcs.h:731 #7 0xf716b75c in vlib_buffer_free_inline (vm=0xb6d5c740, buffers=0xb88bd1d4, n_buffers=0, maybe_next=1) at /work/build/vpp/src/vlib/buffer_funcs.h:917 #8 0xf716b7c8 in vlib_buffer_free (vm=0xb6d5c740, buffers=0xb88bd1d0, n_buffers=1) at /work/build/vpp/src/vlib/buffer_funcs.h:936 #9 0xf716c424 in process_drop_punt (vm=0xb6d5c740, node=0xb7844300, frame=0xb88bd1c0, disposition=ERROR_DISPOSITION_DROP) at /work/build/vpp/src/vlib/drop.c:235 #10 0xf716c4fc in error_drop_node_fn_cortexa72 (vm=0xb6d5c740, node=0xb7844300, frame=0xb88bd1c0) at /work/build/vpp/src/vlib/drop.c:251 #11 0xf70f512c in dispatch_node (vm=0xb6d5c740, node=0xb7844300, type=VLIB_NODE_TYPE_INTERNAL, dispatch_state=VLIB_NODE_STATE_POLLING, frame=0xb88bd1c0, last_time_stamp=233164692224) at /work/build/vpp/src/vlib/main.c:960 #12 0xf70f585c in dispatch_pending_node (vm=0xb6d5c740, pending_frame_index=4, last_time_stamp=233164692224) at /work/build/vpp/src/vlib/main.c:1119 #13 0xf70f6be8 in vlib_main_or_worker_loop (vm=0xb6d5c740, is_main=1) at /work/build/vpp/src/vlib/main.c:1588 #14 0xf70f71ec in vlib_main_loop (vm=0xb6d5c740) at /work/build/vpp/src/vlib/main.c:1716 #15 0xf70f7d1c in vlib_main (vm=0xb6d5c740, input=0xb4badfc8) at /work/build/vpp/src/vlib/main.c:2010 #16 0xf7145044 in thread0 (arg=281473749206848) at /work/build/vpp/src/vlib/unix/main.c:667 #17 0xf6fb84c0 in clib_calljmp () at /work/build/vpp/src/vppinfra/longjmp.S:809 Backtrace stopped: previous frame identical to this frame (corrupt stack?) -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22367): https://lists.fd.io/g/vpp-dev/message/22367 Mute This Topic: https://lists.fd.io/mt/95380982/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [SUSPECTED SPAM] [vpp-dev] VPP crashes on LX2160A platform
Hi, I think the 1st thing to try would be whether you can reproduce it with a debug build. The backtrace you have is not usable unfortunately. The easiest should be to build debug binaries: ~# make rebuild ~# ./build-root/install-vpp_debug-native/vpp/bin/vpp -c /etc/vpp/startup.conf Best Ben > -Original Message- > From: vpp-dev@lists.fd.io On Behalf Of > agv...@gmail.com > Sent: Thursday, December 1, 2022 12:59 > To: vpp-dev@lists.fd.io > Subject: [SUSPECTED SPAM] [vpp-dev] VPP crashes on LX2160A platform > > Dear VPP community, > > > I'm trying to operate VPP on SolidRun LX2160 board, which is based on 16 > cores A72 NXP SoC, unfortunately, with little success. Does anybody have > any experience with running VPP on such boards? > > > The performance in my tests is quite good (more then 4mpps NDR) , but VPP > works very unstable and segfaults in time interval from seconds to hours > after start. > The events causing segfaults were not identified. It may happen (and > usually) when you walk through CLI. It may happen (less frequently) when > just forwarding packets without a touch to vppctl. Applying config longer > then few lines usually cause that. Second VPPCTL connection usually couses > that, > > I was trying the following versions of VPP with literally same results: > > - VPP 21.01 from LSDK distribution, built on the board natively > - VPP 22.10, from Master branch, crossbuilt using > https://docs.nxp.com/bundle/GUID-87AD3497-0BD4-4492-8040- > 3F3BE0F2B087/page/GUID-8A75A4AD-2EB9-4A5A-A784-465B98E67951.html > - VPP 22.08, built using flexbuild tool (from same link above). > > I was trying different settings of main_heap memory pool (size, pagesize), > different hugepages settings (standard 4k, huge 2M, huge 1G), but there > were no serious improvement. It looks like 22.08 most stable and may last > for few hours. > > As performance looks promising, I'm really looking forward to make it work > stable. Can somebody please advice , where do I need to look at to fix > the problem? There are , according to CSIT, good results on other ARM v8 > platforms. > As for OS, I'm using pre-built Ubuntu Core-based distribution from > SolidRun. > > See below OS information, logs with crash. See in attachement: Platform > dmesg and GDB trace of 22.10 crash. > Below are system logs of VPP crashes. > > abramov@nc2s5:~$ cat /etc/lsb-release > DISTRIB_ID=Ubuntu > DISTRIB_RELEASE=20.04 > DISTRIB_CODENAME=focal > DISTRIB_DESCRIPTION="Ubuntu 20.04.5 LTS" > abramov@nc2s5:~$ uname -a > Linux nc2s5 5.10.35-00018-gbb124648d42c #1 SMP PREEMPT Wed May 11 17:07:05 > UTC 2022 aarch64 aarch64 aarch64 GNU/Linux > abramov@nc2s5:~$ > > > Dec 01 10:35:42 nc2s5 vnet[2259]: received signal SIGSEGV, PC unsupported, > faulting address 0x2d3ba50a885 > Dec 01 10:35:42 nc2s5 vnet[2259]: #0 0xa7df2e2c 0xa7df2e2c > Dec 01 10:35:42 nc2s5 vnet[2259]: #1 0xa95ad588 0xa95ad588 > Dec 01 10:35:42 nc2s5 vnet[2259]: #2 0xa7da0090 > vlib_node_runtime_sync_stats + 0x0 > Dec 01 10:35:42 nc2s5 vnet[2259]: #3 0xa7da191c > vlib_node_sync_stats + 0x4c > Dec 01 10:35:42 nc2s5 vnet[2259]: #4 0xa7dd973c > vlib_worker_thread_barrier_release + 0x45c > Dec 01 10:35:42 nc2s5 vnet[2259]: #5 0xa7de6ef4 0xa7de6ef4 > Dec 01 10:35:42 nc2s5 vnet[2259]: #6 0xa7de827c 0xa7de827c > Dec 01 10:35:42 nc2s5 vnet[2259]: #7 0xa7df00dc 0xa7df00dc > Dec 01 10:35:42 nc2s5 vnet[2259]: #8 0xa7da5e04 vlib_main + 0x8f4 > Dec 01 10:35:42 nc2s5 vnet[2259]: #9 0xa7df1d8c 0xa7df1d8c > Dec 01 10:35:42 nc2s5 vnet[2259]: #10 0xa7c36f8c clib_calljmp + > 0x24 > > Dec 01 10:26:56 nc2s5 vnet[2232]: received signal SIGSEGV, PC unsupported, > faulting address 0x208 > Dec 01 10:26:56 nc2s5 vnet[2232]: #0 0xa4bebe2c 0xa4bebe2c > Dec 01 10:26:56 nc2s5 vnet[2232]: #1 0xa63a6588 0xa63a6588 > Dec 01 10:26:56 nc2s5 vnet[2232]: #2 0xa6340aa8 0xa6340aa8 > Dec 01 10:26:56 nc2s5 vnet[2232]: #3 0xa4b9f150 vlib_main + 0xc40 > Dec 01 10:26:56 nc2s5 vnet[2232]: #4 0xa4bead8c 0xa4bead8c > Dec 01 10:26:56 nc2s5 vnet[2232]: #5 0xa4a2ff8c clib_calljmp + > 0x24 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22264): https://lists.fd.io/g/vpp-dev/message/22264 Mute This Topic: https://lists.fd.io/mt/95380982/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
