Hi , We are seeing a crash while doing add_trace for a vlib_buffer in our graph node.
#0 0x00007ffff4ee0feb in raise () from /lib64/libc.so.6 #1 0x00007ffff4ecb5c1 in abort () from /lib64/libc.so.6 #2 0x000000000040831c in os_panic () at /fdio/src/fdio.1810/src/vpp/vnet/main.c:368 #3 0x00007ffff5f28f2f in debugger () at /fdio/src/fdio.1810/src/vppinfra/error.c:84 #4 0x00007ffff5f2936a in _clib_error (how_to_die=2, function_name=0x0, line_number=0, fmt=0x7fffb2083920 "%s:%d (%s) assertion `%s' fails") at /fdio/src/fdio.1810/src/vppinfra/error.c:143 #5 0x00007fffb2035d79 in vlib_validate_trace (tm=0x7fffbaccfd58, b=0x2aaab447d940) at /fdio/src/fdio.1810/src/vlib/trace_funcs.h:53 #6 0x00007fffb2035ec8 in vlib_add_trace (vm=0x7fffbaccfb40, r=0x7fffbb430f40, b=0x2aaab447d940, n_data_bytes=36) The assert is pointing to following line in the code. ASSERT (!pool_is_free_index (tm->trace_buffer_pool, vlib_buffer_get_trace_index (b))); One specific point with this crash is: This crash is happening only when we have multiple workers in VPP. Following is the scenario this crash is happening. 1. Packet comes to worker-1 from nic card 2. graph node on worker-1 hands off the packet to worker-2 3. In worker-2, while processing the packet, we are trying to add the trace using vlib_add_trace and this crash occurs. The trace buffer within vlib_buffer_t, is it specific to worker ? If so, what happens, when the buffer gets hand off to another worker ? Can this cause the above crash ? -- Thanks & Regards, Murthy
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#16613): https://lists.fd.io/g/vpp-dev/message/16613 Mute This Topic: https://lists.fd.io/mt/74625018/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
