Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session
Thank you for you reply. I did use a VMware virtual machine before, I switched to a physical NIC for testing and now there is no problem setting rx-num to 5. Also, I tested redis using redis-benchmark, the specific command is as follows: sudo taskset -c 9 bash -c "VCL_CONFIG=/home/chenweihao/vpp_config/vcl.conf LD_PRELOAD=/home/chenweihao/vpp_dev/build-root/install-vpp-native/vpp/lib/libvcl_ldpreload.so /home/chenweihao/redis_6.0/src/redis-server --protected-mode no" sudo taskset -c 11 bash -c "VCL_CONFIG=/home/chenweihao/vpp_config/vcl.conf LD_PRELOAD=/home/chenweihao/vpp_dev/build-root/install-vpp-native/vpp/lib/libvcl_ldpreload.so /home/chenweihao/redis_6.0/src/redis-benchmark -h 127.0.0.1 -n 1 -q -t set,get -d 4096" In the case of running both commands, the problems I mentioned before still occur (even though I applied yesterday's patch). However, the problem is not triggered every time the test is run, and it may take several runs before the problem occurs once. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22745): https://lists.fd.io/g/vpp-dev/message/22745 Mute This Topic: https://lists.fd.io/mt/97707720/21656 Mute #vnet:https://lists.fd.io/g/vpp-dev/mutehashtag/vnet Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session
Thank you for your reply. I tried to change num-tx-queues from 2 to 5, but it got a SIGSEGV, the backtrace is: #0 0x7fffb453ff89 in rte_write32_relaxed (addr=0x80007ef0, value=0) at ../src-dpdk/lib/eal/include/generic/rte_io.h:310 #1 rte_write32 (addr=0x80007ef0, value=0) at ../src-dpdk/lib/eal/include/generic/rte_io.h:373 #2 vmxnet3_enable_intr (hw=0xac03b0600, intr_idx=4294967262) at ../src-dpdk/drivers/net/vmxnet3/vmxnet3_ethdev.c:210 #3 0x7fffb4544d35 in vmxnet3_dev_rx_queue_intr_enable ( dev=0x7fffb5186980 , queue_id=0) at ../src-dpdk/drivers/net/vmxnet3/vmxnet3_ethdev.c:1815 #4 0x7fffaff4bbf2 in rte_eth_dev_rx_intr_enable (port_id=0, queue_id=0) at ../src-dpdk/lib/ethdev/rte_ethdev.c:4740 #5 0x7fffb49f4564 in dpdk_setup_interrupts (xd=0x7fffbdbb2940) at /home/chenweihao/vpp_dev/src/plugins/dpdk/device/common.c:336 #6 0x7fffb49f4430 in dpdk_device_start (xd=0x7fffbdbb2940) at /home/chenweihao/vpp_dev/src/plugins/dpdk/device/common.c:411 #7 0x7fffb49ff713 in dpdk_interface_admin_up_down ( vnm=0x77e2b828 , hw_if_index=1, flags=1) at /home/chenweihao/vpp_dev/src/plugins/dpdk/device/device.c:476 #8 0x770d60e8 in vnet_sw_interface_set_flags_helper ( vnm=0x77e2b828 , sw_if_index=1, flags=VNET_SW_INTERFACE_FLAG_ADMIN_UP, helper_flags=0) at /home/chenweihao/vpp_dev/src/vnet/interface.c:470 #9 0x770d645a in vnet_sw_interface_set_flags ( vnm=0x77e2b828 , sw_if_index=1, flags=VNET_SW_INTERFACE_FLAG_ADMIN_UP) at /home/chenweihao/vpp_dev/src/vnet/interface.c:524 #10 0x7710515f in set_state (vm=0x7fffb6a00740, input=0x7fffa9f84bb8, cmd=0x7fffb7180850) at /home/chenweihao/vpp_dev/src/vnet/interface_cli.c:946 #11 0x77e72257 in vlib_cli_dispatch_sub_commands (vm=0x7fffb6a00740, cm=0x77f6a770 , input=0x7fffa9f84bb8, parent_command_index=20) at /home/chenweihao/vpp_dev/src/vlib/cli.c:650 #12 0x77e71fea in vlib_cli_dispatch_sub_commands (vm=0x7fffb6a00740, cm=0x77f6a770 , input=0x7fffa9f84bb8, parent_command_index=7) at /home/chenweihao/vpp_dev/src/vlib/cli.c:607 #13 0x77e71fea in vlib_cli_dispatch_sub_commands (vm=0x7fffb6a00740, cm=0x77f6a770 , input=0x7fffa9f84bb8, parent_command_index=0) at /home/chenweihao/vpp_dev/src/vlib/cli.c:607 #14 0x77e7122a in vlib_cli_input (vm=0x7fffb6a00740, input=0x7fffa9f84bb8, function=0x0, function_arg=0) at /home/chenweihao/vpp_dev/src/vlib/cli.c:753 #15 0x77ef7e23 in unix_cli_exec (vm=0x7fffb6a00740, input=0x7fffa9f84f30, cmd=0x7fffb71815b8) at /home/chenweihao/vpp_dev/src/vlib/unix/cli.c:3431 #16 0x77e72257 in vlib_cli_dispatch_sub_commands (vm=0x7fffb6a00740, cm=0x77f6a770 , input=0x7fffa9f84f30, --Type for more, q to quit, c to continue without paging-- parent_command_index=0) at /home/chenweihao/vpp_dev/src/vlib/cli.c:650 #17 0x77e7122a in vlib_cli_input (vm=0x7fffb6a00740, input=0x7fffa9f84f30, function=0x0, function_arg=0) at /home/chenweihao/vpp_dev/src/vlib/cli.c:753 #18 0x77efdfc5 in startup_config_process (vm=0x7fffb6a00740, rt=0x7fffb9194080, f=0x0) at /home/chenweihao/vpp_dev/src/vlib/unix/main.c:291 #19 0x77ea2c5d in vlib_process_bootstrap (_a=140736084405176) at /home/chenweihao/vpp_dev/src/vlib/main.c:1221 #20 0x76f1ffd8 in clib_calljmp () at /home/chenweihao/vpp_dev/src/vppinfra/longjmp.S:123 #21 0x7fffac516bb0 in ?? () #22 0x77ea26f9 in vlib_process_startup (vm=0x8, p=0x77ea53bb , f=0x7fffac516cc0) at /home/chenweihao/vpp_dev/src/vlib/main.c:1246 #23 0x76f7aa1c in vec_mem_size (v=0x7fffb6a00740) at /home/chenweihao/vpp_dev/src/vppinfra/vec.c:15 #24 0x0581655dfd1c in ?? () #25 0x00330004 in ?? () #26 0x0030 in ?? () #27 0x7fffbdbc7240 in ?? () #28 0x7fffbdbc7240 in ?? () #29 0x7fffb80e5498 in ?? () #30 0x0001 in ?? () #31 0x in ?? () I tried to change num-rx-queues and num-tx-queues to 4, then SIGSEGV not happened. I applied the patch https://gerrit.fd.io/r/c/vpp/+/38529 , and the problem of redis 6.0 seemed still exist, the stack backtrace is same with https://lists.fd.io/g/vpp-dev/message/22731 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22740): https://lists.fd.io/g/vpp-dev/message/22740 Mute This Topic: https://lists.fd.io/mt/97707720/21656 Mute #vnet:https://lists.fd.io/g/vpp-dev/mutehashtag/vnet Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session
Thank you for your reply. I did use a VMware virtual machine before, I switched to a physical NIC for testing and now there is no problem setting rx-num to 5. Also, I tested redis using redis-benchmark, the specific command is as follows: sudo taskset -c 9 bash -c "VCL_CONFIG=/home/chenweihao/vpp_config/vcl.conf LD_PRELOAD=/home/chenweihao/vpp_dev/build-root/install-vpp-native/vpp/lib/libvcl_ldpreload.so /home/chenweihao/redis_6.0/src/redis-server --protected-mode no" sudo taskset -c 11 bash -c "VCL_CONFIG=/home/chenweihao/vpp_config/vcl.conf LD_PRELOAD=/home/chenweihao/vpp_dev/build-root/install-vpp-native/vpp/lib/libvcl_ldpreload.so /home/chenweihao/redis_6.0/src/redis-benchmark -h 127.0.0.1 -n 1 -q -t set,get -d 4096" In the case of running both commands, the problems I mentioned before still occur (even though I applied the patch https://gerrit.fd.io/r/c/vpp/+/38529 ). However, the problem is not triggered every time the test is run, and it may take several runs before the problem occurs once. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22752): https://lists.fd.io/g/vpp-dev/message/22752 Mute This Topic: https://lists.fd.io/mt/97707720/21656 Mute #vnet:https://lists.fd.io/g/vpp-dev/mutehashtag/vnet Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session
Thank you for your reply. This is the stacktrace captured by gdb: 2: /home/chenweihao/vpp_dev/src/vnet/session/session.c:233 (session_is_valid) assertion `! pool_is_free (session_main.wrk[thread_index].sessions, _e)' fails Thread 4 "vpp_wk_1" received signal SIGABRT, Aborted. [Switching to Thread 0x7fffa93f5640 (LWP 4575)] __pthread_kill_implementation (no_tid=0, signo=6, threadid=140736032888384) at ./nptl/pthread_kill.c:44 44./nptl/pthread_kill.c: No such file or directory. (gdb) bt #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140736032888384) at ./nptl/pthread_kill.c:44 #1 __pthread_kill_internal (signo=6, threadid=140736032888384) at ./nptl/pthread_kill.c:78 #2 __GI___pthread_kill (threadid=140736032888384, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 #3 0x76a42476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #4 0x76a287f3 in __GI_abort () at ./stdlib/abort.c:79 #5 0xb073 in os_panic () at /home/chenweihao/vpp_dev/src/vpp/vnet/main.c:417 #6 0x76f0a699 in debugger () at /home/chenweihao/vpp_dev/src/vppinfra/error.c:84 #7 0x76f0a450 in _clib_error (how_to_die=2, function_name=0x0, line_number=0, fmt=0x77b88208 "%s:%d (%s) assertion `%s' fails") at /home/chenweihao/vpp_dev/src/vppinfra/error.c:143 #8 0x775f31c7 in session_is_valid (si=4294967295, thread_index=1 '\001') at /home/chenweihao/vpp_dev/src/vnet/session/session.c:233 #9 0x775f177c in session_get (si=4294967295, thread_index=1) at /home/chenweihao/vpp_dev/src/vnet/session/session.h:373 #10 0x775f3770 in ho_session_get (ho_index=4294967295) at /home/chenweihao/vpp_dev/src/vnet/session/session.h:689 #11 0x775f37d8 in session_half_open_migrate_notify (tc=0x7fffbdba00c0) at /home/chenweihao/vpp_dev/src/vnet/session/session.c:357 #12 0x77648600 in ct_accept_one (thread_index=2, ho_index=2) at /home/chenweihao/vpp_dev/src/vnet/session/application_local.c:669 #13 0x77648243 in ct_accept_rpc_wrk_handler (rpc_args=0x2) at /home/chenweihao/vpp_dev/src/vnet/session/application_local.c:760 #14 0x77620b42 in session_event_dispatch_ctrl (wrk=0x7fffbdb6fe00, elt=0x7fffbdccaf2c) at /home/chenweihao/vpp_dev/src/vnet/session/session_node.c:1656 #15 0x776175c2 in session_queue_node_fn (vm=0x7fffb82da700, node=0x7fffbdbc21c0, frame=0x0) at /home/chenweihao/vpp_dev/src/vnet/session/session_node.c:1962 #16 0x77ea3a62 in dispatch_node (vm=0x7fffb82da700, node=0x7fffbdbc21c0, type=VLIB_NODE_TYPE_INPUT, dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x0, last_time_stamp=2911620275652) at /home/chenweihao/vpp_dev/src/vlib/main.c:960 #17 0x77e9f7d1 in vlib_main_or_worker_loop (vm=0x7fffb82da700, is_main=0) at /home/chenweihao/vpp_dev/src/vlib/main.c:1557 #18 0x77e9f1c7 in vlib_worker_loop (vm=0x7fffb82da700) at /home/chenweihao/vpp_dev/src/vlib/main.c:1722 #19 0x77edb020 in vlib_worker_thread_fn (arg=0x7fffb9182980) at /home/chenweihao/vpp_dev/src/vlib/threads.c:1598 #20 0x77ed62b6 in vlib_worker_thread_bootstrap_fn (arg=0x7fffb9182980) at /home/chenweihao/vpp_dev/src/vlib/threads.c:418 #21 0x76a94b43 in start_thread (arg=) at ./nptl/pthread_create.c:442 #22 0x76b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 The resaon why assertion failed is that c_s_index value is -1 (or ~0). The redis veision I used: git clone https://github.com/redis/redis.git cd redis git checkout 6.0 make My startup.conf: unix { nodaemon interactive full-coredump cli-listen /run/vpp/cli.sock gid vpp startup-config /home/chenweihao/startup.txt } api-trace { on } cpu { main-core 0 workers 4 } tcp { cc-algo cubic } dpdk { uio-driver vfio-pci dev :03:00.0 { name eth1 num-rx-queues 2 num-tx-queues 2 } } buffers { buffers-per-numa 131072 } session { evt_qs_memfd_seg } My startup.txt: set interface ip address eth1 192.168.0.2/24 set interface state eth1 up set interface rx-mode eth1 polling create loopback interface set interface ip address loop0 127.0.0.1/8 set interface state loop0 up My vcl.conf: vcl { rx-fifo-size 4000 tx-fifo-size 4000 app-scope-local app-scope-global api-socket-name /run/vpp/api.sock use-mq-eventfd } I tried iperf3 as you said,indeed I did not encounter this problem. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22731): https://lists.fd.io/g/vpp-dev/message/22731 Mute This Topic: https://lists.fd.io/mt/97707720/21656 Mute #vnet:https://lists.fd.io/g/vpp-dev/mutehashtag/vnet Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session
Thanks for your reply. I give a more detailed backtrace and config in https://lists.fd.io/g/vpp-dev/message/22731. ( https://lists.fd.io/g/vpp-dev/message/22731, ) My installation method is to clone vpp from github and make build on Ubuntu 22.04(Kernel version is 5.19),and I use make run for test and make debug for debugging. Yes, I yried to make the server and client are attached to the same vpp instance.I tried the latest version of vpp on github on yesterday, the problem is still exist. I am looking forward to your reply. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22732): https://lists.fd.io/g/vpp-dev/message/22732 Mute This Topic: https://lists.fd.io/mt/97707720/21656 Mute #vnet:https://lists.fd.io/g/vpp-dev/mutehashtag/vnet Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-