subject:"Re\: \[vpp\-dev\] #vnet A bug which may cause assertion error in vnet\/session"

Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session

2023-03-22 Thread Chen Weihao

Thank you for your reply.
I did use a VMware virtual machine before, I switched to a physical NIC for 
testing and now there is no problem setting rx-num to 5.
Also, I tested redis using redis-benchmark, the specific command is as follows:
sudo taskset -c 9 bash -c "VCL_CONFIG=/home/chenweihao/vpp_config/vcl.conf 
LD_PRELOAD=/home/chenweihao/vpp_dev/build-root/install-vpp-native/vpp/lib/libvcl_ldpreload.so
 /home/chenweihao/redis_6.0/src/redis-server --protected-mode no"
sudo taskset -c 11 bash -c "VCL_CONFIG=/home/chenweihao/vpp_config/vcl.conf 
LD_PRELOAD=/home/chenweihao/vpp_dev/build-root/install-vpp-native/vpp/lib/libvcl_ldpreload.so
 /home/chenweihao/redis_6.0/src/redis-benchmark -h 127.0.0.1 -n 1 -q -t 
set,get -d 4096"

In the case of running both commands, the problems I mentioned before still 
occur (even though I applied the patch https://gerrit.fd.io/r/c/vpp/+/38529 ).
However, the problem is not triggered every time the test is run, and it may 
take several runs before the problem occurs once.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22752): https://lists.fd.io/g/vpp-dev/message/22752
Mute This Topic: https://lists.fd.io/mt/97707720/21656
Mute #vnet:https://lists.fd.io/g/vpp-dev/mutehashtag/vnet
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session

2023-03-22 Thread Chen Weihao

Thank you for you reply.
I did use a VMware virtual machine before, I switched to a physical NIC for 
testing and now there is no problem setting rx-num to 5.
Also, I tested redis using redis-benchmark, the specific command is as follows:
sudo taskset -c 9 bash -c "VCL_CONFIG=/home/chenweihao/vpp_config/vcl.conf 
LD_PRELOAD=/home/chenweihao/vpp_dev/build-root/install-vpp-native/vpp/lib/libvcl_ldpreload.so
 /home/chenweihao/redis_6.0/src/redis-server --protected-mode no"
sudo taskset -c 11 bash -c "VCL_CONFIG=/home/chenweihao/vpp_config/vcl.conf 
LD_PRELOAD=/home/chenweihao/vpp_dev/build-root/install-vpp-native/vpp/lib/libvcl_ldpreload.so
 /home/chenweihao/redis_6.0/src/redis-benchmark -h 127.0.0.1 -n 1 -q -t 
set,get -d 4096"

In the case of running both commands, the problems I mentioned before still 
occur (even though I applied yesterday's patch).
However, the problem is not triggered every time the test is run, and it may 
take several runs before the problem occurs once.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22745): https://lists.fd.io/g/vpp-dev/message/22745
Mute This Topic: https://lists.fd.io/mt/97707720/21656
Mute #vnet:https://lists.fd.io/g/vpp-dev/mutehashtag/vnet
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session

2023-03-21 Thread Florin Coras

Hi, 

The problem seems to be that you’re using a vmxnet3 interface, so I suspect 
this might be a vm configuration issue. Your current config should work but 
could end up being inefficient. 

With respect to your problem, I just built redis and ran redis-server and cli 
over LDP. Everything seems to be working fine so I’m assuming you’re doing some 
stress tests of redis? Could you provide more info about your client? 

Regards,
Florin

> On Mar 21, 2023, at 6:18 AM, Chen Weihao  wrote:
> 
> Thank you for your reply.
> I tried to change num-tx-queues from 2 to 5, but it got a SIGSEGV, the 
> backtrace is:
> #0  0x7fffb453ff89 in rte_write32_relaxed (addr=0x80007ef0, value=0)
> at ../src-dpdk/lib/eal/include/generic/rte_io.h:310
> #1  rte_write32 (addr=0x80007ef0, value=0)
> at ../src-dpdk/lib/eal/include/generic/rte_io.h:373
> #2  vmxnet3_enable_intr (hw=0xac03b0600, intr_idx=4294967262)
> at ../src-dpdk/drivers/net/vmxnet3/vmxnet3_ethdev.c:210
> #3  0x7fffb4544d35 in vmxnet3_dev_rx_queue_intr_enable (
> dev=0x7fffb5186980 , queue_id=0)
> at ../src-dpdk/drivers/net/vmxnet3/vmxnet3_ethdev.c:1815
> #4  0x7fffaff4bbf2 in rte_eth_dev_rx_intr_enable (port_id=0, queue_id=0)
> at ../src-dpdk/lib/ethdev/rte_ethdev.c:4740
> #5  0x7fffb49f4564 in dpdk_setup_interrupts (xd=0x7fffbdbb2940)
> at /home/chenweihao/vpp_dev/src/plugins/dpdk/device/common.c:336
> #6  0x7fffb49f4430 in dpdk_device_start (xd=0x7fffbdbb2940)
> at /home/chenweihao/vpp_dev/src/plugins/dpdk/device/common.c:411
> #7  0x7fffb49ff713 in dpdk_interface_admin_up_down (
> vnm=0x77e2b828 , hw_if_index=1, flags=1)
> at /home/chenweihao/vpp_dev/src/plugins/dpdk/device/device.c:476
> #8  0x770d60e8 in vnet_sw_interface_set_flags_helper (
> vnm=0x77e2b828 , sw_if_index=1, 
> flags=VNET_SW_INTERFACE_FLAG_ADMIN_UP, helper_flags=0)
> at /home/chenweihao/vpp_dev/src/vnet/interface.c:470
> #9  0x770d645a in vnet_sw_interface_set_flags (
> vnm=0x77e2b828 , sw_if_index=1, 
> flags=VNET_SW_INTERFACE_FLAG_ADMIN_UP)
> at /home/chenweihao/vpp_dev/src/vnet/interface.c:524
> #10 0x7710515f in set_state (vm=0x7fffb6a00740, input=0x7fffa9f84bb8, 
> cmd=0x7fffb7180850)
> at /home/chenweihao/vpp_dev/src/vnet/interface_cli.c:946
> #11 0x77e72257 in vlib_cli_dispatch_sub_commands (vm=0x7fffb6a00740, 
> cm=0x77f6a770 , input=0x7fffa9f84bb8, 
> parent_command_index=20) at /home/chenweihao/vpp_dev/src/vlib/cli.c:650
> #12 0x77e71fea in vlib_cli_dispatch_sub_commands (vm=0x7fffb6a00740, 
> cm=0x77f6a770 , input=0x7fffa9f84bb8, 
> parent_command_index=7) at /home/chenweihao/vpp_dev/src/vlib/cli.c:607
> #13 0x77e71fea in vlib_cli_dispatch_sub_commands (vm=0x7fffb6a00740, 
> cm=0x77f6a770 , input=0x7fffa9f84bb8, 
> parent_command_index=0) at /home/chenweihao/vpp_dev/src/vlib/cli.c:607
> #14 0x77e7122a in vlib_cli_input (vm=0x7fffb6a00740, 
> input=0x7fffa9f84bb8, function=0x0, function_arg=0)
> at /home/chenweihao/vpp_dev/src/vlib/cli.c:753
> #15 0x77ef7e23 in unix_cli_exec (vm=0x7fffb6a00740, 
> input=0x7fffa9f84f30, cmd=0x7fffb71815b8)
> at /home/chenweihao/vpp_dev/src/vlib/unix/cli.c:3431
> #16 0x77e72257 in vlib_cli_dispatch_sub_commands (vm=0x7fffb6a00740, 
> cm=0x77f6a770 , input=0x7fffa9f84f30, 
> --Type  for more, q to quit, c to continue without paging--
> parent_command_index=0) at /home/chenweihao/vpp_dev/src/vlib/cli.c:650
> #17 0x77e7122a in vlib_cli_input (vm=0x7fffb6a00740, 
> input=0x7fffa9f84f30, function=0x0, function_arg=0)
> at /home/chenweihao/vpp_dev/src/vlib/cli.c:753
> #18 0x77efdfc5 in startup_config_process (vm=0x7fffb6a00740, 
> rt=0x7fffb9194080, f=0x0)
> at /home/chenweihao/vpp_dev/src/vlib/unix/main.c:291
> #19 0x77ea2c5d in vlib_process_bootstrap (_a=140736084405176)
> at /home/chenweihao/vpp_dev/src/vlib/main.c:1221
> #20 0x76f1ffd8 in clib_calljmp ()
> at /home/chenweihao/vpp_dev/src/vppinfra/longjmp.S:123
> #21 0x7fffac516bb0 in ?? ()
> #22 0x77ea26f9 in vlib_process_startup (vm=0x8, 
> p=0x77ea53bb , f=0x7fffac516cc0)
> at /home/chenweihao/vpp_dev/src/vlib/main.c:1246
> #23 0x76f7aa1c in vec_mem_size (v=0x7fffb6a00740)
> at /home/chenweihao/vpp_dev/src/vppinfra/vec.c:15
> #24 0x0581655dfd1c in ?? ()
> #25 0x00330004 in ?? ()
> #26 0x0030 in ?? ()
> #27 0x7fffbdbc7240 in ?? ()
> #28 0x7fffbdbc7240 in ?? ()
> #29 0x7fffb80e5498 in ?? ()
> #30 0x0001 in ?? ()
> #31 0x in ?? ()
> 
> I tried to change num-rx-queues and num-tx-queues to 4, then SIGSEGV not 
> happened.
> I applied the patch https://gerrit.fd.io/r/c/vpp/+/38529 , and the problem of 
> redis 6.0 seemed still exist, the stack backtrace is same with 
>

Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session

2023-03-21 Thread Chen Weihao

Thank you for your reply.
I tried to change num-tx-queues from 2 to 5, but it got a SIGSEGV, the 
backtrace is:
#0  0x7fffb453ff89 in rte_write32_relaxed (addr=0x80007ef0, value=0)
at ../src-dpdk/lib/eal/include/generic/rte_io.h:310
#1  rte_write32 (addr=0x80007ef0, value=0)
at ../src-dpdk/lib/eal/include/generic/rte_io.h:373
#2  vmxnet3_enable_intr (hw=0xac03b0600, intr_idx=4294967262)
at ../src-dpdk/drivers/net/vmxnet3/vmxnet3_ethdev.c:210
#3  0x7fffb4544d35 in vmxnet3_dev_rx_queue_intr_enable (
dev=0x7fffb5186980 , queue_id=0)
at ../src-dpdk/drivers/net/vmxnet3/vmxnet3_ethdev.c:1815
#4  0x7fffaff4bbf2 in rte_eth_dev_rx_intr_enable (port_id=0, queue_id=0)
at ../src-dpdk/lib/ethdev/rte_ethdev.c:4740
#5  0x7fffb49f4564 in dpdk_setup_interrupts (xd=0x7fffbdbb2940)
at /home/chenweihao/vpp_dev/src/plugins/dpdk/device/common.c:336
#6  0x7fffb49f4430 in dpdk_device_start (xd=0x7fffbdbb2940)
at /home/chenweihao/vpp_dev/src/plugins/dpdk/device/common.c:411
#7  0x7fffb49ff713 in dpdk_interface_admin_up_down (
vnm=0x77e2b828 , hw_if_index=1, flags=1)
at /home/chenweihao/vpp_dev/src/plugins/dpdk/device/device.c:476
#8  0x770d60e8 in vnet_sw_interface_set_flags_helper (
vnm=0x77e2b828 , sw_if_index=1,
flags=VNET_SW_INTERFACE_FLAG_ADMIN_UP, helper_flags=0)
at /home/chenweihao/vpp_dev/src/vnet/interface.c:470
#9  0x770d645a in vnet_sw_interface_set_flags (
vnm=0x77e2b828 , sw_if_index=1,
flags=VNET_SW_INTERFACE_FLAG_ADMIN_UP)
at /home/chenweihao/vpp_dev/src/vnet/interface.c:524
#10 0x7710515f in set_state (vm=0x7fffb6a00740, input=0x7fffa9f84bb8,
cmd=0x7fffb7180850)
at /home/chenweihao/vpp_dev/src/vnet/interface_cli.c:946
#11 0x77e72257 in vlib_cli_dispatch_sub_commands (vm=0x7fffb6a00740,
cm=0x77f6a770 , input=0x7fffa9f84bb8,
parent_command_index=20) at /home/chenweihao/vpp_dev/src/vlib/cli.c:650
#12 0x77e71fea in vlib_cli_dispatch_sub_commands (vm=0x7fffb6a00740,
cm=0x77f6a770 , input=0x7fffa9f84bb8,
parent_command_index=7) at /home/chenweihao/vpp_dev/src/vlib/cli.c:607
#13 0x77e71fea in vlib_cli_dispatch_sub_commands (vm=0x7fffb6a00740,
cm=0x77f6a770 , input=0x7fffa9f84bb8,
parent_command_index=0) at /home/chenweihao/vpp_dev/src/vlib/cli.c:607
#14 0x77e7122a in vlib_cli_input (vm=0x7fffb6a00740,
input=0x7fffa9f84bb8, function=0x0, function_arg=0)
at /home/chenweihao/vpp_dev/src/vlib/cli.c:753
#15 0x77ef7e23 in unix_cli_exec (vm=0x7fffb6a00740,
input=0x7fffa9f84f30, cmd=0x7fffb71815b8)
at /home/chenweihao/vpp_dev/src/vlib/unix/cli.c:3431
#16 0x77e72257 in vlib_cli_dispatch_sub_commands (vm=0x7fffb6a00740,
cm=0x77f6a770 , input=0x7fffa9f84f30,
--Type  for more, q to quit, c to continue without paging--
parent_command_index=0) at /home/chenweihao/vpp_dev/src/vlib/cli.c:650
#17 0x77e7122a in vlib_cli_input (vm=0x7fffb6a00740,
input=0x7fffa9f84f30, function=0x0, function_arg=0)
at /home/chenweihao/vpp_dev/src/vlib/cli.c:753
#18 0x77efdfc5 in startup_config_process (vm=0x7fffb6a00740,
rt=0x7fffb9194080, f=0x0)
at /home/chenweihao/vpp_dev/src/vlib/unix/main.c:291
#19 0x77ea2c5d in vlib_process_bootstrap (_a=140736084405176)
at /home/chenweihao/vpp_dev/src/vlib/main.c:1221
#20 0x76f1ffd8 in clib_calljmp ()
at /home/chenweihao/vpp_dev/src/vppinfra/longjmp.S:123
#21 0x7fffac516bb0 in ?? ()
#22 0x77ea26f9 in vlib_process_startup (vm=0x8,
p=0x77ea53bb , f=0x7fffac516cc0)
at /home/chenweihao/vpp_dev/src/vlib/main.c:1246
#23 0x76f7aa1c in vec_mem_size (v=0x7fffb6a00740)
at /home/chenweihao/vpp_dev/src/vppinfra/vec.c:15
#24 0x0581655dfd1c in ?? ()
#25 0x00330004 in ?? ()
#26 0x0030 in ?? ()
#27 0x7fffbdbc7240 in ?? ()
#28 0x7fffbdbc7240 in ?? ()
#29 0x7fffb80e5498 in ?? ()
#30 0x0001 in ?? ()
#31 0x in ?? ()

I tried to change num-rx-queues and num-tx-queues to 4, then SIGSEGV not 
happened.
I applied the patch https://gerrit.fd.io/r/c/vpp/+/38529 , and the problem of 
redis 6.0 seemed still exist, the stack backtrace is same with 
https://lists.fd.io/g/vpp-dev/message/22731

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22740): https://lists.fd.io/g/vpp-dev/message/22740
Mute This Topic: https://lists.fd.io/mt/97707720/21656
Mute #vnet:https://lists.fd.io/g/vpp-dev/mutehashtag/vnet
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session

2023-03-20 Thread Florin Coras

Hi, 

First of all, could you try this [1] with latest vpp? It’s really interesting 
that iperf does not exhibit this issue. 

Regarding your config, some observations:
- I see you have configured 4 worker. I would then recommend to use 4 rx-queues 
and 5 tx-queues (main can send packets), as opposed to 2. 
- tcp defaults to cubic, so config can be omitted.
- evt_qs_memfd_seg is not deprecated, so it can be omitted as well
- any particular reason for "set interface rx-mode eth1 polling”? dpdk 
interfaces are in polling mode by default
- you’re using binary api socket "api-socket-name /run/vpp/api.sock”. That 
works, but going forward we’ll slowly deprecate that api. So it’d recommend 
using the app socket api. See for instance [2] for changes needed to session 
stanza and vcl. 

Regards,
Florin

[1] https://gerrit.fd.io/r/c/vpp/+/38529
[2] https://wiki.fd.io/view/VPP/HostStack/LDP/iperf


> On Mar 20, 2023, at 5:50 AM, Chen Weihao  wrote:
> 
> Thanks for your reply.
> I give a more detailed backtrace and config in 
> https://lists.fd.io/g/vpp-dev/message/22731.  
> My installation method is to 
> clone vpp from github and make build on Ubuntu 22.04(Kernel version is 
> 5.19)，and I use make run for test and make debug for debugging. Yes, I yried 
> to make the server and client are attached to the same vpp instance.I tried 
> the latest version of vpp on github on yesterday, the problem is still exist.
> I am looking forward to your reply.
> 
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22734): https://lists.fd.io/g/vpp-dev/message/22734
Mute This Topic: https://lists.fd.io/mt/97707720/21656
Mute #vnet:https://lists.fd.io/g/vpp-dev/mutehashtag/vnet
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session

2023-03-20 Thread Chen Weihao

Thanks for your reply.
I give a more detailed backtrace and config in 
https://lists.fd.io/g/vpp-dev/message/22731. ( 
https://lists.fd.io/g/vpp-dev/message/22731, ) My installation method is to 
clone vpp from github and make build on Ubuntu 22.04(Kernel version is 
5.19)，and I use make run for test and make debug for debugging. Yes, I yried to 
make the server and client are attached to the same vpp instance.I tried the 
latest version of vpp on github on yesterday, the problem is still exist.
I am looking forward to your reply.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22732): https://lists.fd.io/g/vpp-dev/message/22732
Mute This Topic: https://lists.fd.io/mt/97707720/21656
Mute #vnet:https://lists.fd.io/g/vpp-dev/mutehashtag/vnet
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session

2023-03-20 Thread Chen Weihao

Thank you for your reply.
This is the stacktrace captured by gdb:

2: /home/chenweihao/vpp_dev/src/vnet/session/session.c:233 (session_is_valid) 
assertion `! pool_is_free (session_main.wrk[thread_index].sessions, _e)' fails

Thread 4 "vpp_wk_1" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffa93f5640 (LWP 4575)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140736032888384) at 
./nptl/pthread_kill.c:44
44./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140736032888384)
at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140736032888384)
at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140736032888384, signo=signo@entry=6)
at ./nptl/pthread_kill.c:89
#3  0x76a42476 in __GI_raise (sig=sig@entry=6)
at ../sysdeps/posix/raise.c:26
#4  0x76a287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0xb073 in os_panic ()
at /home/chenweihao/vpp_dev/src/vpp/vnet/main.c:417
#6  0x76f0a699 in debugger ()
at /home/chenweihao/vpp_dev/src/vppinfra/error.c:84
#7  0x76f0a450 in _clib_error (how_to_die=2, function_name=0x0,
line_number=0, fmt=0x77b88208 "%s:%d (%s) assertion `%s' fails")
at /home/chenweihao/vpp_dev/src/vppinfra/error.c:143
#8  0x775f31c7 in session_is_valid (si=4294967295,
thread_index=1 '\001')
at /home/chenweihao/vpp_dev/src/vnet/session/session.c:233
#9  0x775f177c in session_get (si=4294967295, thread_index=1)
at /home/chenweihao/vpp_dev/src/vnet/session/session.h:373
#10 0x775f3770 in ho_session_get (ho_index=4294967295)
at /home/chenweihao/vpp_dev/src/vnet/session/session.h:689
#11 0x775f37d8 in session_half_open_migrate_notify (tc=0x7fffbdba00c0)
at /home/chenweihao/vpp_dev/src/vnet/session/session.c:357
#12 0x77648600 in ct_accept_one (thread_index=2, ho_index=2)
at /home/chenweihao/vpp_dev/src/vnet/session/application_local.c:669
#13 0x77648243 in ct_accept_rpc_wrk_handler (rpc_args=0x2)
at /home/chenweihao/vpp_dev/src/vnet/session/application_local.c:760
#14 0x77620b42 in session_event_dispatch_ctrl (wrk=0x7fffbdb6fe00,
elt=0x7fffbdccaf2c)
at /home/chenweihao/vpp_dev/src/vnet/session/session_node.c:1656
#15 0x776175c2 in session_queue_node_fn (vm=0x7fffb82da700,
node=0x7fffbdbc21c0, frame=0x0)
at /home/chenweihao/vpp_dev/src/vnet/session/session_node.c:1962
#16 0x77ea3a62 in dispatch_node (vm=0x7fffb82da700,
node=0x7fffbdbc21c0, type=VLIB_NODE_TYPE_INPUT,
dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x0,
last_time_stamp=2911620275652)
at /home/chenweihao/vpp_dev/src/vlib/main.c:960
#17 0x77e9f7d1 in vlib_main_or_worker_loop (vm=0x7fffb82da700,
is_main=0) at /home/chenweihao/vpp_dev/src/vlib/main.c:1557
#18 0x77e9f1c7 in vlib_worker_loop (vm=0x7fffb82da700)
at /home/chenweihao/vpp_dev/src/vlib/main.c:1722
#19 0x77edb020 in vlib_worker_thread_fn (arg=0x7fffb9182980)
at /home/chenweihao/vpp_dev/src/vlib/threads.c:1598
#20 0x77ed62b6 in vlib_worker_thread_bootstrap_fn (arg=0x7fffb9182980)
at /home/chenweihao/vpp_dev/src/vlib/threads.c:418
#21 0x76a94b43 in start_thread (arg=)
at ./nptl/pthread_create.c:442
#22 0x76b26a00 in clone3 ()
at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
The resaon why assertion failed is that  c_s_index value is -1 (or ~0).

The redis veision I used:
git clone https://github.com/redis/redis.git
cd redis
git checkout 6.0
make

My startup.conf:
unix {
nodaemon
interactive
full-coredump
cli-listen /run/vpp/cli.sock
gid vpp
startup-config /home/chenweihao/startup.txt
}

api-trace {
on
}

cpu {
main-core 0
workers 4
}

tcp {
cc-algo cubic
}

dpdk {
uio-driver vfio-pci
dev :03:00.0 {
name eth1
num-rx-queues 2
num-tx-queues 2
}
}

buffers {
buffers-per-numa 131072
}

session { evt_qs_memfd_seg  }

My startup.txt:
set interface ip address eth1 192.168.0.2/24
set interface state eth1 up
set interface rx-mode eth1 polling
create loopback interface
set interface ip address loop0 127.0.0.1/8
set interface state loop0 up

My vcl.conf:
vcl {
rx-fifo-size 4000
tx-fifo-size 4000
app-scope-local
app-scope-global
api-socket-name /run/vpp/api.sock
use-mq-eventfd
}
I tried iperf3 as you said,indeed I did not encounter this problem.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22731): https://lists.fd.io/g/vpp-dev/message/22731
Mute This Topic: https://lists.fd.io/mt/97707720/21656
Mute #vnet:https://lists.fd.io/g/vpp-dev/mutehashtag/vnet
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session

2023-03-19 Thread Florin Coras

I just tried iperf3 in cut-through mode, i.e., server and client attached to 
the same vpp instance running 4 workers, with 128 connections and this seems to 
be working fine.

Could you try that out and see if it’s also working for you? It might be that 
this is something specific to how Redis uses sockets, so to reproduce we’ll 
need to replicate your testbed.

Regards,
Florin

> On Mar 19, 2023, at 2:58 PM, Florin Coras via lists.fd.io 
>  wrote:
> 
> Hi, 
> 
> That may very well be a problem introduced by the move of connects to first 
> worker. Unfortunately, I we don’t have tests for all of those corner cases 
> yet.
> 
> However, to replicate this issue, could you provide a bit more details about 
> your setup and the exact backtrace? It looks like you’re leverage cut-through 
> sessions so the server and client are attached to the same vpp instance? 
> Also, could you try vpp latest to see check if the issue still persists? 
> 
> Regards,
> Florin
> 
>> On Mar 19, 2023, at 1:53 AM, chenwei...@outlook.com wrote:
>> 
>> Hi vpp-team,
>>  I'm new to VPP and I'm trying to run Redis 6.0.18 in VCL with LD_PRELOAD 
>> using VPP 22.10 and VPP 23.02. I found that assert fails frequently in VPP 
>> 23.02, and after checking, I found that the assert fails in the session_get 
>> function in vnet/session/session.h. The cause was an invalid session_id with 
>> a value of -1 (or ~0).
>>  This function is called by the session_half_open_migrate_notify function in 
>> vnet/session/session.c, which is called by ct_accept_one in 
>> vnet/session/application_local.c. Function ct_accept_one is called because 
>> of an accept RPC request handled by the session_event_dispatch_ctrl function 
>> from the ct_connect function in vnet/session/application_local.c. Function 
>> ct_connect allocates and initializes a half-open transport object. However, 
>> its c_s_index value is -1 (or ~0), i.e., no session is allocated. allocating 
>> a session is implemented by calling session_alloc_for_half_open in the 
>> session_open_vc function of ct_connect (located in vnet/session/session.c). 
>> Therefore, I think the assertion failure is a case that ct_accept_one 
>> function accesses half-open tc without a session being allocated.
>>  I found that this problem does not exist on VPP 22.10. I checked the 
>> patches between 22.10 and 23.02 and found “session: move connects to first 
>> worker (https://gerrit.fd.io/r/c/vpp/+/35713)” that might be related to this 
>> issue, but I can't give a definite statement and I don’t know how fix it. I 
>> would be very grateful if you could address this issue.
>> Thanks,
>> 
>> 
>> 
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22726): https://lists.fd.io/g/vpp-dev/message/22726
Mute This Topic: https://lists.fd.io/mt/97707720/21656
Mute #vnet:https://lists.fd.io/g/vpp-dev/mutehashtag/vnet
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session

2023-03-19 Thread Florin Coras

Hi, 

That may very well be a problem introduced by the move of connects to first 
worker. Unfortunately, I we don’t have tests for all of those corner cases yet.

However, to replicate this issue, could you provide a bit more details about 
your setup and the exact backtrace? It looks like you’re leverage cut-through 
sessions so the server and client are attached to the same vpp instance? Also, 
could you try vpp latest to see check if the issue still persists? 

Regards,
Florin

> On Mar 19, 2023, at 1:53 AM, chenwei...@outlook.com wrote:
> 
> Hi vpp-team,
>   I'm new to VPP and I'm trying to run Redis 6.0.18 in VCL with LD_PRELOAD 
> using VPP 22.10 and VPP 23.02. I found that assert fails frequently in VPP 
> 23.02, and after checking, I found that the assert fails in the session_get 
> function in vnet/session/session.h. The cause was an invalid session_id with 
> a value of -1 (or ~0).
>   This function is called by the session_half_open_migrate_notify function in 
> vnet/session/session.c, which is called by ct_accept_one in 
> vnet/session/application_local.c. Function ct_accept_one is called because of 
> an accept RPC request handled by the session_event_dispatch_ctrl function 
> from the ct_connect function in vnet/session/application_local.c. Function 
> ct_connect allocates and initializes a half-open transport object. However, 
> its c_s_index value is -1 (or ~0), i.e., no session is allocated. allocating 
> a session is implemented by calling session_alloc_for_half_open in the 
> session_open_vc function of ct_connect (located in vnet/session/session.c). 
> Therefore, I think the assertion failure is a case that ct_accept_one 
> function accesses half-open tc without a session being allocated.
>   I found that this problem does not exist on VPP 22.10. I checked the 
> patches between 22.10 and 23.02 and found “session: move connects to first 
> worker (https://gerrit.fd.io/r/c/vpp/+/35713)” that might be related to this 
> issue, but I can't give a definite statement and I don’t know how fix it. I 
> would be very grateful if you could address this issue.
> Thanks,
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22724): https://lists.fd.io/g/vpp-dev/message/22724
Mute This Topic: https://lists.fd.io/mt/97707720/21656
Mute #vnet:https://lists.fd.io/g/vpp-dev/mutehashtag/vnet
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session

Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session

Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session

Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session

Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session

Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session

Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session

Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session

Re: [vpp-dev] #vnet A bug which may cause assertion error in vnet/session

9 matches

Site Navigation

Mail list logo

Footer information