[vpp-dev] Support for VPP compilation in offline mode
Hi, Currently as part of VPP compilation, there are commands such as "make install-dep" and "make install-ext-deps" which downloads required dependent packages from internet. We want to automate the VPP compilation in offline mode i.e. our environment will not have internet access and we want to compile VPP in such an environment. Since the above mentioned commands by default reach out to internet for downloading the required packages, how can we have an offline setup for VPP compilation? Thanks and Regards, Chinmaya Agarwal. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22410): https://lists.fd.io/g/vpp-dev/message/22410 Mute This Topic: https://lists.fd.io/mt/96035761/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Slow VPP performance vs. DPDK l2fwd / l3wfd
Hi Matt, thanks. The *no-multi-seq* option is actually dropping the performance even more. Once enable it drops from 5 Mpps ( out of expected 10 Mpps) to less than < 1 Mpps. Therefore I disabled the option again. The dpdk applications foward the full 10 Mpps without any dev-args: > > ./dpdk-l2fwd -n 4 -l 6 -a :4b:00.0 -a :4b:00.1 -- -q 2 -p 0x3 but also adding those options confirm the full 10 Mpps. Either way *l2fwd* will not drop any packets and run the full load. > > ./dpdk-l2fwd -n 4 -l 6 -a > :4b:00.0,mprq_en=1,rxqs_min_mprq=1,mprq_log_stride_num=9,txq_inline_mpw=128,rxq_pkt_pad_en=1,dv_flow_en=0 > -a > :4b:00.1,mprq_en=1,rxqs_min_mprq=1,mprq_log_stride_num=9,txq_inline_mpw=128,rxq_pkt_pad_en=1,dv_flow_en=0 > -- -q 2 -p 0x3 Contrary by using the default */etc/vpp/startup.conf* options of DPDK I only yield 4 Mpps out of 10 Mpps expected. Once I added the devargs *mprq_en=1,rxqs_min_mprq=1,mprq_log_stride_num=9,txq_inline_mpw=128,rxq_pkt_pad_en=1,dv_flow_en=0* it gained another 20% ( as recommended in DPDK mellanox perf-report ) -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22409): https://lists.fd.io/g/vpp-dev/message/22409 Mute This Topic: https://lists.fd.io/mt/95959719/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Slow VPP performance vs. DPDK l2fwd / l3wfd
What arguments did you pass to l2fwd and l3fwd when you started them up? Your NIC statistics show RX misses due to lack of available buffers HundredGigabitEthernet4b/0/0 1 up HundredGigabitEthernet4b/0/0 [...] rx_out_of_buffer 86941 HundredGigabitEthernet4b/0/1 2 up HundredGigabitEthernet4b/0/1 [...] rx_out_of_buffer155395 I don't know if those errors are enough to cause the difference in your throughput measurements between l2fwd/l3fwd and VPP, but it would probably be worthwhile to increase buffers-per-numa and repeat your tests. I have heard advice in the past that performance with Mellanox DPDK PMDs can be improved by setting the no-multi-seg option. I don't know whether that is still true and I never compared performance with that option set vs without, so I'm not sure how much it would help. But it may be worth trying. Thanks, -Matt On Tue, Jan 3, 2023 at 8:21 AM wrote: > Hi @Benoit, yes I can confirm NIC and VPP worker are on same node-0 . I am > also using the same core-id for the benchmark comparison against plain dpdk > l2fwd/l3fwd. > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22408): https://lists.fd.io/g/vpp-dev/message/22408 Mute This Topic: https://lists.fd.io/mt/95959719/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Slow VPP performance vs. DPDK l2fwd / l3wfd
Hi @Benoit, yes I can confirm NIC and VPP worker are on same node-0 . I am also using the same core-id for the benchmark comparison against plain dpdk l2fwd/l3fwd. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22407): https://lists.fd.io/g/vpp-dev/message/22407 Mute This Topic: https://lists.fd.io/mt/95959719/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] VPP crashes with vlib_worker_thread_barrier_syn_int:thread deadlock
The issue seems to be that the main thread wants to reply to an ARP, but it
timeouts on the worker barrier:
1) why the worker does not yield to the main thread in a timely manner:
workers should always complete processing in less than 1s. You can try to use
elog to identify which nodes takes too long:
https://s3-docs.fd.io/vpp/23.02/developer/corefeatures/eventviewer.html
2) why does the main thread is receiving ARP to begin with? Can you share the
output of 'show int rx'?
Best
ben
> -Original Message-
> From: vpp-dev@lists.fd.io On Behalf Of Mechthild
> Buescher via lists.fd.io
> Sent: Thursday, December 22, 2022 12:25
> To: vpp-dev@lists.fd.io
> Subject: Re: [vpp-dev] VPP crashes with
> vlib_worker_thread_barrier_syn_int:thread deadlock
>
> Hi Piotr,
>
>
>
> Thanks for your hint. We could nail down the problem a bit and would like
> to ask for your suggestion on how to solve it.
>
>
>
> The problem occurs when there is traffic received on a DPDK-interface and
> shall be forwarded to a host interface. Here are the snippets of the
> configuration which we think is relevant (note, it’s a different setup
> compared to previous email, here we have one socket and less cpus):
>
>
>
> We configure CPU’s 2,17,18 to be isolated:
>
> # cat /proc/cmdline
>
> BOOT_IMAGE=/vmlinuz-5.3.18-150300.59.76-default root=UUID=439b3b24-9c1d-
> 4b6f-b024-539b50cb7480 rootflags=subvol=@ intel_iommu=on iommu=pt
> intel_idle.max_cstate=0 processor.max_cstate=0 idle=poll
> intel_pstate=disable isolcpus=2,3,4,5,6,17,18,19,20,21,22 nohz=on
> nohz_full=2,3,4,5,6,17,18,19,20,21,22
> rcu_nocbs=2,3,4,5,6,17,18,19,20,21,22 rcu_nocb_poll
> irqaffinity=0,1,7,8,9,10,11,12,13,14,15,16,23,24,25,26,27,28,29,30,31
> hugepagesz=2M hugepages=2048 hugepagesz=1G hugepages=4
> default_hugepagesz=2M transparent_hugepage=never nosoftlookup
> nmi_watchdog=0 tsc=reliable hpet=disable clocksource=tsc skew_tick=1
> mce=ignore_ce splash console=ttyS0,115200 psi=1 audit=1 apparmor=1
> security=apparmor
>
>
>
> And we use those isolated CPU’s for the workers and one non-isolated CPU
> for the main-thread:
>
> cpu {
>
> main-core 1
>
> corelist-workers 2,17,18
>
> }
>
>
>
> The relevant DPDK-interface is Radio-0:
>
> dpdk {
>
> dev default {
>
> num-rx-queues 3
>
> }
>
>
>
> uio-driver vfio-pci
>
>
>
> dev :17:00.1 {
>
> name Radio-0
>
> }
>
> :
>
> }
>
>
>
> And then we have the following configuration:
>
> set interface state Radio-0 up
>
> create host-interface name Vpp2Host
>
> set interface state host-Vpp2Host up
>
> set interface rx-placement host-Vpp2Host main
>
> create sub-interfaces Radio-0 3092
>
> set interface state Radio-0.3092 up
>
> create sub-interfaces host-Vpp2Host 3092
>
> set interface state host-Vpp2Host.3092 up
>
> set interface l2 bridge Radio-0.3092 3092
>
> set interface l2 bridge host-Vpp2Host.3092 3092
>
>
>
> This means, we receive traffic on a DPDK interface and try to forward it
> via a L2 bridge to the host. The DPDK interface is on an isolated CPU
> while the host interface is on a non-isolated CPU. My suspicion is that
> this is the problem – do you agree? Do you have any idea how we can solve
> this? The Radio-0 interface is used for OAM via vlan 3092 (this is what
> you see in the above configuration) as well as for traffic (untagged),
> that’s why we want to have it on an isolated CPU.
>
>
>
> Thank you for your support,
>
>
>
> BR/Mechthild
>
>
>
> From: vpp-dev@lists.fd.io on behalf of Bronowski,
> PiotrX via lists.fd.io
> Date: Wednesday, 21. December 2022 at 17:24
> To: vpp-dev@lists.fd.io
> Subject: Re: [vpp-dev] VPP crashes with
> vlib_worker_thread_barrier_syn_int:thread deadlock
>
> Hi Mechthild,
>
>
>
> Your issue is caused by the main thread waiting too long for a worker to
> finish. You may examine these lines in src/vlib/threads.h:
>
>
>
> 171 /* Check for a barrier sync request every 30ms */
>
> 172 #define BARRIER_SYNC_DELAY (0.03)
>
> 173
>
> 174 #if CLIB_DEBUG > 0
>
> 175 /* long barrier timeout, for gdb... */
>
> 176 #define BARRIER_SYNC_TIMEOUT (600.1)
>
> 177 #else
>
> 178 #define BARRIER_SYNC_TIMEOUT (1.0)
>
> 179 #endif
>
>
>
> Your restart is caused by the timeout defined in these lines. You may
> increase it to investigate your issue (of course it is not a fix). After
> increasing timeout and being in interactive mode you can issue command
> “show run” it will tell you in which node you are spending most of your
> time and potentially identify source of your problem. Alternatively, you
> may use perf tool for that task.
>
> BR,
>
> Piotr
>
>
>
> From: vpp-dev@lists.fd.io On Behalf Of Mechthild
> Buescher via lists.fd.io
> Sent: Wednesday, December 21, 2022 3:24 PM
> To: vpp-dev@lists.fd.io
> Subject: [vpp-dev] VPP crashes with
> vlib_worker_thread_barrier_syn_int:thread deadlock
>
>
>
> Hi,
>
Re: [vpp-dev] Unable to run make pkg-rpm with Sanitizer enable.
> I am using GCC version 8.5.0 20210514 (Red Hat 8.5.0-17) (GCC) which is > upgraded and has the above errors. > How it can be fixed? My point was to use either a newer version of GCC or LLVM - AddressSanitizer is quite demanding for the compiler, the newest the better. I think CentOS maintains 'devtoolset' packages with newer compilers. Best ben -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22405): https://lists.fd.io/g/vpp-dev/message/22405 Mute This Topic: https://lists.fd.io/mt/96024389/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Unable to run make pkg-rpm with Sanitizer enable.
Thanks for the frequent reply. Hi Benoit, I am using GCC version 8.5.0 20210514 (Red Hat 8.5.0-17) (GCC) which is upgraded and has the above errors. How it can be fixed? Regards, Kamaljeet -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22404): https://lists.fd.io/g/vpp-dev/message/22404 Mute This Topic: https://lists.fd.io/mt/96024389/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Slow VPP performance vs. DPDK l2fwd / l3wfd
I see you run the VPP worker on core 6, is it on the same NUMA node as the NIC? You can check with 'sh pci' and 'sh thr'. Best ben > -Original Message- > From: vpp-dev@lists.fd.io On Behalf Of r...@gmx.net > Sent: Friday, December 30, 2022 20:05 > To: vpp-dev@lists.fd.io > Subject: [vpp-dev] Slow VPP performance vs. DPDK l2fwd / l3wfd > > > Hi VPP team, > > need to wrap my head around why VPP is not even able to process 10 Mpps > one single-core setup. > Afaik VPP xconnect ( comparable to l2fwd) or L3-routing ( l3wfd) should > yield 10 Mpps even back in 2017 ( slide 15 > https://wiki.fd.io/images/3/31/Benchmarking-sw-data-planes-Dec5_2017.pdf > or p16 https://lfnetworking.org/wp- > content/uploads/sites/7/2022/06/benchmarking_sw_data_planes_skx_bdx_mar07_ > 2019.pdf ) > > My setup is quite up-to-date: > > Traffic Gen: TRex running on Intel(R) Xeon(R) Gold 6354 CPU @ 3.00GHz > DuT: VPP 22.10 running on Icelake Intel(R) Xeon(R) Platinum 8352Y CPU @ > 2.20GHz > > NICs: ConnectX-6 Dx Dualport > OS: Ubuntu Server 20.04 > > VPP compiled from src . Tag 22.10. Incl. changes to compile for MLX5_PMD. > > VPP show dpdk version: > DPDK Version: DPDK 22.07.0 > DPDK EAL init args: --in-memory --no-telemetry --file-prefix vpp -a > :4b:00.0,mprq_en=1,rxqs_min_mprq=1,mprq_log_stride_num=9,txq_inline_mp > w=128,rxq_pkt_pad_en=1 -a > :4b:00.1,mprq_en=1,rxqs_min_mprq=1,mprq_log_stride_num=9,txq_inline_mp > w=128,rxq_pkt_pad_en=1 > > QUESTION: > > Running the a plain DPDK 22.07 example l2fwd or l3fwd yields > 10 Mpps ( > 100 Gbps line rate) with 1 lcore. Also no additional dpdk options > required. > > Once I run either xconnect or L3 routing with VPP, the same test drops to > 6 Mpps or less. It is puzzling why VPP is not even close the same exact > dpdk l2fwd/ l3wfd example? Any advise? > > > Here are the additional cmds to configure either l2fwd or L3-routing in > VPP: > > ```bash > set interface l2 xconnect HundredGigabitEthernet4b/0/0 > HundredGigabitEthernet4b/0/1 > set interface l2 xconnect HundredGigabitEthernet4b/0/1 > HundredGigabitEthernet4b/0/0 > > # l3 routing > set interface ip address HundredGigabitEthernet4b/0/0 10.10.1.1/24 > set interface ip address HundredGigabitEthernet4b/0/1 10.10.2.1/24 > ip route add 16.0.0.0/8 via 10.10.1.2 > ip route add 48.0.0.0/8 via 10.10.2.2 > > ``` > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22403): https://lists.fd.io/g/vpp-dev/message/22403 Mute This Topic: https://lists.fd.io/mt/95959719/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Unable to run make pkg-rpm with Sanitizer enable.
Which compiler do you use? Try upgrading to a newer toolchain or switch between GCC and LLVM. Best ben > -Original Message- > From: vpp-dev@lists.fd.io On Behalf Of > 2201kamalj...@gmail.com > Sent: Tuesday, January 3, 2023 8:09 > To: vpp-dev@lists.fd.io > Subject: [vpp-dev] Unable to run make pkg-rpm with Sanitizer enable. > > Hi All, > > > I want to build and make pkg-rpm with Address sanitizer=ON on CentOS > Stream release 8 > > While compiling it is working fine with this command : make build > VPP_EXTRA_CMAKE_ARGS=-DVPP_ENABLE_SANITIZE_ADDR=ON, and also showing that > address sanitizer(flag) is enabled. > But, when while doing make pkg-rpm with the command : make pkg-rpm > VPP_EXTRA_CMAKE_ARGS=-DVPP_ENABLE_SANITIZE_ADDR=ON > It is producing the below error: > > > home/usr/vpp/build-root/rpmbuild/vpp-23.02/src/vppinfra/vec_bootstrap.h: > In function 'vec_header': > /home/usr/vpp/build-root/rpmbuild/vpp- > 23.02/src/vppinfra/vec_bootstrap.h:90:1: error: inlining failed in call to > always_inline 'vec_get_header_size': function attribute mismatch > vec_get_header_size (void *v) > ^~~ > /home/usr/vpp/build-root/rpmbuild/vpp- > 23.02/src/vppinfra/vec_bootstrap.h:105:18: note: called from here >return v ? v - vec_get_header_size (v) : 0; > ^~~ > Regards, > Kamaljeet -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22402): https://lists.fd.io/g/vpp-dev/message/22402 Mute This Topic: https://lists.fd.io/mt/96024389/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
