Re: [vpp-dev] vlib_get_plugin_symbol returning null after upgrading to 21.06 version

[Mohammed Hawari]

The cleanest solution would be to write a wrapper for each one of those 
autogenerated functions, and mark the wrapper as __clib_export

Best regards

Mohammed
> On 29 Sep 2021, at 15:46, Satya Murthy  wrote:
> 
> This works fine for me where the code is fully written by us.
> But, I have few functions which are kind of generated routines from a third 
> party library, where we do not have control to add this __clib_export.
> Is there any way to work around this case?
> 
> -- 
> Thanks & Regards,
> Murthy 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#20228): https://lists.fd.io/g/vpp-dev/message/20228
Mute This Topic: https://lists.fd.io/mt/85945383/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vlib_get_plugin_symbol returning null after upgrading to 21.06 version

[Mohammed Hawari]

Hi,

Please note that starting with v21.01, plugin symbols are hidden by default and 
can’t be used from another plugin (or any other linked object). You need to add 
the __clib_export attribute to the symbols you want to expose to other plugins. 
For example, in the dns plugin, the dns_resolve_name function is marked with 
__clib_export, in file plugins/dns/dns.c:

__clib_export int
dns_resolve_name (u8 *name, dns_cache_entry_t **ep, dns_pending_request_t *t0,
  dns_resolve_name_t *rn)
{

This enables its use by the ikev2 plugin. In file plugins/ikev2/ikev2.c you 
have 
 
km->dns_resolve_name =
vlib_get_plugin_symbol ("dns_plugin.so", "dns_resolve_name”);

Best regards,

Mohammed

> On 29 Sep 2021, at 12:59, Dave Barach  wrote:
> 
> It’s worth checking that the symbol in question is actually exported. As part 
> of upgrading, you may have started using a different toolchain.
>  
> $ nm -go .o | grep 
>  
> From: vpp-dev@lists.fd.io   > On Behalf Of Satya Murthy
> Sent: Wednesday, September 29, 2021 6:43 AM
> To: vpp-dev@lists.fd.io 
> Subject: [vpp-dev] vlib_get_plugin_symbol returning null after upgrading to 
> 21.06 version
>  
> Hi,
> 
> We are trying to upgrade to fdio.2106 version from a previous version. 
> 
> After upgrading to 21.06 version with our custom plugins, we are seeing that 
> vlib_get_plugin_symbol()  to resolve a symbol from another plugin is always 
> returning null.
> This was working fine in the earlier version.
> 
> I cross checked that load_one_plugin() is successful in loading all the 
> plugins, before we call the vlib_get_plugin_symbol().
> 
> Any pointers on what could be happening here ?
> 
> -- 
> Thanks & Regards,
> Murthy 
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#20223): https://lists.fd.io/g/vpp-dev/message/20223
Mute This Topic: https://lists.fd.io/mt/85945383/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] How to enable Mellanox compilation in VPP 21.01

[Mohammed Hawari]

Hi Julius,

Thanks for finding that issue on master, could you verify that this patch fixes 
it? https://gerrit.fd.io/r/c/vpp/+/32042 <https://gerrit.fd.io/r/c/vpp/+/32042>

Best regards,

Mohammed

> On 19 Apr 2021, at 09:08, Július Milan  wrote:
> 
> Hi Mohammed
> 
> Thanks a lot, I confirm that it worked well on branch stable/2101.
> 
> However on master (ae34872077c956aa34ee816b55ccb5c5f6ab40a1), I got the 
> following error:
> $ make run
> /home/pantheontech/dev/vpp_master/build-root/install-vpp_debug-native/vpp/bin/vpp:
>  symbol lookup error: 
> /home/pantheontech/dev/vpp_master/build-root/install-vpp_debug-native/vpp/lib/vpp_plugins/dpdk_plugin.so:
>  undefined symbol: ibv_fork_init
> 
> Following the same procedure as for stable/2101, compiling from the clean 
> workspace, having the only changes:
> $ git diff
> diff --git a/build/external/packages/dpdk.mk <http://dpdk.mk/> 
> b/build/external/packages/dpdk.mk <http://dpdk.mk/>
> index c7eb0fc3f..f1663a1aa 100644
> --- a/build/external/packages/dpdk.mk <http://dpdk.mk/>
> +++ b/build/external/packages/dpdk.mk <http://dpdk.mk/>
> @@ -14,9 +14,9 @@
>  DPDK_PKTMBUF_HEADROOM?= 128
>  DPDK_USE_LIBBSD  ?= n
>  DPDK_DEBUG   ?= n
> -DPDK_MLX4_PMD?= n
> -DPDK_MLX5_PMD?= n
> -DPDK_MLX5_COMMON_PMD ?= n
> +DPDK_MLX4_PMD:= y
> +DPDK_MLX5_PMD:= y
> +DPDK_MLX5_COMMON_PMD := y
>  DPDK_TAP_PMD ?= n
>  DPDK_FAILSAFE_PMD    ?= n
>  DPDK_MACHINE ?= default
> 
> Best regards
> Julius
> 
> On Fri, 16 Apr 2021 at 17:27, Mohammed Hawari  <mailto:moham...@hawari.fr>> wrote:
> Hi Julius,
> 
> You don’t need any patch for Ubuntu it should work out of the box on 21.01… 
> This patch is CentOS specific because the default libdir on CentOS is ‘lib64' 
> (while it is ‘lib' on Ubuntu/Debian). On Ubuntu, you only need to edit 
> dpdk.mk <http://dpdk.mk/> to enable MLX4_PMD, MLX5_PMD and MLX5_COMMON_PMD. 
> Internally it works like this:
> - rdma-core (and the libibverbs *static* library) is built and installed in 
> some internal staging directory (build-root/install-vpp-native/external/lib/).
> - DPDK is configured to be statically linked against this libibverb statis 
> library, and is also statically built.
> - then the VPP dpdk plugin is statically linked against DPDK.
> 
> The patch fixes the first step, as on CentOS, cmake will install the 
> rdma-core libraries in build-root/install-vpp-native/external/lib64 while the 
> DPDK build system will look for them in 
> build-root/install-vpp-native/external/lib. Note that this patch is not 
> “specific” to CentOS, but is *required* for CentOS support (and is merged in 
> master).
> 
> Best regards,
> Mohammed
> 
>> On 16 Apr 2021, at 16:55, Július Milan > <mailto:jmilan@gmail.com>> wrote:
>> 
>> Hi Mohammed
>> 
>> I am trying to create the patch similar to yours: 
>> https://gerrit.fd.io/r/c/vpp/+/31876 <https://gerrit.fd.io/r/c/vpp/+/31876>
>> But being able to run on Ubuntu.
>> Can you point me a bit? I would like to understand how your patch is CentOS 
>> specific.
>> Probably it has something to do with linking of libibverbs.so into 
>> dpdk_plugin, which I was not able to achieve yet.
>> Or am I completely wrong?
>> 
>> Best regards
>> Julius
>> 
>> On Fri, 16 Apr 2021 at 09:49, Mohammed Hawari > <mailto:moham...@hawari.fr>> wrote:
>> 
>> Hi Chetan,
>> 
>>> On 16 Apr 2021, at 07:04, chetan bhasin >> <mailto:chetan.bhasin...@gmail.com>> wrote:
>>> 
>>> Hi Mohammed ,
>>> 
>>> After applying the patch you have mentioned , I am no longer facing the 
>>> compilation issue.
>>> 
>>> Thanks for your support. 
>>> 
>>> With Static RDMA linking , do we need to do anything extra while bringing 
>>> our application on the server having Mellanox Nic ?
>> 
>> No we don’t, it should work out of the box!
>>> 
>>> Thanks & Regards,
>>> Chetan
>> 
>> Regards
>> 
>> Mohammed
>>> 
>>> On Wed, Apr 14, 2021 at 8:33 PM chetan bhasin via lists.fd.io 
>>> <http://lists.fd.io/> >> <mailto:gmail@lists.fd.io>> wrote:
>>> Thanks a lot Mohammed!
>>> 
>>> Let me try with the rdma-core and patch mentioned by you and 
>>> 
>>> Thanks,
>>> Chetan
>>> 
>>> On Wed, Apr 14, 2021 at 6:20 PM Mohammed Hawari >> <m

Re: [vpp-dev] How to enable Mellanox compilation in VPP 21.01

[Mohammed Hawari]

Hi Chetan,

> On 16 Apr 2021, at 07:04, chetan bhasin  wrote:
> 
> Hi Mohammed ,
> 
> After applying the patch you have mentioned , I am no longer facing the 
> compilation issue.
> 
> Thanks for your support. 
> 
> With Static RDMA linking , do we need to do anything extra while bringing our 
> application on the server having Mellanox Nic ?

No we don’t, it should work out of the box!
> 
> Thanks & Regards,
> Chetan

Regards

Mohammed
> 
> On Wed, Apr 14, 2021 at 8:33 PM chetan bhasin via lists.fd.io 
> <http://lists.fd.io/>  <mailto:gmail@lists.fd.io>> wrote:
> Thanks a lot Mohammed!
> 
> Let me try with the rdma-core and patch mentioned by you and 
> 
> Thanks,
> Chetan
> 
> On Wed, Apr 14, 2021 at 6:20 PM Mohammed Hawari  <mailto:moham...@hawari.fr>> wrote:
> Hi Chetan
> 
> You are building DPDK with Mellanox support relying on dlopen and the glue 
> library which is fine. The default approach implemented in the VPP build 
> system consists of statically linking DPDK with the the libraries provided by 
> rdma-core (as it’s also built by the VPP build system). The default approach 
> should work fine, provided this patch https://gerrit.fd.io/r/c/vpp/+/31876 
> <https://gerrit.fd.io/r/c/vpp/+/31876> is applied (specific to CentOS). But 
> what you should work just fine. Now the reason why  you need to comment 
> dpdk_depends is kind of mysterious here…
> 
> Regards,
> 
> Mohammed
>> On 14 Apr 2021, at 10:07, chetan bhasin > <mailto:chetan.bhasin...@gmail.com>> wrote:
>> 
>> Hi,
>> 
>> I have to do the following to enable Mellanox compilation under VPP21.01 
>> with dpdk 20.11.
>> 
>> If I dont comment rdma-core dependencies , it will lead to undefined symbols 
>> . 
>> 
>> Can anybody please correct , is this the right way to do it?
>> 
>> -DPDK_MLX4_PMD?= n
>> -DPDK_MLX5_PMD?= n
>> -DPDK_MLX5_COMMON_PMD ?= n
>> +DPDK_MLX4_PMD:= y
>> +DPDK_MLX5_PMD:= y
>> +DPDK_MLX5_COMMON_PMD = y
>> 
>> -DPDK_MLX_IBV_LINK?= static
>> +DPDK_MLX_IBV_LINK:= dlopen
>> 
>> -dpdk_depends:= rdma-core $(if $(ARCH_X86_64), ipsec-mb)
>> +#dpdk_depends   := rdma-core $(if $(ARCH_X86_64), ipsec-mb)
>> 
>> Thanks,
>> Chetan
>> 
>> On Tue, Apr 6, 2021 at 4:10 PM chetan bhasin via lists.fd.io 
>> <http://lists.fd.io/> > <mailto:gmail@lists.fd.io>> wrote:
>> Thanks Guys !
>> 
>> We have a requirement to compile vppp with DPDK mellanox driver . 
>> 
>> Right now issue what i understood is , when we compile vpp_2101  , using 
>> make build-release , it builds mellanox , but when it is trying to install 
>> vpp-ext-dep rpm , it got failed because glue library created of mellanox has 
>> some undefined symbols ,
>> For your reference , i have set following in dpdk.mk <http://dpdk.mk/> file
>> 
>> -DPDK_MLX4_PMD?= n
>> -DPDK_MLX5_PMD?= n
>> +DPDK_MLX4_PMD:= y
>> +DPDK_MLX5_PMD:= y
>> -DPDK_MLX_IBV_LINK?= static
>> +DPDK_MLX_IBV_LINK:= shared
>> 
>>  please check below logs for reference- 
>> 
>> FAILED: drivers/net/mlx4/librte_net_mlx4_glue.so.21.0
>> cc  -o drivers/net/mlx4/librte_net_mlx4_glue.so.21.0 
>> 'drivers/net/mlx4/8672f8e@@rte_net_mlx4_glue@sha/mlx4_glue.c.o' 
>> -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -shared -fPIC -Wl,--start-group 
>> -Wl,-soname,librte_net_mlx4_glue.so.21.0 -Wl,--no-as-needed -pthread -lm 
>> -ldl -lnuma -Wl,-export-dynamic -Wl,-h,librte_net_mlx4_glue.so.21.0 
>> /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libmlx4.so 
>> /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libibverbs.so 
>> -Wl,--end-group
>> drivers/net/mlx4/8672f8e@@rte_net_mlx4_glue@sha/mlx4_glue.c.o: In function 
>> `mlx4_glue_reg_mr':
>> mlx4_glue.c:(.text+0x277): undefined reference to `ibv_reg_mr_iova2'
>> collect2: error: ld returned 1 exit status
>> [8/547] Compiling C object 
>> 'drivers/a715181@@tmp_rte_net_mlx4@sta/net_mlx4_mlx4_flow.c.o'
>> 
>> ninja: build stopped: subcommand failed.
>> 
>> On Thu, Apr 1, 2021 at 4:33 PM Юрий Иванов > <mailto:format_...@outlook.com>> wrote:
>> Hi,
>> 
>> Or you can use nativa rdma driver as written in this post and build vpp as 
>> usual. 
>> 
>> Regards.
>> От: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.

Re: [vpp-dev] How to enable Mellanox compilation in VPP 21.01

[Mohammed Hawari]

Hi Chetan

You are building DPDK with Mellanox support relying on dlopen and the glue 
library which is fine. The default approach implemented in the VPP build system 
consists of statically linking DPDK with the the libraries provided by 
rdma-core (as it’s also built by the VPP build system). The default approach 
should work fine, provided this patch https://gerrit.fd.io/r/c/vpp/+/31876 
<https://gerrit.fd.io/r/c/vpp/+/31876> is applied (specific to CentOS). But 
what you should work just fine. Now the reason why  you need to comment 
dpdk_depends is kind of mysterious here…

Regards,

Mohammed
> On 14 Apr 2021, at 10:07, chetan bhasin  wrote:
> 
> Hi,
> 
> I have to do the following to enable Mellanox compilation under VPP21.01 with 
> dpdk 20.11.
> 
> If I dont comment rdma-core dependencies , it will lead to undefined symbols 
> . 
> 
> Can anybody please correct , is this the right way to do it?
> 
> -DPDK_MLX4_PMD?= n
> -DPDK_MLX5_PMD?= n
> -DPDK_MLX5_COMMON_PMD ?= n
> +DPDK_MLX4_PMD:= y
> +DPDK_MLX5_PMD:= y
> +DPDK_MLX5_COMMON_PMD = y
> 
> -DPDK_MLX_IBV_LINK?= static
> +DPDK_MLX_IBV_LINK:= dlopen
> 
> -dpdk_depends:= rdma-core $(if $(ARCH_X86_64), ipsec-mb)
> +#dpdk_depends   := rdma-core $(if $(ARCH_X86_64), ipsec-mb)
> 
> Thanks,
> Chetan
> 
> On Tue, Apr 6, 2021 at 4:10 PM chetan bhasin via lists.fd.io 
> <http://lists.fd.io/>  <mailto:gmail@lists.fd.io>> wrote:
> Thanks Guys !
> 
> We have a requirement to compile vppp with DPDK mellanox driver . 
> 
> Right now issue what i understood is , when we compile vpp_2101  , using make 
> build-release , it builds mellanox , but when it is trying to install 
> vpp-ext-dep rpm , it got failed because glue library created of mellanox has 
> some undefined symbols ,
> For your reference , i have set following in dpdk.mk <http://dpdk.mk/> file
> 
> -DPDK_MLX4_PMD?= n
> -DPDK_MLX5_PMD?= n
> +DPDK_MLX4_PMD:= y
> +DPDK_MLX5_PMD:= y
> -DPDK_MLX_IBV_LINK?= static
> +DPDK_MLX_IBV_LINK:= shared
> 
>  please check below logs for reference- 
> 
> FAILED: drivers/net/mlx4/librte_net_mlx4_glue.so.21.0
> cc  -o drivers/net/mlx4/librte_net_mlx4_glue.so.21.0 
> 'drivers/net/mlx4/8672f8e@@rte_net_mlx4_glue@sha/mlx4_glue.c.o' 
> -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -shared -fPIC -Wl,--start-group 
> -Wl,-soname,librte_net_mlx4_glue.so.21.0 -Wl,--no-as-needed -pthread -lm -ldl 
> -lnuma -Wl,-export-dynamic -Wl,-h,librte_net_mlx4_glue.so.21.0 
> /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libmlx4.so 
> /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libibverbs.so 
> -Wl,--end-group
> drivers/net/mlx4/8672f8e@@rte_net_mlx4_glue@sha/mlx4_glue.c.o: In function 
> `mlx4_glue_reg_mr':
> mlx4_glue.c:(.text+0x277): undefined reference to `ibv_reg_mr_iova2'
> collect2: error: ld returned 1 exit status
> [8/547] Compiling C object 
> 'drivers/a715181@@tmp_rte_net_mlx4@sta/net_mlx4_mlx4_flow.c.o'
> 
> ninja: build stopped: subcommand failed.
> 
> On Thu, Apr 1, 2021 at 4:33 PM Юрий Иванов  <mailto:format_...@outlook.com>> wrote:
> Hi,
> 
> Or you can use nativa rdma driver as written in this post and build vpp as 
> usual. 
> 
> Regards.
> От: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>  <mailto:vpp-dev@lists.fd.io>> от имени Mohammed Hawari  <mailto:moham...@hawari.fr>>
> Отправлено: 1 апреля 2021 г. 13:16
> Кому: chetan bhasin  <mailto:chetan.bhasin...@gmail.com>>
> Копия: vpp-dev mailto:vpp-dev@lists.fd.io>>
> Тема: Re: [vpp-dev] How to enable Mellanox compilation in VPP 21.01
>
> 
> Hi Chetan,
> 
> If you are using CentOS, I’d suggest to cherry-pick 
> https://gerrit.fd.io/r/c/vpp/+/31876 <https://gerrit.fd.io/r/c/vpp/+/31876>. 
> Also please change the dpdk.mk <http://dpdk.mk/> to also set 
> DPDK_MLX5_COMMON_PMD = y. I hope this solves your issue. Otherwise, please 
> consider using the rdma native driver that does not rely on DPDK.
> 
> Regards
> 
> Mohammed
>> On 30 Mar 2021, at 12:41, chetan bhasin > <mailto:chetan.bhasin...@gmail.com>> wrote:
>> 
>> Hello Everyone,
>> 
>> I am upgrading to vpp 2101 . I am facing a compilation issue after enabling 
>> Mellanox compilation in dpdk.mk <http://dpdk.mk/> .
>> 
>> --- a/build/external/packages/dpdk.mk <http://dpdk.mk/>
>> +++ b/build/external/packages/dpdk.mk <http://dpdk.mk/>
>> @@ -14,8 +14,8 @@
>>  DPDK_PKTMBUF_HEA

Re: [vpp-dev] How to enable Mellanox compilation in VPP 21.01

[Mohammed Hawari]

Hi Chetan,

If you are using CentOS, I’d suggest to cherry-pick 
https://gerrit.fd.io/r/c/vpp/+/31876 . 
Also please change the dpdk.mk to also set DPDK_MLX5_COMMON_PMD = y. I hope 
this solves your issue. Otherwise, please consider using the rdma native driver 
that does not rely on DPDK.

Regards

Mohammed
> On 30 Mar 2021, at 12:41, chetan bhasin  wrote:
> 
> Hello Everyone,
> 
> I am upgrading to vpp 2101 . I am facing a compilation issue after enabling 
> Mellanox compilation in dpdk.mk  .
> 
> --- a/build/external/packages/dpdk.mk 
> +++ b/build/external/packages/dpdk.mk 
> @@ -14,8 +14,8 @@
>  DPDK_PKTMBUF_HEADROOM?= 128
>  DPDK_USE_LIBBSD  ?= n
>  DPDK_DEBUG   ?= n
> -DPDK_MLX4_PMD?= n
> -DPDK_MLX5_PMD?= n
> +DPDK_MLX4_PMD?= y
> +DPDK_MLX5_PMD?= y
>  DPDK_MLX5_COMMON_PMD ?= n
> 
> 
> Getting below errors . Anybody please help here ?
> 
> [1344/1846] Compiling C object 
> 'drivers/a715181@@tmp_rte_net_nfp@sta/net_nfp_nfpcore_nfp_nsp_cmds.c.o'
> [1345/1846] Linking target drivers/librte_net_mlx4.so.21.0
> FAILED: drivers/librte_net_mlx4.so.21.0
> cc  -o drivers/librte_net_mlx4.so.21.0 
> 'drivers/a715181@@rte_net_mlx4@sha/meson-generated_.._rte_net_mlx4.pmd.c.o' 
> 'drivers/a715181@@tmp_r
> te_net_mlx4@sta/net_mlx4_mlx4.c.o' 
> 'drivers/a715181@@tmp_rte_net_mlx4@sta/net_mlx4_mlx4_ethdev.c.o' 
> 'drivers/a715181@@tmp_rte_net_mlx4@sta
> /net_mlx4_mlx4_flow.c.o' 
> 'drivers/a715181@@tmp_rte_net_mlx4@sta/net_mlx4_mlx4_intr.c.o' 
> 'drivers/a715181@@tmp_rte_net_mlx4@sta/net_mlx4_ml
> x4_mp.c.o' 'drivers/a715181@@tmp_rte_net_mlx4@sta/net_mlx4_mlx4_mr.c.o' 
> 'drivers/a715181@@tmp_rte_net_mlx4@sta/net_mlx4_mlx4_rxq.c.o' 'dri
> vers/a715181@@tmp_rte_net_mlx4@sta/net_mlx4_mlx4_rxtx.c.o' 
> 'drivers/a715181@@tmp_rte_net_mlx4@sta/net_mlx4_mlx4_txq.c.o' 
> 'drivers/a715181@
> @tmp_rte_net_mlx4@sta/net_mlx4_mlx4_utils.c.o' 
> 'drivers/a715181@@tmp_rte_net_mlx4@sta/net_mlx4_mlx4_glue.c.o' 
> -Wl,--as-needed -Wl,--no-und
> efined -Wl,-O1 -shared -fPIC -Wl,--start-group 
> -Wl,-soname,librte_net_mlx4.so.21 -Wl,--no-as-needed -pthread -lm -ldl -lnuma 
> lib/librte_et
> hdev.so.21.0 lib/librte_eal.so.21.0 lib/librte_kvargs.so.21.0 
> lib/librte_telemetry.so.21.0 lib/librte_net.so.21.0 lib/librte_mbuf.so.21.0
> lib/librte_mempool.so.21.0 lib/librte_ring.so.21.0 lib/librte_meter.so.21.0 
> drivers/librte_bus_pci.so.21.0 lib/librte_pci.so.21.0 drivers/
> librte_bus_vdev.so.21.0 
> -Wl,--version-script=/root/tmp/vpp/build-root/build-vpp-native/external/src-dpdk/drivers/net/mlx4/version.map
>  -lpt
> hread -L/root/tmp/vpp/build-root/install-vpp-native/external/lib64 
> -l:libbnxt_re-rdmav25.a -l:libcxgb4-rdmav25.a -l:libefa.a -l:libhns-rdm
> av25.a -l:libi40iw-rdmav25.a -l:libmlx4.a -l:libmlx5.a -l:libmthca-rdmav25.a 
> -l:libocrdma-rdmav25.a -l:libqedr-rdmav25.a -l:libvmw_pvrdma-rdmav25.a 
> -l:libhfi1verbs-rdmav25.a -l:libipathverbs-rdmav25.a -l:librxe-rdmav25.a 
> -l:libsiw-rdmav25.a -l:libibverbs.a -l:librdma_util.a -l:libccan.a 
> -Wl,--end-group '-Wl,-rpath,$ORIGIN/../lib:$ORIGIN/' 
> -Wl,-rpath-link,/root/tmp/vpp/build-root/build-vpp-native/external/build-dpdk/lib
>  
> -Wl,-rpath-link,/root/tmp/vpp/build-root/build-vpp-native/external/build-dpdk/drivers
> /bin/ld: cannot find -l:libbnxt_re-rdmav25.a
> /bin/ld: cannot find -l:libcxgb4-rdmav25.a
> /bin/ld: cannot find -l:libefa.a
> /bin/ld: cannot find -l:libhns-rdmav25.a
> /bin/ld: cannot find -l:libi40iw-rdmav25.a
> /bin/ld: cannot find -l:libmlx4.a
> /bin/ld: cannot find -l:libmlx5.a
> /bin/ld: cannot find -l:libmthca-rdmav25.a
> /bin/ld: cannot find -l:libocrdma-rdmav25.a
> /bin/ld: cannot find -l:libqedr-rdmav25.a
> /bin/ld: cannot find -l:libvmw_pvrdma-rdmav25.a
> /bin/ld: cannot find -l:libhfi1verbs-rdmav25.a
> /bin/ld: cannot find -l:libipathverbs-rdmav25.a
> /bin/ld: cannot find -l:librxe-rdmav25.a
> /bin/ld: cannot find -l:libsiw-rdmav25.a
> /bin/ld: cannot find -l:libibverbs.a
> /bin/ld: cannot find -l:librdma_util.a
> /bin/ld: cannot find -l:libccan.a
> collect2: error: ld returned 1 exit status
> 
> 
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#19085): https://lists.fd.io/g/vpp-dev/message/19085
Mute This Topic: https://lists.fd.io/mt/81718770/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp crashing on latest master branch with mlx5 enabled

[Mohammed Hawari]

Hi Ashish,

Building DPDK with mlx support in the VPP build system is experimental and not 
well tested (especially with CentOS), as I explained in my previous email. 
Don’t hesitate to contribute patches if you want to improve that support, I’ll 
happily take a look and help.
About your issue, please remove OFED as it might conflict with the rdma-core 
built by the VPP build system. Also, you can try this patch 
https://gerrit.fd.io/r/c/vpp/+/30842  it 
might fix your issue.

Best regards

Mohammed

> On 19 Jan 2021, at 11:20, ashish.sax...@hsc.com wrote:
> 
> Hi Mohammed,
> 
> I am still not able to compile latest master with mlx5 support. Can you 
> please give me some pointers on the compilation ? I have send the compilation 
> error logs in the earlier mail.
> 
> Thanks and Regards,
> Ashish
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18547): https://lists.fd.io/g/vpp-dev/message/18547
Mute This Topic: https://lists.fd.io/mt/78363477/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] DPDK change in link speed convention

[Mohammed Hawari]

Hi everyone,

I agree with opinion #3 as well. It allows us to keep 0 as “unknown" (i.e., not 
set by the driver), and we can specify ~0 as “Not Applicable” (typically the 
case for virtual links) and change format_vnet_hw_interface_link_speed 
accordingly.

Best regards,

Mohammed

> On 18 Jan 2021, at 12:06, Benoit Ganne (bganne) via lists.fd.io 
>  wrote:
> 
> Hi Matt,
> 
> I am adding the list so that maybe VPP DPDK plugin users/maintainers can 
> chime in.
> In my opinion #3 setting link speed to ~0 makes sense as VPP usually use ~0 
> as "invalid".
> 
> The original issue I attempted to fix in DPDK was that the Mellanox DPDK 
> driver was not able to return link status in Azure because it refused to 
> return link status if it could not determine link speed. My original proposed 
> change in DPDK was just for the Mellanox DPDK driver to return link status 
> anyway but not link speed in that case (as eg. Intel drivers do). It ended up 
> being a bit more complicated in the end 
> 
> Best
> ben
> 
>> -Original Message-
>> From: Matthew Smith 
>> Sent: vendredi 15 janvier 2021 21:12
>> To: Benoit Ganne (bganne) 
>> Cc: Jim Thompson 
>> Subject: DPDK change in link speed convention
>> 
>> 
>> Hi Benoit,
>> 
>> i saw a change to DPDK which listed "Suggested-by: Benoit Ganne
>> mailto:bga...@cisco.com> >" -
>> -
>> https://github.com/DPDK/dpdk/commit/810b17d116f03783843d23ac900a3794675f66
>> 16#diff-2e1c01c3a61689ba8e51c0cf41d4d8153870532b8d64ea0847dff7a169e54bef.
>> 
>> Due to this change, it seems that rte_eth_link_get_nowait() can now return
>> ~0 for ports where the link speed is unknown or for virtual links. Along
>> with another change
>> (https://github.com/DPDK/dpdk/commit/b415b7a06856b148ace29ffcc6e9740a12b9e
>> c33#diff-c943f71001a82a273432fb087f7cd461d9666ee5cfd58a0b1b2b93b68d9f6234)
>> this has resulted in VPP reporting the link speed of DPDK virtio
>> interfaces as 4294966296 kbps. This is because the VPP DPDK plugin calls
>> vnet_hw_interface_set_link_speed() with the link speed value retrieved by
>> rte_eth_link_get_nowait() multiplied by 1000 to convert Mbps to kbps. When
>> rte_eth_link_get_nowait() returns a link speed of ~0, multiplying it by
>> 1000 overflows a u32 and the value that is set on the vnet_hw_interface_t
>> is 4294966296.
>> 
>> It seems like something different should happen in the VPP DPDK plugin
>> when the link speed is reported by DPDK as ~0. Maybe one of these options
>> would be good:
>> 
>> 1. Don't call vnet_hw_interface_set_link_speed() at all in this case. If
>> the current link speed is being reported as "unknown", maybe whatever
>> value is set on the VPP hw interface should be left as it is.
>> 2. Call vnet_hw_interface_set_link_speed() with a link speed of 0. The
>> default link speed value of a new vnet_hw_interface_t is 0, so 0 could be
>> viewed as the de facto VPP version of "unknown".
>> 3. Call vnet_hw_interface_set_link_speed() with a link speed of ~0 rather
>> than (~0 * 1000) as is currently done. And agree/document that this value
>> means "unknown" in VPP.
>> 
>> Since the DPDK commit that announced this change had a Suggested-by header
>> that mentioned you, I wanted to find out if you had thoughts on how this
>> should be handled or if you already had some work in progress to deal with
>> this.
>> 
>> Thanks!
>> -Matt
> 
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18534): https://lists.fd.io/g/vpp-dev/message/18534
Mute This Topic: https://lists.fd.io/mt/79920343/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp unable to send or receive packets

[Mohammed Hawari]

Hi Ashish,

Given that -12 corresponds to -ENOMEM in the DPDK source code, I would suggest 
to increase the number of buffers allocated, via the buffers-per-numa option in 
startup.conf.

Regards

Mohammed

> On 11 Jan 2021, at 15:27, ashish.sax...@hsc.com wrote:
> 
> Hi All,
> 
> We are using vpp-20.05-1 version on Centos 8.2 machine. When we are 
> increasing the number of rx and tx descriptors in the startup.conf file as 
> follows:
> ## Number of descriptors in transmit and receive rings
> ## increasing or reducing number can impact performance
> ## Default is 1024 for both rx and tx
> num-rx-desc 4096
> num-tx-desc 4096
> 
> Then vpp is unable to receive to send any packets.
> 
> Following is the output of vpp error and show interface command:
> vpp# show errors
>CountNode  Reason
>  1tuntap-rx   unknown packet type
> 15ip6-glean   neighbor solicitations sent
>  7ip4-glean   ARP requests sent
> 46ip6-input   drops due to concurrent 
> reassemblies limit
>  6 ip6-icmp-input router advertisements sent
> 11 HundredGigabitEthernet12/0/0-txTx packet drops (dpdk tx 
> failure)
> 19 HundredGigabitEthernet12/0/1-txTx packet drops (dpdk tx 
> failure)
>
> Getting error  rte_eth_dev_start[port:0, errno:-12]: Unknown error -12 in the 
>  show hardware-interfaces command as follows:
> vpp# show hardware-interfaces
>   NameIdx   Link  Hardware
> HundredGigabitEthernet12/0/0   2 up   HundredGigabitEthernet12/0/0
>   Link speed: 100 Gbps
>   Ethernet address b8:83:03:9e:68:4c
>   Mellanox ConnectX-4 Family
> carrier up full duplex mtu 9206
> flags: admin-up pmd maybe-multiseg subif rx-ip4-cksum
> rx: queues 4 (max 1024), desc 4096 (min 0 max 65535 align 1)
> tx: queues 4 (max 1024), desc 4096 (min 0 max 65535 align 1)
> pci: device 15b3:1013 subsystem 1590:00c8 address :12:00.00 numa 0
> switch info: name :12:00.0 domain id 0 port id 65535
> max rx packet len: 65536
> promiscuous: unicast off all-multicast off
> vlan offload: strip off filter off qinq off
> rx offload avail:  vlan-strip ipv4-cksum udp-cksum tcp-cksum vlan-filter
>jumbo-frame scatter timestamp keep-crc rss-hash
> rx offload active: ipv4-cksum udp-cksum tcp-cksum jumbo-frame scatter
> tx offload avail:  vlan-insert ipv4-cksum udp-cksum tcp-cksum tcp-tso
>outer-ipv4-cksum vxlan-tnl-tso gre-tnl-tso 
> geneve-tnl-tso
>multi-segs udp-tnl-tso ip-tnl-tso
> tx offload active: multi-segs
> rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv4 ipv6-tcp-ex
>ipv6-udp-ex ipv6-frag ipv6-tcp ipv6-udp ipv6-other
>ipv6-ex ipv6 l4-dst-only l4-src-only l3-dst-only 
> l3-src-only
> rss active:ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv4 ipv6-tcp-ex
>ipv6-udp-ex ipv6-frag ipv6-tcp ipv6-udp ipv6-other
>ipv6-ex ipv6
> tx burst function: removed_tx_burst
> rx burst function: removed_rx_burst
>
> extended stats:
>   rx port unicast packets  2
>   rx port unicast bytes  172
>   tx port unicast packets  2
>   tx port unicast bytes  172
>   rx port multicast packets10143
>   rx port multicast bytes1120826
>   tx port multicast packets   24
>   tx port multicast bytes   2408
>   rx port broadcast packets   41
>   rx port broadcast bytes  13735
>   tx port broadcast packets7
>   tx port broadcast bytes   1795
>   tx packets phy  36
>   rx packets phy   10186
>   tx bytes phy  4951
>   rx bytes phy   1175477  Errors:
> rte_eth_dev_start[port:0, errno:-12]: Unknown error -12
>
> HundredGigabitEthernet12/0/1   3 up   HundredGigabitEthernet12/0/1
>   Link speed: 100 Gbps
>   Ethernet address b8:83:03:9e:68:4d
>   Mellanox ConnectX-4 Family
> carrier up full duplex mtu 9206
> flags: admin-up pmd maybe-multiseg subif rx-ip4-cksum
> rx: queues 4 (max 1024), desc 4096 (min 0 max 65535 align 1)
> tx: queues 4 (max 1024), desc 4096 (min 0 max 65535 align 1)
> pci: device 15b3:1013 subsystem 1590:00c8 address :12:00.01 numa 0
> switch 

Re: [vpp-dev] vpp crashing on latest master branch with mlx5 enabled

[Mohammed Hawari]

Hi Ashish,

Support for the DPDK plugin with Mellanox cards has been restored on master 
since https://gerrit.fd.io/r/c/vpp/+/30051 
. However, I can’t stress enough that you 
should get better performance (and more stability) with the rdma plugin.
Due to the DPDK build itself relying on libibverbs for mlx support, ensuring a 
consistent build of DPDK, rdma-core, and VPP is a bit challenging, and may 
break upon build system evolutions (this is what happened when we moved to 
meson-based build for DPDK). Long story short, please use the rdma plugin if 
you can, it has better support in VPP than DPDK-MLX :)

Best regards

Mohammed


> On 7 Jan 2021, at 12:54, ashish.sax...@hsc.com wrote:
> 
> Hi Mohammed,
> 
> Thanks for the reply.
> I will try with rdma plugin, But I wants to know why support for DPDK plugin 
> with Mellanox cards has been discontinued ? It was working perfectly till 
> release 20.05 .
> 
> Thanks and Regards,
> Ashish 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18483): https://lists.fd.io/g/vpp-dev/message/18483
Mute This Topic: https://lists.fd.io/mt/78363477/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp crashing on latest master branch with mlx5 enabled

[Mohammed Hawari]

Hi Ashish,

The DPDK plugin for mlx5 NICs is not supported by the current code on master. 
We strongly encourage you to try the rdma plugin, which offers a native VPP 
driver for mlx5 NICs without relying on DPDK. The use of this driver is 
documented in  src/plugins/rdma/rdma_doc.md.

Best regards

Mohammed

> On 19 Nov 2020, at 13:18, ashish.sax...@hsc.com wrote:
> 
> Hi vpp devs,
> 
> I have been trying to run vpp on latest master branch with mlx 5 enables on 
> centos 8.2 machine .
> I have downloaded and installed the mellanox driver from 
> url:https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed and 
> installed the following dependencies:
> #yum install perl-Term-ANSIColor
> #yum install tcl tk python36
> #./mlnxofedinstall --upstream-libs --dpdk --without-fw-update
> #/etc/init.d/openibd restart
> 
> I cloned the latest master branch (v21.01-rc0) (git clone -b master 
> https://gerrit.fd.io/r/vpp)
> I enabled mlx5 from file:build/external/packages/dpdk.mk
> DPDK_MLX5_PMD?= y
> DPDK_MLX5_COMMON_PMD ?= y
> 
> Then I followed these steps to compile vpp :
> 
> #make wipe-release
> #make install-dep
> #make install-ext-deps
> #make build-release
> # make pkg-rpm
> #rpm -ivh *.rpm
> 
> The compilation went fine , but got the following issue while installing the 
> rpm files:
> #rpm -ivh *.rpm
> error: Failed dependencies:
>  python(abi) = 2.7 is needed by vpp-api-python-21.01-rc0~352_g1f85dad1e.x86_64
>  python-setuptools is needed by vpp-api-python-21.01-rc0~352_g1f85dad1e.x86_64
> 
> I changed line 143 on extras/rpm/vpp.spec: From "python-setuptools" to " 
> python3-setuptools".
> Then , I  proceeded with re-compile, and was able to install vpp successfully 
> after that.
> 
> But when starting vpp, it kept crashing with error below:
>
> Nov  6 15:01:44 sim01 systemd[1]: Starting Vector Packet Processing Process...
> Nov  6 15:01:44 sim01 systemd[1]: Started Vector Packet Processing Process.
> Nov  6 15:01:44 sim01 vpp[1557233]: /usr/bin/vpp: symbol lookup error: 
> /usr/lib/vpp_plugins/dpdk_plugin.so: undefined symbol: ibv_fork_init
> Nov  6 15:01:44 sim01 systemd[1]: vpp.service: Main process exited, 
> code=exited, status=127/n/a
> Nov  6 15:01:44 sim01 systemd[1]: vpp.service: Failed with result 'exit-code'.
> Nov  6 15:01:48 sim01 systemd[1]: Stopped Vector Packet Processing Process.
> 
> How can I compile vpp with mlx so that it didn't crash .
> 
> Thanks and Regards,
> Ashish
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18113): https://lists.fd.io/g/vpp-dev/message/18113
Mute This Topic: https://lists.fd.io/mt/78363477/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Latest master, dpdk and mlx5 failing

[Mohammed HAWARI]

Hello Chris, Matthew,

Thanks for raising that issue. Just to be clear and better understand, does the 
problem occur with the default config, i.e., without trying to compile any MLX 
driver in DPDK? Or does it only appear when setting DPDK_MLX5_PMD=y ?
Thanks
Best regards
Mohammed
> On 1 Oct 2020, at 22:33, Matthew Smith via lists.fd.io 
>  wrote:
> 
> Hi Chris,
> 
> I did this in my local build:
> 
> diff --git a/build/external/packages/dpdk.mk  
> b/build/external/packages/dpdk.mk 
> index 49761cd56..a30ffd2ac 100644
> --- a/build/external/packages/dpdk.mk 
> +++ b/build/external/packages/dpdk.mk 
> @@ -139,6 +139,7 @@ DPDK_MESON_ARGS = \
> -Dtests=false \
> "-Ddisable_drivers=$(DPDK_DRIVERS_DISABLED)" \
> "-Ddisable_libs=$(DPDK_LIBS_DISABLED)" \
> +   -Dibverbs_link=dlopen \
> -Db_pie=true \
> -Dmachine=$(DPDK_MACHINE) \
> --buildtype=$(DPDK_BUILD_TYPE) 
> 
> If I try to submit it upstream, I would probably do something nicer using a 
> DPDK_ environment variable, but for the moment this got me past 
> that error. I have not actually tested with an mlx5 device yet, so I don't 
> know if something additional will be required in order to forward packets via 
> an mlx5 NIC, but it did fix the error you pasted and allow dpdk_plugin.so to 
> be loaded.
> 
> -Matt
> 
> 
> On Thu, Oct 1, 2020 at 2:10 PM Christian Hopps  > wrote:
> I've rebased my local branch on the latest master and dpdk is failing to load 
> now b/c
> 
> 2020/10/01 18:31:54:514 errplugin/load
> /home/chopps/w/vpp/build-root/install-vpp-native/vpp/lib/vpp_plugins/dpdk_plugin.so:
>  undefined symbol: ibv_fork_init
> 
> I noticed that the dpdk build system has been changed is there something I 
> need to do to get it to link properly with the required libraries now?
> 
> I did change "n" in
> 
>   DPDK_MLX5_PMD?= n
>   DPDK_MLX5_COMMON_PMD ?= n
> 
> to "y" to try and get MLX5 PMD to build.
> 
> I also added
> 
>   vpp_uses_dpdk_mlx5_pmd = yes
> 
> to build-data/platforms/vpp.mk 
> 
> I also tried adding:
> 
>   vpp_uses_dpdk_ibverbs_link_dlopen = yes
> 
> which didn't fix the problem.
> 
> Any suggestions on how to fix this?
> 
> Thanks,
> Chris.
> 
> 
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#17628): https://lists.fd.io/g/vpp-dev/message/17628
Mute This Topic: https://lists.fd.io/mt/77247865/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Packet processing time.

[Mohammed Hawari]

Hi Chris,

Evaluating packet processing time in software is a very challenging issue, as 
mentioned by Dave, it is likely to impact the performance we are trying to 
evaluate. I worked on that issue and have an unpublished, under review, 
academic paper proposing a solution using the NetFPGA-SUME platform. Basically, 
I built a custom FPGA design, mimicking a NIC capable of timestamping every 
packets at the ingress and the egress (immediately after packets arrivals from 
the wire, and immediately before the packets departures on the wire). I also 
wrote a DPDK driver for that NIC, and made it work with VPP, so that the 
latency introduced by (VPP+PCI-based DMA) can be evaluated. I played with this 
design and VPP in various configurations (l2-patch l2 crossconnect and l3 
forward) and I think it could be an interesting tool to diagnose latency issues 
on a “per-packet” basis.
Downside is, of course, from the perspective of VPP, this is a custom NIC, with 
a custom driver (not necessarily super-optimised), and the evaluated packet 
forwarding latency takes the driver’s performance into account.

If you are interested in discussing this work, I can give you more details and 
resources in unicast, don’t hesitate to contact me :)

Cheers,

Mohammed Hawari
Software Engineer & PhD student
Cisco Systems


> On 18 Apr 2020, at 22:14, Dave Barach via lists.fd.io 
>  wrote:
> 
> If you turn on the main loop dispatch event logs and look at the results in 
> the g2 viewer [or dump them in ascii] you can make pretty accurate lap time 
> estimates for any workload. Roughly speaking, packets take 1 lap time to 
> arrive and then leave. 
>
> The “circuit-node ” game produces one elog event per frame, so you 
> can look at several million frame circuit times.
>
> Individually timestamping packets would be more precise, but calling 
> clib_cpu_time_now(...) (rdtsc instrs on x86_64) twice per packet would almost 
> certainly affect forwarding performance.
>
> See https://fd.io/docs/vpp/master/gettingstarted/developers/eventviewer.html 
> <https://fd.io/docs/vpp/master/gettingstarted/developers/eventviewer.html>
>
> /*?
> * Control event logging of api, cli, and thread barrier events
> * With no arguments, displays the current trace status.
> * Name the event groups you wish to trace or stop tracing.
> *
> * @cliexpar
> * @clistart
> * elog trace api cli barrier
> * elog trace api cli barrier disable
> * elog trace dispatch
> * elog trace circuit-node ethernet-input
> * elog trace
> * @cliend
> * @cliexcmd{elog trace [api][cli][barrier][disable]}
> ?*/
> /* *INDENT-OFF* */
>
> From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>  <mailto:vpp-dev@lists.fd.io>> On Behalf Of Christian Hopps
> Sent: Saturday, April 18, 2020 3:14 PM
> To: vpp-dev mailto:vpp-dev@lists.fd.io>>
> Cc: Christian Hopps mailto:cho...@chopps.org>>
> Subject: [vpp-dev] Packet processing time.
>
> The recent discussion on reference counting and barrier timing has got me 
> interested in packet processing time. I realize there's a way to use "show 
> runtime" along with knowledge of the arc a packet follows, but I'm curious if 
> something more straight-forward has been attempted where packets are 
> timestamped on ingress (or creation) and stats are collected on egress 
> (transmission)?
> 
> I also have an unrelated interest in hooking into the graph 
> immediate-post-transmission -- I'd like to adjust an input queue size only 
> when the packet that enqueued on it is actually transmitted on the wire, and 
> not just handed off downstream on the arc -- this would be a likely the same 
> place packet stat collection might occur. :)
> 
> Thanks,
> Chris.
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16123): https://lists.fd.io/g/vpp-dev/message/16123
Mute This Topic: https://lists.fd.io/mt/73114130/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-