Re: [vpp-dev] VPP on a Bluefield-2 smartNIC
As shown below, running the test echo server on the Bluefield-2 smartNIC fails: ``` DBGvpp# test echo server ssvm_server_init_memfd:258: memfd map (fd 27): File exists (errno 17) session_vpp_event_queues_allocate:1528: failed to initialize queue segment echo_server_create_command_fn:532: No uri provided! Using default: tcp://0.0.0.0/1234 DBGvpp# Aborted make: *** [Makefile:519: run] Error 134 ``` The mmap error (file exists) appears when MAP_FIXED_NOREPLACE was specified in flags, and the range covered by addr and length clashes with an existing mapping, which is the case in clib_mem_vm_reserve() in src/vppinfra/linux/mem.c:355. The base address of this mmap is derived from the HIGH_SEGMENT_BASEVA macro defined in src/svm/svm_common.h. By default its value is 128 << 30 == 0x20 on a 64-bits system. |Doing info proc mappings in gdb tells me that ||indeed ||this address is an already mapped range: ``` Start Addr End Addr Size Offset objfile 0x198400 0x218400 0x8 0x0 ``` What is the logic behind this value? To which value should I set it instead? Best Pierre Louis | On 2021/07/02 19:54, Pierre Louis Aublin wrote: I do not have a dedicated Bluefield-2 at my disposal; instead, I am using one in a shared cluster, chosen at random by the cluster job scheduler. Anyway, today I tried on a different one which has the Mellanox OFED driver installed. I first had to apply this patch so that VPP compiles: https://gerrit.fd.io/r/c/vpp/+/30842 . My startup.conf is: ``` unix { interactive cli-listen /run/vpp/cli.sock gid 0 } dpdk { dev :03:00.0 } plugins { plugin dpdk_plugin.so { enable } ``` This time I could see and configure the network interface, but VPP crashed when launching the echo server: ``` $ sudo -E make run dpdk [warn ]: not enough DPDK crypto resources dpdk/cryptodev [warn ]: dpdk_cryptodev_init: Not enough cryptodevs vat-plug/load [error ]: vat_plugin_register: oddbuf plugin not loaded... ___ _ _ _ ___ __/ __/ _ \ (_)__ | | / / _ \/ _ \ _/ _// // / / / _ \ | |/ / ___/ ___/ /_/ /(_)_/\___/ |___/_/ /_/ DBGvpp# sh int Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count HundredGigabitEthernet3/0/0 1 down 9000/0/0/0 local0 0 down 0/0/0/0 DBGvpp# set interface ip address HundredGigabitEthernet3/0/0 192.168.10.1/24 DBGvpp# set interface state HundredGigabitEthernet3/0/0 up DBGvpp# sh int addr HundredGigabitEthernet3/0/0 (up): L3 192.168.10.1/24 local0 (dn): DBGvpp# test echo server ssvm_server_init_memfd:258: memfd map (fd 27): File exists (errno 17) session_vpp_event_queues_allocate:1528: failed to initialize queue segment echo_server_create_command_fn:532: No uri provided! Using default: tcp://0.0.0.0/1234 DBGvpp# Aborted make: *** [Makefile:519: run] Error 134 ``` Please find attached the output of `sh log`. Best Pierre Louis On 2021/07/01 18:38, Benoit Ganne (bganne) wrote: What is your VPP dpdk config looks like in startup.conf? Esp. did you whitelist the device? See https://fd.io/docs/vpp/master/gettingstarted/users/configuring/startup.html#the-dpdk-section Also, please share the output of 'show logs'. Best ben -Original Message- From: Pierre Louis Aublin Sent: jeudi 1 juillet 2021 11:08 To: Benoit Ganne (bganne) ; vpp-dev@lists.fd.io Subject: Re: [vpp-dev] VPP on a Bluefield-2 smartNIC The"Unsupported PCI device 0x15b3:0xa2d6 found at PCI address :03:00.0" message disappears; however the network interface still doesn't show up. Interestingly, vpp on the host also prints this message, yet the interface can be used. By any chance, would you have any clue on what I could try to further debug this issue? Best Pierre Louis On 2021/07/01 17:50, Benoit Ganne (bganne) via lists.fd.io wrote: Please try https://gerrit.fd.io/r/c/vpp/+/32965 and reports if it works. Best ben -Original Message- From: vpp-dev@lists.fd.io On Behalf Of Pierre Louis Aublin Sent: jeudi 1 juillet 2021 07:36 To: vpp-dev@lists.fd.io Subject: [vpp-dev] VPP on a Bluefield-2 smartNIC Dear VPP developers I would like to run VPP on the Bluefield-2 smartNIC, but even though I managed to compile it the interface doesn't show up inside the CLI. By any chance, would you know how to compile and configure vpp for this device? I am using VPP v21.06-rc2 and did the following modifications so that it can compile: ``` diff --git a/build/external/packages/dpdk.mk b/build/external/packages/dpdk.mk index c7eb0fc3f..31a5c764e 100644 --- a/build/external/packages/dpdk.mk +++ b/build/external/packages/dpdk.mk @@ -15,8 +15,8 @@ DPDK_PKTMBUF_HEADROOM ?= 128 DPDK_USE_LIBBSD ?= n DPDK_DEBUG ?= n DPDK_MLX4_PMD ?= n -DPDK_MLX5_PMD ?= n -DPDK_MLX5_COMMON_PMD ?=
Re: [vpp-dev] VPP on a Bluefield-2 smartNIC
I do not have a dedicated Bluefield-2 at my disposal; instead, I am using one in a shared cluster, chosen at random by the cluster job scheduler. Anyway, today I tried on a different one which has the Mellanox OFED driver installed. I first had to apply this patch so that VPP compiles: https://gerrit.fd.io/r/c/vpp/+/30842 . My startup.conf is: ``` unix { interactive cli-listen /run/vpp/cli.sock gid 0 } dpdk { dev :03:00.0 } plugins { plugin dpdk_plugin.so { enable } ``` This time I could see and configure the network interface, but VPP crashed when launching the echo server: ``` $ sudo -E make run dpdk [warn ]: not enough DPDK crypto resources dpdk/cryptodev [warn ]: dpdk_cryptodev_init: Not enough cryptodevs vat-plug/load [error ]: vat_plugin_register: oddbuf plugin not loaded... ___ _ _ _ ___ __/ __/ _ \ (_)__ | | / / _ \/ _ \ _/ _// // / / / _ \ | |/ / ___/ ___/ /_/ /(_)_/\___/ |___/_/ /_/ DBGvpp# sh int Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count HundredGigabitEthernet3/0/0 1 down 9000/0/0/0 local0 0 down 0/0/0/0 DBGvpp# set interface ip address HundredGigabitEthernet3/0/0 192.168.10.1/24 DBGvpp# set interface state HundredGigabitEthernet3/0/0 up DBGvpp# sh int addr HundredGigabitEthernet3/0/0 (up): L3 192.168.10.1/24 local0 (dn): DBGvpp# test echo server ssvm_server_init_memfd:258: memfd map (fd 27): File exists (errno 17) session_vpp_event_queues_allocate:1528: failed to initialize queue segment echo_server_create_command_fn:532: No uri provided! Using default: tcp://0.0.0.0/1234 DBGvpp# Aborted make: *** [Makefile:519: run] Error 134 ``` Please find attached the output of `sh log`. Best Pierre Louis On 2021/07/01 18:38, Benoit Ganne (bganne) wrote: What is your VPP dpdk config looks like in startup.conf? Esp. did you whitelist the device? See https://fd.io/docs/vpp/master/gettingstarted/users/configuring/startup.html#the-dpdk-section Also, please share the output of 'show logs'. Best ben -Original Message- From: Pierre Louis Aublin Sent: jeudi 1 juillet 2021 11:08 To: Benoit Ganne (bganne) ; vpp-dev@lists.fd.io Subject: Re: [vpp-dev] VPP on a Bluefield-2 smartNIC The"Unsupported PCI device 0x15b3:0xa2d6 found at PCI address :03:00.0" message disappears; however the network interface still doesn't show up. Interestingly, vpp on the host also prints this message, yet the interface can be used. By any chance, would you have any clue on what I could try to further debug this issue? Best Pierre Louis On 2021/07/01 17:50, Benoit Ganne (bganne) via lists.fd.io wrote: Please try https://gerrit.fd.io/r/c/vpp/+/32965 and reports if it works. Best ben -Original Message- From: vpp-dev@lists.fd.io On Behalf Of Pierre Louis Aublin Sent: jeudi 1 juillet 2021 07:36 To: vpp-dev@lists.fd.io Subject: [vpp-dev] VPP on a Bluefield-2 smartNIC Dear VPP developers I would like to run VPP on the Bluefield-2 smartNIC, but even though I managed to compile it the interface doesn't show up inside the CLI. By any chance, would you know how to compile and configure vpp for this device? I am using VPP v21.06-rc2 and did the following modifications so that it can compile: ``` diff --git a/build/external/packages/dpdk.mk b/build/external/packages/dpdk.mk index c7eb0fc3f..31a5c764e 100644 --- a/build/external/packages/dpdk.mk +++ b/build/external/packages/dpdk.mk @@ -15,8 +15,8 @@ DPDK_PKTMBUF_HEADROOM ?= 128 DPDK_USE_LIBBSD ?= n DPDK_DEBUG ?= n DPDK_MLX4_PMD ?= n -DPDK_MLX5_PMD ?= n -DPDK_MLX5_COMMON_PMD ?= n +DPDK_MLX5_PMD ?= y +DPDK_MLX5_COMMON_PMD ?= y DPDK_TAP_PMD ?= n DPDK_FAILSAFE_PMD ?= n DPDK_MACHINE ?= default diff --git a/build/external/packages/ipsec-mb.mk b/build/external/packages/ipsec-mb.mk index d0bd2af19..119eb5219 100644 --- a/build/external/packages/ipsec-mb.mk +++ b/build/external/packages/ipsec-mb.mk @@ -34,7 +34,7 @@ define ipsec-mb_build_cmds SAFE_DATA=n \ PREFIX=$(ipsec-mb_install_dir) \ NASM=$(ipsec-mb_install_dir)/bin/nasm \ - EXTRA_CFLAGS="-g -msse4.2" > $(ipsec-mb_build_log) + EXTRA_CFLAGS="-g" > $(ipsec-mb_build_log) endef define ipsec-mb_install_cmds ``` However, when running the VPP CLI, the network interface does not show up: ``` $ sudo -E make run clib_sysfs_prealloc_hugepages:261: pre-allocating 6 additional 2048K hugepages on numa node 0 dpdk [warn ]: Unsupported PCI device 0x15b3:0xa2d6 found at PCI address :03:00.0 dpdk/cryptodev [warn ]: dpdk_cryptodev_init: Failed to configure cryptodev vat-plug/load [error ]: vat_plugi
Re: [vpp-dev] VPP on a Bluefield-2 smartNIC
Indeed, it is not needed. I previously removed it to compile on an old Intel CPU without sse4.2 and kept it for the Bluefield. Thank you On 2021/07/01 20:45, Damjan Marion wrote: On 01.07.2021., at 07:35, Pierre Louis Aublin wrote: diff --git a/build/external/packages/ipsec-mb.mk b/build/external/packages/ipsec-mb.mk index d0bd2af19..119eb5219 100644 --- a/build/external/packages/ipsec-mb.mk +++ b/build/external/packages/ipsec-mb.mk @@ -34,7 +34,7 @@ define ipsec-mb_build_cmds SAFE_DATA=n \ PREFIX=$(ipsec-mb_install_dir) \ NASM=$(ipsec-mb_install_dir)/bin/nasm \ - EXTRA_CFLAGS="-g -msse4.2" > $(ipsec-mb_build_log) + EXTRA_CFLAGS="-g" > $(ipsec-mb_build_log) Why do you need this change? If i get it right bluefield uses ARM cpus and we don’t compile intel ipsecmb lib on arm. $ git grep ARCH_X86_64 build/external/Makefile build/external/Makefile:ARCH_X86_64=$(filter x86_64,$(shell uname -m)) build/external/Makefile:install: $(if $(ARCH_X86_64), nasm-install ipsec-mb-install) dpdk-install rdma-core-install quicly-install libbpf-install build/external/Makefile:config: $(if $(ARCH_X86_64), nasm-config ipsec-mb-config) dpdk-config rdma-core-config quicly-build — Damjan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19692): https://lists.fd.io/g/vpp-dev/message/19692 Mute This Topic: https://lists.fd.io/mt/83910198/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] VPP on a Bluefield-2 smartNIC
> On 01.07.2021., at 07:35, Pierre Louis Aublin > wrote: > > diff --git a/build/external/packages/ipsec-mb.mk > b/build/external/packages/ipsec-mb.mk > index d0bd2af19..119eb5219 100644 > --- a/build/external/packages/ipsec-mb.mk > +++ b/build/external/packages/ipsec-mb.mk > @@ -34,7 +34,7 @@ define ipsec-mb_build_cmds > SAFE_DATA=n \ > PREFIX=$(ipsec-mb_install_dir) \ > NASM=$(ipsec-mb_install_dir)/bin/nasm \ > - EXTRA_CFLAGS="-g -msse4.2" > $(ipsec-mb_build_log) > + EXTRA_CFLAGS="-g" > $(ipsec-mb_build_log) Why do you need this change? If i get it right bluefield uses ARM cpus and we don’t compile intel ipsecmb lib on arm. $ git grep ARCH_X86_64 build/external/Makefile build/external/Makefile:ARCH_X86_64=$(filter x86_64,$(shell uname -m)) build/external/Makefile:install: $(if $(ARCH_X86_64), nasm-install ipsec-mb-install) dpdk-install rdma-core-install quicly-install libbpf-install build/external/Makefile:config: $(if $(ARCH_X86_64), nasm-config ipsec-mb-config) dpdk-config rdma-core-config quicly-build — Damjan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19682): https://lists.fd.io/g/vpp-dev/message/19682 Mute This Topic: https://lists.fd.io/mt/83910198/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] VPP on a Bluefield-2 smartNIC
Might be worth trying our native driver (rdma) instead of using dpdk….. — Damjan > On 01.07.2021., at 11:07, Pierre Louis Aublin > wrote: > > The"Unsupported PCI device 0x15b3:0xa2d6 found at PCI address :03:00.0" > message disappears; however the network interface still doesn't show up. > Interestingly, vpp on the host also prints this message, yet the interface > can be used. > > By any chance, would you have any clue on what I could try to further debug > this issue? > > Best > Pierre Louis > > On 2021/07/01 17:50, Benoit Ganne (bganne) via lists.fd.io wrote: >> Please try https://gerrit.fd.io/r/c/vpp/+/32965 and reports if it works. >> >> Best >> ben >> >>> -Original Message- >>> From: vpp-dev@lists.fd.io On Behalf Of Pierre Louis >>> Aublin >>> Sent: jeudi 1 juillet 2021 07:36 >>> To: vpp-dev@lists.fd.io >>> Subject: [vpp-dev] VPP on a Bluefield-2 smartNIC >>> >>> Dear VPP developers >>> >>> I would like to run VPP on the Bluefield-2 smartNIC, but even though I >>> managed to compile it the interface doesn't show up inside the CLI. By >>> any chance, would you know how to compile and configure vpp for this >>> device? >>> >>> I am using VPP v21.06-rc2 and did the following modifications so that it >>> can compile: >>> ``` >>> diff --git a/build/external/packages/dpdk.mk >>> b/build/external/packages/dpdk.mk >>> index c7eb0fc3f..31a5c764e 100644 >>> --- a/build/external/packages/dpdk.mk >>> +++ b/build/external/packages/dpdk.mk >>> @@ -15,8 +15,8 @@ DPDK_PKTMBUF_HEADROOM?= 128 >>> DPDK_USE_LIBBSD ?= n >>> DPDK_DEBUG ?= n >>> DPDK_MLX4_PMD?= n >>> -DPDK_MLX5_PMD?= n >>> -DPDK_MLX5_COMMON_PMD ?= n >>> +DPDK_MLX5_PMD?= y >>> +DPDK_MLX5_COMMON_PMD ?= y >>> DPDK_TAP_PMD ?= n >>> DPDK_FAILSAFE_PMD?= n >>> DPDK_MACHINE ?= default >>> diff --git a/build/external/packages/ipsec-mb.mk >>> b/build/external/packages/ipsec-mb.mk >>> index d0bd2af19..119eb5219 100644 >>> --- a/build/external/packages/ipsec-mb.mk >>> +++ b/build/external/packages/ipsec-mb.mk >>> @@ -34,7 +34,7 @@ define ipsec-mb_build_cmds >>>SAFE_DATA=n \ >>>PREFIX=$(ipsec-mb_install_dir) \ >>>NASM=$(ipsec-mb_install_dir)/bin/nasm \ >>> - EXTRA_CFLAGS="-g -msse4.2" > $(ipsec-mb_build_log) >>> + EXTRA_CFLAGS="-g" > $(ipsec-mb_build_log) >>> endef >>> >>> define ipsec-mb_install_cmds >>> ``` >>> >>> >>> However, when running the VPP CLI, the network interface does not show up: >>> ``` >>> $ sudo -E make run >>> clib_sysfs_prealloc_hugepages:261: pre-allocating 6 additional 2048K >>> hugepages on numa node 0 >>> dpdk [warn ]: Unsupported PCI device 0x15b3:0xa2d6 found >>> at PCI address :03:00.0 >>> >>> dpdk/cryptodev [warn ]: dpdk_cryptodev_init: Failed to configure >>> cryptodev >>> vat-plug/load [error ]: vat_plugin_register: oddbuf plugin not >>> loaded... >>> _____ _ ___ >>> __/ __/ _ \ (_)__| | / / _ \/ _ \ >>> _/ _// // / / / _ \ | |/ / ___/ ___/ >>> /_/ /(_)_/\___/ |___/_/ /_/ >>> >>> DBGvpp# show int >>>Name IdxState MTU >>> (L3/IP4/IP6/MPLS) Counter Count >>> local00 down 0/0/0/0 >>> DBGvpp# sh hard >>>NameIdx Link Hardware >>> local0 0down local0 >>>Link speed: unknown >>>local >>> ``` >>> >>> >>> The dpdk-testpmd application seems to start correctly though: >>> ``` >>> $ sudo ./build-root/install-vpp_debug-native/external/bin/dpdk-testpmd >>> -l 0-2 -a :03:00.00 -- -i --nb-cores=2 --nb-ports=1 >>> --total-num-mbufs=2048 >>> EAL: Detected 8 lcore(s) >>> EAL: Detected 1 NUMA nodes >>> EAL: Detected static linkage of DPDK >>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket >>> EAL: Selected IOVA mode 'VA' >>> EAL: No available 32768 kB hug
Re: [vpp-dev] VPP on a Bluefield-2 smartNIC
What is your VPP dpdk config looks like in startup.conf? Esp. did you whitelist the device? See https://fd.io/docs/vpp/master/gettingstarted/users/configuring/startup.html#the-dpdk-section Also, please share the output of 'show logs'. Best ben > -Original Message- > From: Pierre Louis Aublin > Sent: jeudi 1 juillet 2021 11:08 > To: Benoit Ganne (bganne) ; vpp-dev@lists.fd.io > Subject: Re: [vpp-dev] VPP on a Bluefield-2 smartNIC > > The"Unsupported PCI device 0x15b3:0xa2d6 found at PCI address > :03:00.0" message disappears; however the network interface still > doesn't show up. Interestingly, vpp on the host also prints this > message, yet the interface can be used. > > By any chance, would you have any clue on what I could try to further > debug this issue? > > Best > Pierre Louis > > On 2021/07/01 17:50, Benoit Ganne (bganne) via lists.fd.io wrote: > > Please try https://gerrit.fd.io/r/c/vpp/+/32965 and reports if it works. > > > > Best > > ben > > > >> -Original Message- > >> From: vpp-dev@lists.fd.io On Behalf Of Pierre > Louis > >> Aublin > >> Sent: jeudi 1 juillet 2021 07:36 > >> To: vpp-dev@lists.fd.io > >> Subject: [vpp-dev] VPP on a Bluefield-2 smartNIC > >> > >> Dear VPP developers > >> > >> I would like to run VPP on the Bluefield-2 smartNIC, but even though I > >> managed to compile it the interface doesn't show up inside the CLI. By > >> any chance, would you know how to compile and configure vpp for this > >> device? > >> > >> I am using VPP v21.06-rc2 and did the following modifications so that > it > >> can compile: > >> ``` > >> diff --git a/build/external/packages/dpdk.mk > >> b/build/external/packages/dpdk.mk > >> index c7eb0fc3f..31a5c764e 100644 > >> --- a/build/external/packages/dpdk.mk > >> +++ b/build/external/packages/dpdk.mk > >> @@ -15,8 +15,8 @@ DPDK_PKTMBUF_HEADROOM ?= 128 > >> DPDK_USE_LIBBSD ?= n > >> DPDK_DEBUG ?= n > >> DPDK_MLX4_PMD ?= n > >> -DPDK_MLX5_PMD ?= n > >> -DPDK_MLX5_COMMON_PMD ?= n > >> +DPDK_MLX5_PMD ?= y > >> +DPDK_MLX5_COMMON_PMD ?= y > >> DPDK_TAP_PMD ?= n > >> DPDK_FAILSAFE_PMD ?= n > >> DPDK_MACHINE ?= default > >> diff --git a/build/external/packages/ipsec-mb.mk > >> b/build/external/packages/ipsec-mb.mk > >> index d0bd2af19..119eb5219 100644 > >> --- a/build/external/packages/ipsec-mb.mk > >> +++ b/build/external/packages/ipsec-mb.mk > >> @@ -34,7 +34,7 @@ define ipsec-mb_build_cmds > >> SAFE_DATA=n \ > >> PREFIX=$(ipsec-mb_install_dir) \ > >> NASM=$(ipsec-mb_install_dir)/bin/nasm \ > >> - EXTRA_CFLAGS="-g -msse4.2" > $(ipsec-mb_build_log) > >> + EXTRA_CFLAGS="-g" > $(ipsec-mb_build_log) > >> endef > >> > >> define ipsec-mb_install_cmds > >> ``` > >> > >> > >> However, when running the VPP CLI, the network interface does not show > up: > >> ``` > >> $ sudo -E make run > >> clib_sysfs_prealloc_hugepages:261: pre-allocating 6 additional 2048K > >> hugepages on numa node 0 > >> dpdk [warn ]: Unsupported PCI device 0x15b3:0xa2d6 found > >> at PCI address :03:00.0 > >> > >> dpdk/cryptodev [warn ]: dpdk_cryptodev_init: Failed to configure > >> cryptodev > >> vat-plug/load [error ]: vat_plugin_register: oddbuf plugin not > >> loaded... > >> ___ _ _ _ ___ > >> __/ __/ _ \ (_)__ | | / / _ \/ _ \ > >> _/ _// // / / / _ \ | |/ / ___/ ___/ > >> /_/ /(_)_/\___/ |___/_/ /_/ > >> > >> DBGvpp# show int > >> Name Idx State MTU > >> (L3/IP4/IP6/MPLS) Counter Count > >> local0 0 down 0/0/0/0 > >> DBGvpp# sh hard > >> Name Idx Link Hardware > >> local0 0 down local0 > >> Link speed: unknown > >> local > >> ``` > >> > >> > >> The dpdk-testpmd application seems to start correctly though: > >> ``` > >> $ sudo ./build-root/install-vpp_debug-native/extern
Re: [vpp-dev] VPP on a Bluefield-2 smartNIC
The"Unsupported PCI device 0x15b3:0xa2d6 found at PCI address :03:00.0" message disappears; however the network interface still doesn't show up. Interestingly, vpp on the host also prints this message, yet the interface can be used. By any chance, would you have any clue on what I could try to further debug this issue? Best Pierre Louis On 2021/07/01 17:50, Benoit Ganne (bganne) via lists.fd.io wrote: Please try https://gerrit.fd.io/r/c/vpp/+/32965 and reports if it works. Best ben -Original Message- From: vpp-dev@lists.fd.io On Behalf Of Pierre Louis Aublin Sent: jeudi 1 juillet 2021 07:36 To: vpp-dev@lists.fd.io Subject: [vpp-dev] VPP on a Bluefield-2 smartNIC Dear VPP developers I would like to run VPP on the Bluefield-2 smartNIC, but even though I managed to compile it the interface doesn't show up inside the CLI. By any chance, would you know how to compile and configure vpp for this device? I am using VPP v21.06-rc2 and did the following modifications so that it can compile: ``` diff --git a/build/external/packages/dpdk.mk b/build/external/packages/dpdk.mk index c7eb0fc3f..31a5c764e 100644 --- a/build/external/packages/dpdk.mk +++ b/build/external/packages/dpdk.mk @@ -15,8 +15,8 @@ DPDK_PKTMBUF_HEADROOM ?= 128 DPDK_USE_LIBBSD ?= n DPDK_DEBUG ?= n DPDK_MLX4_PMD ?= n -DPDK_MLX5_PMD ?= n -DPDK_MLX5_COMMON_PMD ?= n +DPDK_MLX5_PMD ?= y +DPDK_MLX5_COMMON_PMD ?= y DPDK_TAP_PMD ?= n DPDK_FAILSAFE_PMD ?= n DPDK_MACHINE ?= default diff --git a/build/external/packages/ipsec-mb.mk b/build/external/packages/ipsec-mb.mk index d0bd2af19..119eb5219 100644 --- a/build/external/packages/ipsec-mb.mk +++ b/build/external/packages/ipsec-mb.mk @@ -34,7 +34,7 @@ define ipsec-mb_build_cmds SAFE_DATA=n \ PREFIX=$(ipsec-mb_install_dir) \ NASM=$(ipsec-mb_install_dir)/bin/nasm \ - EXTRA_CFLAGS="-g -msse4.2" > $(ipsec-mb_build_log) + EXTRA_CFLAGS="-g" > $(ipsec-mb_build_log) endef define ipsec-mb_install_cmds ``` However, when running the VPP CLI, the network interface does not show up: ``` $ sudo -E make run clib_sysfs_prealloc_hugepages:261: pre-allocating 6 additional 2048K hugepages on numa node 0 dpdk [warn ]: Unsupported PCI device 0x15b3:0xa2d6 found at PCI address :03:00.0 dpdk/cryptodev [warn ]: dpdk_cryptodev_init: Failed to configure cryptodev vat-plug/load [error ]: vat_plugin_register: oddbuf plugin not loaded... ___ _ _ _ ___ __/ __/ _ \ (_)__ | | / / _ \/ _ \ _/ _// // / / / _ \ | |/ / ___/ ___/ /_/ /(_)_/\___/ |___/_/ /_/ DBGvpp# show int Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count local0 0 down 0/0/0/0 DBGvpp# sh hard Name Idx Link Hardware local0 0 down local0 Link speed: unknown local ``` The dpdk-testpmd application seems to start correctly though: ``` $ sudo ./build-root/install-vpp_debug-native/external/bin/dpdk-testpmd -l 0-2 -a :03:00.00 -- -i --nb-cores=2 --nb-ports=1 --total-num-mbufs=2048 EAL: Detected 8 lcore(s) EAL: Detected 1 NUMA nodes EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: No available 32768 kB hugepages reported EAL: No available 64 kB hugepages reported EAL: No available 1048576 kB hugepages reported EAL: Probing VFIO support... EAL: VFIO support initialized EAL: Invalid NUMA socket, default to 0 EAL: Probe PCI driver: mlx5_pci (15b3:a2d6) device: :03:00.0 (socket 0) mlx5_pci: Failed to allocate Tx DevX UAR (BF) mlx5_pci: Failed to allocate Rx DevX UAR (BF) mlx5_pci: Size 0x is not power of 2, will be aligned to 0x1. Interactive-mode selected testpmd: create a new mbuf pool : n=2048, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc Warning! port-topology=paired and odd forward ports number, the last port will pair with itself. Configuring Port 0 (socket 0) Port 0: 0C:42:A1:A4:89:B4 Checking link statuses... Done testpmd> ``` Is the problem related to the failure to allocate Tx and Rx DevX UAR? How can I fix this? I've also tried to set the Bluefield configuration parameters from dpdk (https://github.com/DPDK/dpdk/blob/e2a234488854fdeee267a2aa582aa082fce01d6 e/config/defconfig_arm64-bluefield-linuxapp-gcc) as follows: ``` diff --git a/build-data/packages/vpp.mk b/build-data/packages/vpp.mk index 7db450e05..91017dda0 100644 --- a/build-data/packages/vpp.mk +++ b/build-data/packages/vpp.mk @@ -32,7 +32,8 @@ vpp_cmake_args += -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON endif ifeq (,$(TARGET_PLATFORM)) ifeq ($(MACHINE),aarch64) -vpp_cma
Re: [vpp-dev] VPP on a Bluefield-2 smartNIC
Please try https://gerrit.fd.io/r/c/vpp/+/32965 and reports if it works. Best ben > -Original Message- > From: vpp-dev@lists.fd.io On Behalf Of Pierre Louis > Aublin > Sent: jeudi 1 juillet 2021 07:36 > To: vpp-dev@lists.fd.io > Subject: [vpp-dev] VPP on a Bluefield-2 smartNIC > > Dear VPP developers > > I would like to run VPP on the Bluefield-2 smartNIC, but even though I > managed to compile it the interface doesn't show up inside the CLI. By > any chance, would you know how to compile and configure vpp for this > device? > > I am using VPP v21.06-rc2 and did the following modifications so that it > can compile: > ``` > diff --git a/build/external/packages/dpdk.mk > b/build/external/packages/dpdk.mk > index c7eb0fc3f..31a5c764e 100644 > --- a/build/external/packages/dpdk.mk > +++ b/build/external/packages/dpdk.mk > @@ -15,8 +15,8 @@ DPDK_PKTMBUF_HEADROOM ?= 128 > DPDK_USE_LIBBSD ?= n > DPDK_DEBUG ?= n > DPDK_MLX4_PMD ?= n > -DPDK_MLX5_PMD ?= n > -DPDK_MLX5_COMMON_PMD ?= n > +DPDK_MLX5_PMD ?= y > +DPDK_MLX5_COMMON_PMD ?= y > DPDK_TAP_PMD ?= n > DPDK_FAILSAFE_PMD ?= n > DPDK_MACHINE ?= default > diff --git a/build/external/packages/ipsec-mb.mk > b/build/external/packages/ipsec-mb.mk > index d0bd2af19..119eb5219 100644 > --- a/build/external/packages/ipsec-mb.mk > +++ b/build/external/packages/ipsec-mb.mk > @@ -34,7 +34,7 @@ define ipsec-mb_build_cmds > SAFE_DATA=n \ > PREFIX=$(ipsec-mb_install_dir) \ > NASM=$(ipsec-mb_install_dir)/bin/nasm \ > - EXTRA_CFLAGS="-g -msse4.2" > $(ipsec-mb_build_log) > + EXTRA_CFLAGS="-g" > $(ipsec-mb_build_log) > endef > > define ipsec-mb_install_cmds > ``` > > > However, when running the VPP CLI, the network interface does not show up: > ``` > $ sudo -E make run > clib_sysfs_prealloc_hugepages:261: pre-allocating 6 additional 2048K > hugepages on numa node 0 > dpdk [warn ]: Unsupported PCI device 0x15b3:0xa2d6 found > at PCI address :03:00.0 > > dpdk/cryptodev [warn ]: dpdk_cryptodev_init: Failed to configure > cryptodev > vat-plug/load [error ]: vat_plugin_register: oddbuf plugin not > loaded... > ___ _ _ _ ___ > __/ __/ _ \ (_)__ | | / / _ \/ _ \ > _/ _// // / / / _ \ | |/ / ___/ ___/ > /_/ /(_)_/\___/ |___/_/ /_/ > > DBGvpp# show int > Name Idx State MTU > (L3/IP4/IP6/MPLS) Counter Count > local0 0 down 0/0/0/0 > DBGvpp# sh hard > Name Idx Link Hardware > local0 0 down local0 > Link speed: unknown > local > ``` > > > The dpdk-testpmd application seems to start correctly though: > ``` > $ sudo ./build-root/install-vpp_debug-native/external/bin/dpdk-testpmd > -l 0-2 -a :03:00.00 -- -i --nb-cores=2 --nb-ports=1 > --total-num-mbufs=2048 > EAL: Detected 8 lcore(s) > EAL: Detected 1 NUMA nodes > EAL: Detected static linkage of DPDK > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > EAL: Selected IOVA mode 'VA' > EAL: No available 32768 kB hugepages reported > EAL: No available 64 kB hugepages reported > EAL: No available 1048576 kB hugepages reported > EAL: Probing VFIO support... > EAL: VFIO support initialized > EAL: Invalid NUMA socket, default to 0 > EAL: Probe PCI driver: mlx5_pci (15b3:a2d6) device: :03:00.0 (socket > 0) > mlx5_pci: Failed to allocate Tx DevX UAR (BF) > mlx5_pci: Failed to allocate Rx DevX UAR (BF) > mlx5_pci: Size 0x is not power of 2, will be aligned to 0x1. > Interactive-mode selected > testpmd: create a new mbuf pool : n=2048, size=2176, socket=0 > testpmd: preferred mempool ops selected: ring_mp_mc > > Warning! port-topology=paired and odd forward ports number, the last > port will pair with itself. > > Configuring Port 0 (socket 0) > Port 0: 0C:42:A1:A4:89:B4 > Checking link statuses... > Done > testpmd> > ``` > > Is the problem related to the failure to allocate Tx and Rx DevX UAR? > How can I fix this? > > > I've also tried to set the Bluefield configuration parameters from dpdk > (https://github.com/DPDK/dpdk/blob/e2a234488854fdeee267a2aa582aa082fce01d6 > e/config/defconfig_arm64-bluefield-linuxapp-gcc) > as follows: > ``` > diff --git a/build-data/packages/vpp.mk b/build-data/packages/vpp.mk > index 7db450e05..91017dda0 100644 > --- a/build-data/packages/vpp.mk > +++ b/
[vpp-dev] VPP on a Bluefield-2 smartNIC
Dear VPP developers I would like to run VPP on the Bluefield-2 smartNIC, but even though I managed to compile it the interface doesn't show up inside the CLI. By any chance, would you know how to compile and configure vpp for this device? I am using VPP v21.06-rc2 and did the following modifications so that it can compile: ``` diff --git a/build/external/packages/dpdk.mk b/build/external/packages/dpdk.mk index c7eb0fc3f..31a5c764e 100644 --- a/build/external/packages/dpdk.mk +++ b/build/external/packages/dpdk.mk @@ -15,8 +15,8 @@ DPDK_PKTMBUF_HEADROOM ?= 128 DPDK_USE_LIBBSD ?= n DPDK_DEBUG ?= n DPDK_MLX4_PMD ?= n -DPDK_MLX5_PMD ?= n -DPDK_MLX5_COMMON_PMD ?= n +DPDK_MLX5_PMD ?= y +DPDK_MLX5_COMMON_PMD ?= y DPDK_TAP_PMD ?= n DPDK_FAILSAFE_PMD ?= n DPDK_MACHINE ?= default diff --git a/build/external/packages/ipsec-mb.mk b/build/external/packages/ipsec-mb.mk index d0bd2af19..119eb5219 100644 --- a/build/external/packages/ipsec-mb.mk +++ b/build/external/packages/ipsec-mb.mk @@ -34,7 +34,7 @@ define ipsec-mb_build_cmds SAFE_DATA=n \ PREFIX=$(ipsec-mb_install_dir) \ NASM=$(ipsec-mb_install_dir)/bin/nasm \ - EXTRA_CFLAGS="-g -msse4.2" > $(ipsec-mb_build_log) + EXTRA_CFLAGS="-g" > $(ipsec-mb_build_log) endef define ipsec-mb_install_cmds ``` However, when running the VPP CLI, the network interface does not show up: ``` $ sudo -E make run clib_sysfs_prealloc_hugepages:261: pre-allocating 6 additional 2048K hugepages on numa node 0 dpdk [warn ]: Unsupported PCI device 0x15b3:0xa2d6 found at PCI address :03:00.0 dpdk/cryptodev [warn ]: dpdk_cryptodev_init: Failed to configure cryptodev vat-plug/load [error ]: vat_plugin_register: oddbuf plugin not loaded... ___ _ _ _ ___ __/ __/ _ \ (_)__ | | / / _ \/ _ \ _/ _// // / / / _ \ | |/ / ___/ ___/ /_/ /(_)_/\___/ |___/_/ /_/ DBGvpp# show int Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count local0 0 down 0/0/0/0 DBGvpp# sh hard Name Idx Link Hardware local0 0 down local0 Link speed: unknown local ``` The dpdk-testpmd application seems to start correctly though: ``` $ sudo ./build-root/install-vpp_debug-native/external/bin/dpdk-testpmd -l 0-2 -a :03:00.00 -- -i --nb-cores=2 --nb-ports=1 --total-num-mbufs=2048 EAL: Detected 8 lcore(s) EAL: Detected 1 NUMA nodes EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: No available 32768 kB hugepages reported EAL: No available 64 kB hugepages reported EAL: No available 1048576 kB hugepages reported EAL: Probing VFIO support... EAL: VFIO support initialized EAL: Invalid NUMA socket, default to 0 EAL: Probe PCI driver: mlx5_pci (15b3:a2d6) device: :03:00.0 (socket 0) mlx5_pci: Failed to allocate Tx DevX UAR (BF) mlx5_pci: Failed to allocate Rx DevX UAR (BF) mlx5_pci: Size 0x is not power of 2, will be aligned to 0x1. Interactive-mode selected testpmd: create a new mbuf pool : n=2048, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc Warning! port-topology=paired and odd forward ports number, the last port will pair with itself. Configuring Port 0 (socket 0) Port 0: 0C:42:A1:A4:89:B4 Checking link statuses... Done testpmd> ``` Is the problem related to the failure to allocate Tx and Rx DevX UAR? How can I fix this? I've also tried to set the Bluefield configuration parameters from dpdk (https://github.com/DPDK/dpdk/blob/e2a234488854fdeee267a2aa582aa082fce01d6e/config/defconfig_arm64-bluefield-linuxapp-gcc) as follows: ``` diff --git a/build-data/packages/vpp.mk b/build-data/packages/vpp.mk index 7db450e05..91017dda0 100644 --- a/build-data/packages/vpp.mk +++ b/build-data/packages/vpp.mk @@ -32,7 +32,8 @@ vpp_cmake_args += -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON endif ifeq (,$(TARGET_PLATFORM)) ifeq ($(MACHINE),aarch64) -vpp_cmake_args += -DVPP_LOG2_CACHE_LINE_SIZE=7 +vpp_cmake_args += -DVPP_LOG2_CACHE_LINE_SIZE=6 endif endif diff --git a/build/external/packages/dpdk.mk b/build/external/packages/dpdk.mk index 70ff5c90e..e2a64e67c 100644 --- a/build/external/packages/dpdk.mk +++ b/build/external/packages/dpdk.mk @@ -15,13 +15,20 @@ DPDK_PKTMBUF_HEADROOM ?= 128 DPDK_USE_LIBBSD ?= n DPDK_DEBUG ?= n DPDK_MLX4_PMD ?= n -DPDK_MLX5_PMD ?= n -DPDK_MLX5_COMMON_PMD ?= n +DPDK_MLX5_PMD ?= y +DPDK_MLX5_COMMON_PMD ?= y DPDK_TAP_PMD ?= n DPDK_FAILSAFE_PMD ?= n DPDK_MACHINE ?= default DPDK_MLX_IBV_LINK ?= static +# bluefield specific