As shown below, running the test echo server on the Bluefield-2 smartNIC
fails:
```
DBGvpp# test echo server
ssvm_server_init_memfd:258: memfd map (fd 27): File exists (errno 17)
session_vpp_event_queues_allocate:1528: failed to initialize queue segment
echo_server_create_command_fn:532: No uri provided! Using default:
tcp://0.0.0.0/1234
DBGvpp# Aborted
make: *** [Makefile:519: run] Error 134
```
The mmap error (file exists) appears when MAP_FIXED_NOREPLACE was
specified in flags, and the range covered by addr and length clashes
with an existing mapping, which is the case in clib_mem_vm_reserve() in
src/vppinfra/linux/mem.c:355. The base address of this mmap is derived
from the HIGH_SEGMENT_BASEVA macro defined in src/svm/svm_common.h. By
default its value is 128 << 30 == 0x2000000000 on a 64-bits system.
|Doing info proc mappings in gdb tells me that ||indeed ||this address
is an already mapped range:
```
Start Addr End Addr Size Offset objfile
0x1984000000 0x2184000000 0x800000000 0x0
```
What is the logic behind this value? To which value should I set it
instead?
Best
Pierre Louis
|
On 2021/07/02 19:54, Pierre Louis Aublin wrote:
I do not have a dedicated Bluefield-2 at my disposal; instead, I am
using one in a shared cluster, chosen at random by the cluster job
scheduler. Anyway, today I tried on a different one which has the
Mellanox OFED driver installed. I first had to apply this patch so
that VPP compiles: https://gerrit.fd.io/r/c/vpp/+/30842 .
My startup.conf is:
```
unix { interactive cli-listen /run/vpp/cli.sock gid 0 }
dpdk { dev 0000:03:00.0 }
plugins { plugin dpdk_plugin.so { enable }
```
This time I could see and configure the network interface, but VPP
crashed when launching the echo server:
```
$ sudo -E make run
dpdk [warn ]: not enough DPDK crypto resources
dpdk/cryptodev [warn ]: dpdk_cryptodev_init: Not enough cryptodevs
vat-plug/load [error ]: vat_plugin_register: oddbuf plugin not
loaded...
_______ _ _ _____ ___
__/ __/ _ \ (_)__ | | / / _ \/ _ \
_/ _// // / / / _ \ | |/ / ___/ ___/
/_/ /____(_)_/\___/ |___/_/ /_/
DBGvpp# sh int
Name Idx State MTU
(L3/IP4/IP6/MPLS) Counter Count
HundredGigabitEthernet3/0/0 1 down 9000/0/0/0
local0 0 down 0/0/0/0
DBGvpp# set interface ip address HundredGigabitEthernet3/0/0
192.168.10.1/24
DBGvpp# set interface state HundredGigabitEthernet3/0/0 up
DBGvpp# sh int addr
HundredGigabitEthernet3/0/0 (up):
L3 192.168.10.1/24
local0 (dn):
DBGvpp# test echo server
ssvm_server_init_memfd:258: memfd map (fd 27): File exists (errno 17)
session_vpp_event_queues_allocate:1528: failed to initialize queue
segment
echo_server_create_command_fn:532: No uri provided! Using default:
tcp://0.0.0.0/1234
DBGvpp# Aborted
make: *** [Makefile:519: run] Error 134
```
Please find attached the output of `sh log`.
Best
Pierre Louis
On 2021/07/01 18:38, Benoit Ganne (bganne) wrote:
What is your VPP dpdk config looks like in startup.conf? Esp. did you
whitelist the device? See
https://fd.io/docs/vpp/master/gettingstarted/users/configuring/startup.html#the-dpdk-section
Also, please share the output of 'show logs'.
Best
ben
-----Original Message-----
From: Pierre Louis Aublin <pierrelo...@iij-ii.co.jp>
Sent: jeudi 1 juillet 2021 11:08
To: Benoit Ganne (bganne) <bga...@cisco.com>; vpp-dev@lists.fd.io
Subject: Re: [vpp-dev] VPP on a Bluefield-2 smartNIC
The"Unsupported PCI device 0x15b3:0xa2d6 found at PCI address
0000:03:00.0" message disappears; however the network interface still
doesn't show up. Interestingly, vpp on the host also prints this
message, yet the interface can be used.
By any chance, would you have any clue on what I could try to further
debug this issue?
Best
Pierre Louis
On 2021/07/01 17:50, Benoit Ganne (bganne) via lists.fd.io wrote:
Please try https://gerrit.fd.io/r/c/vpp/+/32965 and reports if it
works.
Best
ben
-----Original Message-----
From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Pierre
Louis
Aublin
Sent: jeudi 1 juillet 2021 07:36
To: vpp-dev@lists.fd.io
Subject: [vpp-dev] VPP on a Bluefield-2 smartNIC
Dear VPP developers
I would like to run VPP on the Bluefield-2 smartNIC, but even
though I
managed to compile it the interface doesn't show up inside the
CLI. By
any chance, would you know how to compile and configure vpp for this
device?
I am using VPP v21.06-rc2 and did the following modifications so that
it
can compile:
```
diff --git a/build/external/packages/dpdk.mk
b/build/external/packages/dpdk.mk
index c7eb0fc3f..31a5c764e 100644
--- a/build/external/packages/dpdk.mk
+++ b/build/external/packages/dpdk.mk
@@ -15,8 +15,8 @@ DPDK_PKTMBUF_HEADROOM ?= 128
DPDK_USE_LIBBSD ?= n
DPDK_DEBUG ?= n
DPDK_MLX4_PMD ?= n
-DPDK_MLX5_PMD ?= n
-DPDK_MLX5_COMMON_PMD ?= n
+DPDK_MLX5_PMD ?= y
+DPDK_MLX5_COMMON_PMD ?= y
DPDK_TAP_PMD ?= n
DPDK_FAILSAFE_PMD ?= n
DPDK_MACHINE ?= default
diff --git a/build/external/packages/ipsec-mb.mk
b/build/external/packages/ipsec-mb.mk
index d0bd2af19..119eb5219 100644
--- a/build/external/packages/ipsec-mb.mk
+++ b/build/external/packages/ipsec-mb.mk
@@ -34,7 +34,7 @@ define ipsec-mb_build_cmds
SAFE_DATA=n \
PREFIX=$(ipsec-mb_install_dir) \
NASM=$(ipsec-mb_install_dir)/bin/nasm \
- EXTRA_CFLAGS="-g -msse4.2" > $(ipsec-mb_build_log)
+ EXTRA_CFLAGS="-g" > $(ipsec-mb_build_log)
endef
define ipsec-mb_install_cmds
```
However, when running the VPP CLI, the network interface does not
show
up:
```
$ sudo -E make run
clib_sysfs_prealloc_hugepages:261: pre-allocating 6 additional 2048K
hugepages on numa node 0
dpdk [warn ]: Unsupported PCI device 0x15b3:0xa2d6
found
at PCI address 0000:03:00.0
dpdk/cryptodev [warn ]: dpdk_cryptodev_init: Failed to configure
cryptodev
vat-plug/load [error ]: vat_plugin_register: oddbuf plugin not
loaded...
_______ _ _ _____ ___
__/ __/ _ \ (_)__ | | / / _ \/ _ \
_/ _// // / / / _ \ | |/ / ___/ ___/
/_/ /____(_)_/\___/ |___/_/ /_/
DBGvpp# show int
Name Idx State MTU
(L3/IP4/IP6/MPLS) Counter Count
local0 0 down 0/0/0/0
DBGvpp# sh hard
Name Idx Link Hardware
local0 0 down local0
Link speed: unknown
local
```
The dpdk-testpmd application seems to start correctly though:
```
$ sudo
./build-root/install-vpp_debug-native/external/bin/dpdk-testpmd
-l 0-2 -a 0000:03:00.00 -- -i --nb-cores=2 --nb-ports=1
--total-num-mbufs=2048
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available 32768 kB hugepages reported
EAL: No available 64 kB hugepages reported
EAL: No available 1048576 kB hugepages reported
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Invalid NUMA socket, default to 0
EAL: Probe PCI driver: mlx5_pci (15b3:a2d6) device: 0000:03:00.0
(socket
0)
mlx5_pci: Failed to allocate Tx DevX UAR (BF)
mlx5_pci: Failed to allocate Rx DevX UAR (BF)
mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000.
Interactive-mode selected
testpmd: create a new mbuf pool <mb_pool_0>: n=2048, size=2176,
socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Warning! port-topology=paired and odd forward ports number, the last
port will pair with itself.
Configuring Port 0 (socket 0)
Port 0: 0C:42:A1:A4:89:B4
Checking link statuses...
Done
testpmd>
```
Is the problem related to the failure to allocate Tx and Rx DevX UAR?
How can I fix this?
I've also tried to set the Bluefield configuration parameters from
dpdk
(https://github.com/DPDK/dpdk/blob/e2a234488854fdeee267a2aa582aa082fce01d6
e/config/defconfig_arm64-bluefield-linuxapp-gcc)
as follows:
```
diff --git a/build-data/packages/vpp.mk b/build-data/packages/vpp.mk
index 7db450e05..91017dda0 100644
--- a/build-data/packages/vpp.mk
+++ b/build-data/packages/vpp.mk
@@ -32,7 +32,8 @@ vpp_cmake_args += -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON
endif
ifeq (,$(TARGET_PLATFORM))
ifeq ($(MACHINE),aarch64)
-vpp_cmake_args += -DVPP_LOG2_CACHE_LINE_SIZE=7
+vpp_cmake_args += -DVPP_LOG2_CACHE_LINE_SIZE=6
endif
endif
diff --git a/build/external/packages/dpdk.mk
b/build/external/packages/dpdk.mk
index 70ff5c90e..e2a64e67c 100644
--- a/build/external/packages/dpdk.mk
+++ b/build/external/packages/dpdk.mk
@@ -15,13 +15,20 @@ DPDK_PKTMBUF_HEADROOM ?= 128
DPDK_USE_LIBBSD ?= n
DPDK_DEBUG ?= n
DPDK_MLX4_PMD ?= n
-DPDK_MLX5_PMD ?= n
-DPDK_MLX5_COMMON_PMD ?= n
+DPDK_MLX5_PMD ?= y
+DPDK_MLX5_COMMON_PMD ?= y
DPDK_TAP_PMD ?= n
DPDK_FAILSAFE_PMD ?= n
DPDK_MACHINE ?= default
DPDK_MLX_IBV_LINK ?= static
+# bluefield specific
+DPDK_ARCH_ARM_TUNE ?= "cortex-a72"
+DPDK_MAX_NUMA_NODES ?= 1
+DPDK_CACHE_LINE_SIZE ?= 64
+DPDK_EAL_NUMA_AWARE_HUGEPAGES ?= n
+DPDK_LIBRTE_VHOST_NUMA ?= n
+
dpdk_version ?= 21.05
dpdk_base_url ?= http://fast.dpdk.org/rel
dpdk_tarball := dpdk-$(dpdk_version).tar.xz
@@ -178,6 +185,11 @@ define dpdk_config_cmds
echo "DPDK post meson configuration" && \
echo "Altering rte_build_config.h" && \
$(call dpdk_config,PKTMBUF_HEADROOM) && \
+ $(call dpdk_config,ARCH_ARM_TUNE) && \
+ $(call dpdk_config,MAX_NUMA_NODES) && \
+ $(call dpdk_config,CACHE_LINE_SIZE) && \
+ $(call dpdk_config_def,EAL_NUMA_AWARE_HUGEPAGES) && \
+ $(call dpdk_config_def,LIBRTE_VHOST_NUMA) && \
$(call dpdk_config_def,USE_LIBBSD)
endef
```
Unfortunately this led to an error when launching the CLI:
```
$ sudo -E make run
dpdk [warn ]: Unsupported PCI device 0x15b3:0xa2d6
found
at PCI address 0000:03:00.0
0: dpdk_buffer_pool_init: failed to create normal mempool for numa
node
0
$
```
Thank you very much
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#19718): https://lists.fd.io/g/vpp-dev/message/19718
Mute This Topic: https://lists.fd.io/mt/83910198/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
