Re: [vpp-dev] VPP on a Bluefield-2 smartNIC
As shown below, running the test echo server on the Bluefield-2 smartNIC
fails:
```
DBGvpp# test echo server
ssvm_server_init_memfd:258: memfd map (fd 27): File exists (errno 17)
session_vpp_event_queues_allocate:1528: failed to initialize queue segment
echo_server_create_command_fn:532: No uri provided! Using default:
tcp://0.0.0.0/1234
DBGvpp# Aborted
make: *** [Makefile:519: run] Error 134
```
The mmap error (file exists) appears when MAP_FIXED_NOREPLACE was
specified in flags, and the range covered by addr and length clashes
with an existing mapping, which is the case in clib_mem_vm_reserve() in
src/vppinfra/linux/mem.c:355. The base address of this mmap is derived
from the HIGH_SEGMENT_BASEVA macro defined in src/svm/svm_common.h. By
default its value is 128 << 30 == 0x20 on a 64-bits system.
|Doing info proc mappings in gdb tells me that ||indeed ||this address
is an already mapped range:
```
Start Addr End Addr Size Offset objfile
0x198400 0x218400 0x8 0x0
```
What is the logic behind this value? To which value should I set it
instead?
Best
Pierre Louis
|
On 2021/07/02 19:54, Pierre Louis Aublin wrote:
I do not have a dedicated Bluefield-2 at my disposal; instead, I am
using one in a shared cluster, chosen at random by the cluster job
scheduler. Anyway, today I tried on a different one which has the
Mellanox OFED driver installed. I first had to apply this patch so
that VPP compiles: https://gerrit.fd.io/r/c/vpp/+/30842 .
My startup.conf is:
```
unix { interactive cli-listen /run/vpp/cli.sock gid 0 }
dpdk { dev :03:00.0 }
plugins { plugin dpdk_plugin.so { enable }
```
This time I could see and configure the network interface, but VPP
crashed when launching the echo server:
```
$ sudo -E make run
dpdk [warn ]: not enough DPDK crypto resources
dpdk/cryptodev [warn ]: dpdk_cryptodev_init: Not enough cryptodevs
vat-plug/load [error ]: vat_plugin_register: oddbuf plugin not
loaded...
___ _ _ _ ___
__/ __/ _ \ (_)__ | | / / _ \/ _ \
_/ _// // / / / _ \ | |/ / ___/ ___/
/_/ /(_)_/\___/ |___/_/ /_/
DBGvpp# sh int
Name Idx State MTU
(L3/IP4/IP6/MPLS) Counter Count
HundredGigabitEthernet3/0/0 1 down 9000/0/0/0
local0 0 down 0/0/0/0
DBGvpp# set interface ip address HundredGigabitEthernet3/0/0
192.168.10.1/24
DBGvpp# set interface state HundredGigabitEthernet3/0/0 up
DBGvpp# sh int addr
HundredGigabitEthernet3/0/0 (up):
L3 192.168.10.1/24
local0 (dn):
DBGvpp# test echo server
ssvm_server_init_memfd:258: memfd map (fd 27): File exists (errno 17)
session_vpp_event_queues_allocate:1528: failed to initialize queue
segment
echo_server_create_command_fn:532: No uri provided! Using default:
tcp://0.0.0.0/1234
DBGvpp# Aborted
make: *** [Makefile:519: run] Error 134
```
Please find attached the output of `sh log`.
Best
Pierre Louis
On 2021/07/01 18:38, Benoit Ganne (bganne) wrote:
What is your VPP dpdk config looks like in startup.conf? Esp. did you
whitelist the device? See
https://fd.io/docs/vpp/master/gettingstarted/users/configuring/startup.html#the-dpdk-section
Also, please share the output of 'show logs'.
Best
ben
-Original Message-
From: Pierre Louis Aublin
Sent: jeudi 1 juillet 2021 11:08
To: Benoit Ganne (bganne) ; vpp-dev@lists.fd.io
Subject: Re: [vpp-dev] VPP on a Bluefield-2 smartNIC
The"Unsupported PCI device 0x15b3:0xa2d6 found at PCI address
:03:00.0" message disappears; however the network interface still
doesn't show up. Interestingly, vpp on the host also prints this
message, yet the interface can be used.
By any chance, would you have any clue on what I could try to further
debug this issue?
Best
Pierre Louis
On 2021/07/01 17:50, Benoit Ganne (bganne) via lists.fd.io wrote:
Please try https://gerrit.fd.io/r/c/vpp/+/32965 and reports if it
works.
Best
ben
-Original Message-
From: vpp-dev@lists.fd.io On Behalf Of Pierre
Louis
Aublin
Sent: jeudi 1 juillet 2021 07:36
To: vpp-dev@lists.fd.io
Subject: [vpp-dev] VPP on a Bluefield-2 smartNIC
Dear VPP developers
I would like to run VPP on the Bluefield-2 smartNIC, but even
though I
managed to compile it the interface doesn't show up inside the
CLI. By
any chance, would you know how to compile and configure vpp for this
device?
I am using VPP v21.06-rc2 and did the following modifications so that
it
can compile:
```
diff --git a/build/external/packages/dpdk.mk
b/build/external/packages/dpdk.mk
index c7eb0fc3f..31a5c764e 100644
--- a/build/external/packages/dpdk.mk
+++ b/build/external/packages/dpdk.mk
@@ -15,8 +15,8 @@ DPDK_PKTMBUF_HEADROOM ?= 128
DPDK_USE_LIBBSD ?= n
DPDK_DEBUG ?= n
DPDK_MLX4_PMD ?= n
-DPDK_MLX5_PMD ?= n
-DPDK_MLX5_COMMON_PMD ?=
Re: [vpp-dev] VPP on a Bluefield-2 smartNIC
I do not have a dedicated Bluefield-2 at my disposal; instead, I am
using one in a shared cluster, chosen at random by the cluster job
scheduler. Anyway, today I tried on a different one which has the
Mellanox OFED driver installed. I first had to apply this patch so that
VPP compiles: https://gerrit.fd.io/r/c/vpp/+/30842 .
My startup.conf is:
```
unix { interactive cli-listen /run/vpp/cli.sock gid 0 }
dpdk { dev :03:00.0 }
plugins { plugin dpdk_plugin.so { enable }
```
This time I could see and configure the network interface, but VPP
crashed when launching the echo server:
```
$ sudo -E make run
dpdk [warn ]: not enough DPDK crypto resources
dpdk/cryptodev [warn ]: dpdk_cryptodev_init: Not enough cryptodevs
vat-plug/load [error ]: vat_plugin_register: oddbuf plugin not loaded...
___ _ _ _ ___
__/ __/ _ \ (_)__ | | / / _ \/ _ \
_/ _// // / / / _ \ | |/ / ___/ ___/
/_/ /(_)_/\___/ |___/_/ /_/
DBGvpp# sh int
Name Idx State MTU
(L3/IP4/IP6/MPLS) Counter Count
HundredGigabitEthernet3/0/0 1 down 9000/0/0/0
local0 0 down 0/0/0/0
DBGvpp# set interface ip address HundredGigabitEthernet3/0/0 192.168.10.1/24
DBGvpp# set interface state HundredGigabitEthernet3/0/0 up
DBGvpp# sh int addr
HundredGigabitEthernet3/0/0 (up):
L3 192.168.10.1/24
local0 (dn):
DBGvpp# test echo server
ssvm_server_init_memfd:258: memfd map (fd 27): File exists (errno 17)
session_vpp_event_queues_allocate:1528: failed to initialize queue segment
echo_server_create_command_fn:532: No uri provided! Using default:
tcp://0.0.0.0/1234
DBGvpp# Aborted
make: *** [Makefile:519: run] Error 134
```
Please find attached the output of `sh log`.
Best
Pierre Louis
On 2021/07/01 18:38, Benoit Ganne (bganne) wrote:
What is your VPP dpdk config looks like in startup.conf? Esp. did you whitelist
the device? See
https://fd.io/docs/vpp/master/gettingstarted/users/configuring/startup.html#the-dpdk-section
Also, please share the output of 'show logs'.
Best
ben
-Original Message-
From: Pierre Louis Aublin
Sent: jeudi 1 juillet 2021 11:08
To: Benoit Ganne (bganne) ; vpp-dev@lists.fd.io
Subject: Re: [vpp-dev] VPP on a Bluefield-2 smartNIC
The"Unsupported PCI device 0x15b3:0xa2d6 found at PCI address
:03:00.0" message disappears; however the network interface still
doesn't show up. Interestingly, vpp on the host also prints this
message, yet the interface can be used.
By any chance, would you have any clue on what I could try to further
debug this issue?
Best
Pierre Louis
On 2021/07/01 17:50, Benoit Ganne (bganne) via lists.fd.io wrote:
Please try https://gerrit.fd.io/r/c/vpp/+/32965 and reports if it works.
Best
ben
-Original Message-
From: vpp-dev@lists.fd.io On Behalf Of Pierre
Louis
Aublin
Sent: jeudi 1 juillet 2021 07:36
To: vpp-dev@lists.fd.io
Subject: [vpp-dev] VPP on a Bluefield-2 smartNIC
Dear VPP developers
I would like to run VPP on the Bluefield-2 smartNIC, but even though I
managed to compile it the interface doesn't show up inside the CLI. By
any chance, would you know how to compile and configure vpp for this
device?
I am using VPP v21.06-rc2 and did the following modifications so that
it
can compile:
```
diff --git a/build/external/packages/dpdk.mk
b/build/external/packages/dpdk.mk
index c7eb0fc3f..31a5c764e 100644
--- a/build/external/packages/dpdk.mk
+++ b/build/external/packages/dpdk.mk
@@ -15,8 +15,8 @@ DPDK_PKTMBUF_HEADROOM ?= 128
DPDK_USE_LIBBSD ?= n
DPDK_DEBUG ?= n
DPDK_MLX4_PMD ?= n
-DPDK_MLX5_PMD ?= n
-DPDK_MLX5_COMMON_PMD ?= n
+DPDK_MLX5_PMD ?= y
+DPDK_MLX5_COMMON_PMD ?= y
DPDK_TAP_PMD ?= n
DPDK_FAILSAFE_PMD ?= n
DPDK_MACHINE ?= default
diff --git a/build/external/packages/ipsec-mb.mk
b/build/external/packages/ipsec-mb.mk
index d0bd2af19..119eb5219 100644
--- a/build/external/packages/ipsec-mb.mk
+++ b/build/external/packages/ipsec-mb.mk
@@ -34,7 +34,7 @@ define ipsec-mb_build_cmds
SAFE_DATA=n \
PREFIX=$(ipsec-mb_install_dir) \
NASM=$(ipsec-mb_install_dir)/bin/nasm \
- EXTRA_CFLAGS="-g -msse4.2" > $(ipsec-mb_build_log)
+ EXTRA_CFLAGS="-g" > $(ipsec-mb_build_log)
endef
define ipsec-mb_install_cmds
```
However, when running the VPP CLI, the network interface does not show
up:
```
$ sudo -E make run
clib_sysfs_prealloc_hugepages:261: pre-allocating 6 additional 2048K
hugepages on numa node 0
dpdk [warn ]: Unsupported PCI device 0x15b3:0xa2d6 found
at PCI address :03:00.0
dpdk/cryptodev [warn ]: dpdk_cryptodev_init: Failed to configure
cryptodev
vat-plug/load [error ]: vat_plugi
Re: [vpp-dev] VPP on a Bluefield-2 smartNIC
Indeed, it is not needed. I previously removed it to compile on an old Intel CPU without sse4.2 and kept it for the Bluefield. Thank you On 2021/07/01 20:45, Damjan Marion wrote: On 01.07.2021., at 07:35, Pierre Louis Aublin wrote: diff --git a/build/external/packages/ipsec-mb.mk b/build/external/packages/ipsec-mb.mk index d0bd2af19..119eb5219 100644 --- a/build/external/packages/ipsec-mb.mk +++ b/build/external/packages/ipsec-mb.mk @@ -34,7 +34,7 @@ define ipsec-mb_build_cmds SAFE_DATA=n \ PREFIX=$(ipsec-mb_install_dir) \ NASM=$(ipsec-mb_install_dir)/bin/nasm \ - EXTRA_CFLAGS="-g -msse4.2" > $(ipsec-mb_build_log) + EXTRA_CFLAGS="-g" > $(ipsec-mb_build_log) Why do you need this change? If i get it right bluefield uses ARM cpus and we don’t compile intel ipsecmb lib on arm. $ git grep ARCH_X86_64 build/external/Makefile build/external/Makefile:ARCH_X86_64=$(filter x86_64,$(shell uname -m)) build/external/Makefile:install: $(if $(ARCH_X86_64), nasm-install ipsec-mb-install) dpdk-install rdma-core-install quicly-install libbpf-install build/external/Makefile:config: $(if $(ARCH_X86_64), nasm-config ipsec-mb-config) dpdk-config rdma-core-config quicly-build — Damjan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19692): https://lists.fd.io/g/vpp-dev/message/19692 Mute This Topic: https://lists.fd.io/mt/83910198/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] VPP on a Bluefield-2 smartNIC
The"Unsupported PCI device 0x15b3:0xa2d6 found at PCI address :03:00.0" message disappears; however the network interface still doesn't show up. Interestingly, vpp on the host also prints this message, yet the interface can be used. By any chance, would you have any clue on what I could try to further debug this issue? Best Pierre Louis On 2021/07/01 17:50, Benoit Ganne (bganne) via lists.fd.io wrote: Please try https://gerrit.fd.io/r/c/vpp/+/32965 and reports if it works. Best ben -Original Message- From: vpp-dev@lists.fd.io On Behalf Of Pierre Louis Aublin Sent: jeudi 1 juillet 2021 07:36 To: vpp-dev@lists.fd.io Subject: [vpp-dev] VPP on a Bluefield-2 smartNIC Dear VPP developers I would like to run VPP on the Bluefield-2 smartNIC, but even though I managed to compile it the interface doesn't show up inside the CLI. By any chance, would you know how to compile and configure vpp for this device? I am using VPP v21.06-rc2 and did the following modifications so that it can compile: ``` diff --git a/build/external/packages/dpdk.mk b/build/external/packages/dpdk.mk index c7eb0fc3f..31a5c764e 100644 --- a/build/external/packages/dpdk.mk +++ b/build/external/packages/dpdk.mk @@ -15,8 +15,8 @@ DPDK_PKTMBUF_HEADROOM ?= 128 DPDK_USE_LIBBSD ?= n DPDK_DEBUG ?= n DPDK_MLX4_PMD ?= n -DPDK_MLX5_PMD ?= n -DPDK_MLX5_COMMON_PMD ?= n +DPDK_MLX5_PMD ?= y +DPDK_MLX5_COMMON_PMD ?= y DPDK_TAP_PMD ?= n DPDK_FAILSAFE_PMD ?= n DPDK_MACHINE ?= default diff --git a/build/external/packages/ipsec-mb.mk b/build/external/packages/ipsec-mb.mk index d0bd2af19..119eb5219 100644 --- a/build/external/packages/ipsec-mb.mk +++ b/build/external/packages/ipsec-mb.mk @@ -34,7 +34,7 @@ define ipsec-mb_build_cmds SAFE_DATA=n \ PREFIX=$(ipsec-mb_install_dir) \ NASM=$(ipsec-mb_install_dir)/bin/nasm \ - EXTRA_CFLAGS="-g -msse4.2" > $(ipsec-mb_build_log) + EXTRA_CFLAGS="-g" > $(ipsec-mb_build_log) endef define ipsec-mb_install_cmds ``` However, when running the VPP CLI, the network interface does not show up: ``` $ sudo -E make run clib_sysfs_prealloc_hugepages:261: pre-allocating 6 additional 2048K hugepages on numa node 0 dpdk [warn ]: Unsupported PCI device 0x15b3:0xa2d6 found at PCI address :03:00.0 dpdk/cryptodev [warn ]: dpdk_cryptodev_init: Failed to configure cryptodev vat-plug/load [error ]: vat_plugin_register: oddbuf plugin not loaded... ___ _ _ _ ___ __/ __/ _ \ (_)__ | | / / _ \/ _ \ _/ _// // / / / _ \ | |/ / ___/ ___/ /_/ /(_)_/\___/ |___/_/ /_/ DBGvpp# show int Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count local0 0 down 0/0/0/0 DBGvpp# sh hard Name Idx Link Hardware local0 0 down local0 Link speed: unknown local ``` The dpdk-testpmd application seems to start correctly though: ``` $ sudo ./build-root/install-vpp_debug-native/external/bin/dpdk-testpmd -l 0-2 -a :03:00.00 -- -i --nb-cores=2 --nb-ports=1 --total-num-mbufs=2048 EAL: Detected 8 lcore(s) EAL: Detected 1 NUMA nodes EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: No available 32768 kB hugepages reported EAL: No available 64 kB hugepages reported EAL: No available 1048576 kB hugepages reported EAL: Probing VFIO support... EAL: VFIO support initialized EAL: Invalid NUMA socket, default to 0 EAL: Probe PCI driver: mlx5_pci (15b3:a2d6) device: :03:00.0 (socket 0) mlx5_pci: Failed to allocate Tx DevX UAR (BF) mlx5_pci: Failed to allocate Rx DevX UAR (BF) mlx5_pci: Size 0x is not power of 2, will be aligned to 0x1. Interactive-mode selected testpmd: create a new mbuf pool : n=2048, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc Warning! port-topology=paired and odd forward ports number, the last port will pair with itself. Configuring Port 0 (socket 0) Port 0: 0C:42:A1:A4:89:B4 Checking link statuses... Done testpmd> ``` Is the problem related to the failure to allocate Tx and Rx DevX UAR? How can I fix this? I've also tried to set the Bluefield configuration parameters from dpdk (https://github.com/DPDK/dpdk/blob/e2a234488854fdeee267a2aa582aa082fce01d6 e/config/defconfig_arm64-bluefield-linuxapp-gcc) as follows: ``` diff --git a/build-data/packages/vpp.mk b/build-data/packages/vpp.mk index 7db450e05..91017dda0 100644 --- a/build-data/packages/vpp.mk +++ b/build-data/packages/vpp.mk @@ -32,7 +32,8 @@ vpp_cmake_args += -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON endif ifeq (,$(TARGET_PLATFORM)) ifeq ($(MACHINE),aarch64) -vpp_cma
[vpp-dev] VPP on a Bluefield-2 smartNIC
Dear VPP developers I would like to run VPP on the Bluefield-2 smartNIC, but even though I managed to compile it the interface doesn't show up inside the CLI. By any chance, would you know how to compile and configure vpp for this device? I am using VPP v21.06-rc2 and did the following modifications so that it can compile: ``` diff --git a/build/external/packages/dpdk.mk b/build/external/packages/dpdk.mk index c7eb0fc3f..31a5c764e 100644 --- a/build/external/packages/dpdk.mk +++ b/build/external/packages/dpdk.mk @@ -15,8 +15,8 @@ DPDK_PKTMBUF_HEADROOM ?= 128 DPDK_USE_LIBBSD ?= n DPDK_DEBUG ?= n DPDK_MLX4_PMD ?= n -DPDK_MLX5_PMD ?= n -DPDK_MLX5_COMMON_PMD ?= n +DPDK_MLX5_PMD ?= y +DPDK_MLX5_COMMON_PMD ?= y DPDK_TAP_PMD ?= n DPDK_FAILSAFE_PMD ?= n DPDK_MACHINE ?= default diff --git a/build/external/packages/ipsec-mb.mk b/build/external/packages/ipsec-mb.mk index d0bd2af19..119eb5219 100644 --- a/build/external/packages/ipsec-mb.mk +++ b/build/external/packages/ipsec-mb.mk @@ -34,7 +34,7 @@ define ipsec-mb_build_cmds SAFE_DATA=n \ PREFIX=$(ipsec-mb_install_dir) \ NASM=$(ipsec-mb_install_dir)/bin/nasm \ - EXTRA_CFLAGS="-g -msse4.2" > $(ipsec-mb_build_log) + EXTRA_CFLAGS="-g" > $(ipsec-mb_build_log) endef define ipsec-mb_install_cmds ``` However, when running the VPP CLI, the network interface does not show up: ``` $ sudo -E make run clib_sysfs_prealloc_hugepages:261: pre-allocating 6 additional 2048K hugepages on numa node 0 dpdk [warn ]: Unsupported PCI device 0x15b3:0xa2d6 found at PCI address :03:00.0 dpdk/cryptodev [warn ]: dpdk_cryptodev_init: Failed to configure cryptodev vat-plug/load [error ]: vat_plugin_register: oddbuf plugin not loaded... ___ _ _ _ ___ __/ __/ _ \ (_)__ | | / / _ \/ _ \ _/ _// // / / / _ \ | |/ / ___/ ___/ /_/ /(_)_/\___/ |___/_/ /_/ DBGvpp# show int Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count local0 0 down 0/0/0/0 DBGvpp# sh hard Name Idx Link Hardware local0 0 down local0 Link speed: unknown local ``` The dpdk-testpmd application seems to start correctly though: ``` $ sudo ./build-root/install-vpp_debug-native/external/bin/dpdk-testpmd -l 0-2 -a :03:00.00 -- -i --nb-cores=2 --nb-ports=1 --total-num-mbufs=2048 EAL: Detected 8 lcore(s) EAL: Detected 1 NUMA nodes EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: No available 32768 kB hugepages reported EAL: No available 64 kB hugepages reported EAL: No available 1048576 kB hugepages reported EAL: Probing VFIO support... EAL: VFIO support initialized EAL: Invalid NUMA socket, default to 0 EAL: Probe PCI driver: mlx5_pci (15b3:a2d6) device: :03:00.0 (socket 0) mlx5_pci: Failed to allocate Tx DevX UAR (BF) mlx5_pci: Failed to allocate Rx DevX UAR (BF) mlx5_pci: Size 0x is not power of 2, will be aligned to 0x1. Interactive-mode selected testpmd: create a new mbuf pool : n=2048, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc Warning! port-topology=paired and odd forward ports number, the last port will pair with itself. Configuring Port 0 (socket 0) Port 0: 0C:42:A1:A4:89:B4 Checking link statuses... Done testpmd> ``` Is the problem related to the failure to allocate Tx and Rx DevX UAR? How can I fix this? I've also tried to set the Bluefield configuration parameters from dpdk (https://github.com/DPDK/dpdk/blob/e2a234488854fdeee267a2aa582aa082fce01d6e/config/defconfig_arm64-bluefield-linuxapp-gcc) as follows: ``` diff --git a/build-data/packages/vpp.mk b/build-data/packages/vpp.mk index 7db450e05..91017dda0 100644 --- a/build-data/packages/vpp.mk +++ b/build-data/packages/vpp.mk @@ -32,7 +32,8 @@ vpp_cmake_args += -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON endif ifeq (,$(TARGET_PLATFORM)) ifeq ($(MACHINE),aarch64) -vpp_cmake_args += -DVPP_LOG2_CACHE_LINE_SIZE=7 +vpp_cmake_args += -DVPP_LOG2_CACHE_LINE_SIZE=6 endif endif diff --git a/build/external/packages/dpdk.mk b/build/external/packages/dpdk.mk index 70ff5c90e..e2a64e67c 100644 --- a/build/external/packages/dpdk.mk +++ b/build/external/packages/dpdk.mk @@ -15,13 +15,20 @@ DPDK_PKTMBUF_HEADROOM ?= 128 DPDK_USE_LIBBSD ?= n DPDK_DEBUG ?= n DPDK_MLX4_PMD ?= n -DPDK_MLX5_PMD ?= n -DPDK_MLX5_COMMON_PMD ?= n +DPDK_MLX5_PMD ?= y +DPDK_MLX5_COMMON_PMD ?= y DPDK_TAP_PMD ?= n DPDK_FAILSAFE_PMD ?= n DPDK_MACHINE ?= default DPDK_MLX_IBV_LINK ?= static +# bluefield specific
