Re: [ovs-dev] [PATCH v2 3/3] system-dpdk: Test with mlx5 devices.
On 24 Nov 2021, at 17:14, Eelco Chaudron wrote:
> On 23 Nov 2021, at 15:15, David Marchand wrote:
>
>> The DPDK unit test only runs if vfio or igb_uio kernel module is loaded:
>> on systems with only mlx5, this test is always skipped.
>>
>> Besides, the test tries to grab the first device listed by dpdk-devbind.py,
>> regardless of the PCI device status regarding kmod binding.
>>
>> Remove dependency on this DPDK script and use a minimal script that
>> reads PCI sysfs.
>>
>> This script is not perfect, as one can imagine PCI devices bound to
>> vfio-pci for virtual machines.
>> Add a new environment variable DPDK_PCI_ADDR for testers to select the
>> PCI device of their liking.
>> For consistency and grep, the temporary file PCI_ADDR is renamed
>> to DPDK_PCI_ADDR.
>>
>> Note: with mlx5 devices, there is now more OVS/DPDK warnings to waive.
>
> Uncovered some build issues with my scripts as now my MLX5 card in the system
> is used :)
>
> Changes look good to me, one small nit.
>
> Acked-by: Eelco Chaudron
>
>> Signed-off-by: David Marchand
>> ---
>> Documentation/topics/testing.rst | 1 -
>> tests/automake.mk| 1 +
>> tests/system-dpdk-find-device.py | 44
>> tests/system-dpdk-macros.at | 10 ++--
>> tests/system-dpdk.at | 4 ++-
>> 5 files changed, 50 insertions(+), 10 deletions(-)
>> create mode 100755 tests/system-dpdk-find-device.py
>>
>> diff --git a/Documentation/topics/testing.rst
>> b/Documentation/topics/testing.rst
>> index 0ddcbd58ab..00f734a77a 100644
>> --- a/Documentation/topics/testing.rst
>> +++ b/Documentation/topics/testing.rst
>> @@ -343,7 +343,6 @@ To see a list of all the available tests, run::
>>
>> These tests support a `DPDK supported NIC`_. The tests operate on a wider
>> set of
>> environments, for instance, when a virtual port is used.
>> -They do require proper DPDK variables (``DPDK_DIR`` and ``DPDK_BUILD``).
>> Moreover you need to have root privileges to load the required modules and
>> to bind
>> the NIC to the DPDK-compatible driver.
>>
>> diff --git a/tests/automake.mk b/tests/automake.mk
>> index c519a5cb36..ead92ace0f 100644
>> --- a/tests/automake.mk
>> +++ b/tests/automake.mk
>> @@ -188,6 +188,7 @@ SYSTEM_OFFLOADS_TESTSUITE_AT = \
>>
>> SYSTEM_DPDK_TESTSUITE_AT = \
>> tests/system-common-macros.at \
>> +tests/system-dpdk-find-device.py \
>> tests/system-dpdk-macros.at \
>> tests/system-dpdk-testsuite.at \
>> tests/system-dpdk.at
>> diff --git a/tests/system-dpdk-find-device.py
>> b/tests/system-dpdk-find-device.py
>> new file mode 100755
>> index 00..4253326e75
>> --- /dev/null
>> +++ b/tests/system-dpdk-find-device.py
>> @@ -0,0 +1,44 @@
>> +#!/usr/bin/env python3
Forgot to complain about the this, i.e. “#!/usr/bin/env python3” ;)
>> +# Copyright (c) 2021 Red Hat, Inc.
>> +#
>> +# Licensed under the Apache License, Version 2.0 (the "License");
>> +# you may not use this file except in compliance with the License.
>> +# You may obtain a copy of the License at:
>> +#
>> +# http://www.apache.org/licenses/LICENSE-2.0
>> +#
>> +# Unless required by applicable law or agreed to in writing, software
>> +# distributed under the License is distributed on an "AS IS" BASIS,
>> +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> +# See the License for the specific language governing permissions and
>> +# limitations under the License.
>> +
>> +
>> +from pathlib import Path
>> +import os
>> +import sys
>> +
>> +# The tester might want to select a PCI device, if so, trust it.
>> +if 'DPDK_PCI_ADDR' in os.environ:
>> +print(os.environ['DPDK_PCI_ADDR'])
>> +sys.exit(0)
>> +
>> +mlx5_ib_available = Path('/sys/module/mlx5_ib').exists()
>> +
>> +for device in sorted(Path('/sys/bus/pci/devices').iterdir()):
>> +class_path = device / 'class'
>> +# Only consider Network class devices
>> +if class_path.read_text().strip() != '0x02':
>> +continue
>> +kmod_path = device / 'driver' / 'module'
>> +kmod_name = kmod_path.resolve().name
>> +# Devices of interest:
>> +# - device is bound to vfio_pci or igb_uio,
>> +# - device is bound to mlx5_core and mlx5 is loaded,
>> +if (kmod_name not in ['vfio_pci', 'igb_uio'] and
>> +(kmod_name not in ['mlx5_core'] or not mlx5_ib_available)):
>
> Flake8 is complaining about this. But for some reason, I like this better ;)
> Quick online search reviews unclear interpretations, so I would leave it as
> is…
>
> system-dpdk-find-device.py:39:9: E129 visually indented line with same indent
> as next logical line
>
>> +continue
>> +print(device.resolve().name)
>> +sys.exit(0)
>> +
>> +sys.exit(1)
>> diff --git a/tests/system-dpdk-macros.at b/tests/system-dpdk-macros.at
>> index b3614f0565..066fae0ba5 100644
>> --- a/tests/system-dpdk-macros.at
>> +++ b/tests/system-dpdk-macros.at
>> @@ -22,14 +22,8 @@
Re: [ovs-dev] [PATCH v2 3/3] system-dpdk: Test with mlx5 devices.
On 23 Nov 2021, at 15:15, David Marchand wrote:
> The DPDK unit test only runs if vfio or igb_uio kernel module is loaded:
> on systems with only mlx5, this test is always skipped.
>
> Besides, the test tries to grab the first device listed by dpdk-devbind.py,
> regardless of the PCI device status regarding kmod binding.
>
> Remove dependency on this DPDK script and use a minimal script that
> reads PCI sysfs.
>
> This script is not perfect, as one can imagine PCI devices bound to
> vfio-pci for virtual machines.
> Add a new environment variable DPDK_PCI_ADDR for testers to select the
> PCI device of their liking.
> For consistency and grep, the temporary file PCI_ADDR is renamed
> to DPDK_PCI_ADDR.
>
> Note: with mlx5 devices, there is now more OVS/DPDK warnings to waive.
Uncovered some build issues with my scripts as now my MLX5 card in the system
is used :)
Changes look good to me, one small nit.
Acked-by: Eelco Chaudron
> Signed-off-by: David Marchand
> ---
> Documentation/topics/testing.rst | 1 -
> tests/automake.mk| 1 +
> tests/system-dpdk-find-device.py | 44
> tests/system-dpdk-macros.at | 10 ++--
> tests/system-dpdk.at | 4 ++-
> 5 files changed, 50 insertions(+), 10 deletions(-)
> create mode 100755 tests/system-dpdk-find-device.py
>
> diff --git a/Documentation/topics/testing.rst
> b/Documentation/topics/testing.rst
> index 0ddcbd58ab..00f734a77a 100644
> --- a/Documentation/topics/testing.rst
> +++ b/Documentation/topics/testing.rst
> @@ -343,7 +343,6 @@ To see a list of all the available tests, run::
>
> These tests support a `DPDK supported NIC`_. The tests operate on a wider
> set of
> environments, for instance, when a virtual port is used.
> -They do require proper DPDK variables (``DPDK_DIR`` and ``DPDK_BUILD``).
> Moreover you need to have root privileges to load the required modules and
> to bind
> the NIC to the DPDK-compatible driver.
>
> diff --git a/tests/automake.mk b/tests/automake.mk
> index c519a5cb36..ead92ace0f 100644
> --- a/tests/automake.mk
> +++ b/tests/automake.mk
> @@ -188,6 +188,7 @@ SYSTEM_OFFLOADS_TESTSUITE_AT = \
>
> SYSTEM_DPDK_TESTSUITE_AT = \
> tests/system-common-macros.at \
> + tests/system-dpdk-find-device.py \
> tests/system-dpdk-macros.at \
> tests/system-dpdk-testsuite.at \
> tests/system-dpdk.at
> diff --git a/tests/system-dpdk-find-device.py
> b/tests/system-dpdk-find-device.py
> new file mode 100755
> index 00..4253326e75
> --- /dev/null
> +++ b/tests/system-dpdk-find-device.py
> @@ -0,0 +1,44 @@
> +#!/usr/bin/env python3
> +# Copyright (c) 2021 Red Hat, Inc.
> +#
> +# Licensed under the Apache License, Version 2.0 (the "License");
> +# you may not use this file except in compliance with the License.
> +# You may obtain a copy of the License at:
> +#
> +# http://www.apache.org/licenses/LICENSE-2.0
> +#
> +# Unless required by applicable law or agreed to in writing, software
> +# distributed under the License is distributed on an "AS IS" BASIS,
> +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> +# See the License for the specific language governing permissions and
> +# limitations under the License.
> +
> +
> +from pathlib import Path
> +import os
> +import sys
> +
> +# The tester might want to select a PCI device, if so, trust it.
> +if 'DPDK_PCI_ADDR' in os.environ:
> +print(os.environ['DPDK_PCI_ADDR'])
> +sys.exit(0)
> +
> +mlx5_ib_available = Path('/sys/module/mlx5_ib').exists()
> +
> +for device in sorted(Path('/sys/bus/pci/devices').iterdir()):
> +class_path = device / 'class'
> +# Only consider Network class devices
> +if class_path.read_text().strip() != '0x02':
> +continue
> +kmod_path = device / 'driver' / 'module'
> +kmod_name = kmod_path.resolve().name
> +# Devices of interest:
> +# - device is bound to vfio_pci or igb_uio,
> +# - device is bound to mlx5_core and mlx5 is loaded,
> +if (kmod_name not in ['vfio_pci', 'igb_uio'] and
> +(kmod_name not in ['mlx5_core'] or not mlx5_ib_available)):
Flake8 is complaining about this. But for some reason, I like this better ;)
Quick online search reviews unclear interpretations, so I would leave it as is…
system-dpdk-find-device.py:39:9: E129 visually indented line with same indent
as next logical line
> +continue
> +print(device.resolve().name)
> +sys.exit(0)
> +
> +sys.exit(1)
> diff --git a/tests/system-dpdk-macros.at b/tests/system-dpdk-macros.at
> index b3614f0565..066fae0ba5 100644
> --- a/tests/system-dpdk-macros.at
> +++ b/tests/system-dpdk-macros.at
> @@ -22,14 +22,8 @@ m4_define([OVS_DPDK_PRE_PHY_SKIP],
>[dnl Perform the precheck
> OVS_DPDK_PRE_CHECK()
>
> - dnl Check if VFIO or UIO driver is loaded
> - AT_SKIP_IF([ ! (lsmod | grep -E "igb_uio|vfio") ], [], [stdout])
> -
> - dnl Find PCI address candidate, skip if there is no
[ovs-dev] [PATCH v2 3/3] system-dpdk: Test with mlx5 devices.
The DPDK unit test only runs if vfio or igb_uio kernel module is loaded:
on systems with only mlx5, this test is always skipped.
Besides, the test tries to grab the first device listed by dpdk-devbind.py,
regardless of the PCI device status regarding kmod binding.
Remove dependency on this DPDK script and use a minimal script that
reads PCI sysfs.
This script is not perfect, as one can imagine PCI devices bound to
vfio-pci for virtual machines.
Add a new environment variable DPDK_PCI_ADDR for testers to select the
PCI device of their liking.
For consistency and grep, the temporary file PCI_ADDR is renamed
to DPDK_PCI_ADDR.
Note: with mlx5 devices, there is now more OVS/DPDK warnings to waive.
Signed-off-by: David Marchand
---
Documentation/topics/testing.rst | 1 -
tests/automake.mk| 1 +
tests/system-dpdk-find-device.py | 44
tests/system-dpdk-macros.at | 10 ++--
tests/system-dpdk.at | 4 ++-
5 files changed, 50 insertions(+), 10 deletions(-)
create mode 100755 tests/system-dpdk-find-device.py
diff --git a/Documentation/topics/testing.rst b/Documentation/topics/testing.rst
index 0ddcbd58ab..00f734a77a 100644
--- a/Documentation/topics/testing.rst
+++ b/Documentation/topics/testing.rst
@@ -343,7 +343,6 @@ To see a list of all the available tests, run::
These tests support a `DPDK supported NIC`_. The tests operate on a wider set
of
environments, for instance, when a virtual port is used.
-They do require proper DPDK variables (``DPDK_DIR`` and ``DPDK_BUILD``).
Moreover you need to have root privileges to load the required modules and to
bind
the NIC to the DPDK-compatible driver.
diff --git a/tests/automake.mk b/tests/automake.mk
index c519a5cb36..ead92ace0f 100644
--- a/tests/automake.mk
+++ b/tests/automake.mk
@@ -188,6 +188,7 @@ SYSTEM_OFFLOADS_TESTSUITE_AT = \
SYSTEM_DPDK_TESTSUITE_AT = \
tests/system-common-macros.at \
+ tests/system-dpdk-find-device.py \
tests/system-dpdk-macros.at \
tests/system-dpdk-testsuite.at \
tests/system-dpdk.at
diff --git a/tests/system-dpdk-find-device.py b/tests/system-dpdk-find-device.py
new file mode 100755
index 00..4253326e75
--- /dev/null
+++ b/tests/system-dpdk-find-device.py
@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+# Copyright (c) 2021 Red Hat, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from pathlib import Path
+import os
+import sys
+
+# The tester might want to select a PCI device, if so, trust it.
+if 'DPDK_PCI_ADDR' in os.environ:
+print(os.environ['DPDK_PCI_ADDR'])
+sys.exit(0)
+
+mlx5_ib_available = Path('/sys/module/mlx5_ib').exists()
+
+for device in sorted(Path('/sys/bus/pci/devices').iterdir()):
+class_path = device / 'class'
+# Only consider Network class devices
+if class_path.read_text().strip() != '0x02':
+continue
+kmod_path = device / 'driver' / 'module'
+kmod_name = kmod_path.resolve().name
+# Devices of interest:
+# - device is bound to vfio_pci or igb_uio,
+# - device is bound to mlx5_core and mlx5 is loaded,
+if (kmod_name not in ['vfio_pci', 'igb_uio'] and
+(kmod_name not in ['mlx5_core'] or not mlx5_ib_available)):
+continue
+print(device.resolve().name)
+sys.exit(0)
+
+sys.exit(1)
diff --git a/tests/system-dpdk-macros.at b/tests/system-dpdk-macros.at
index b3614f0565..066fae0ba5 100644
--- a/tests/system-dpdk-macros.at
+++ b/tests/system-dpdk-macros.at
@@ -22,14 +22,8 @@ m4_define([OVS_DPDK_PRE_PHY_SKIP],
[dnl Perform the precheck
OVS_DPDK_PRE_CHECK()
- dnl Check if VFIO or UIO driver is loaded
- AT_SKIP_IF([ ! (lsmod | grep -E "igb_uio|vfio") ], [], [stdout])
-
- dnl Find PCI address candidate, skip if there is no DPDK-compatible NIC
- AT_CHECK([$DPDK_DIR/usertools/dpdk-devbind.py -s | head -n +4 | tail -1],
[], [stdout])
- AT_CHECK([cat stdout | cut -d" " -s -f1 > PCI_ADDR])
- AT_SKIP_IF([ ! test -s PCI_ADDR ])
-
+ dnl Check if a device is available for DPDK
+ AT_SKIP_IF([ ! $abs_top_srcdir/tests/system-dpdk-find-device.py >
DPDK_PCI_ADDR])
])
diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
index 7b5d063f62..894e01316f 100644
--- a/tests/system-dpdk.at
+++ b/tests/system-dpdk.at
@@ -7,6 +7,8 @@ m4_define([SYSTEM_DPDK_ALLOWED_LOGS],[
\@does not exist. The Open vSwitch kernel module is probably not loaded.@d
\@EAL: Invalid NUMA socket, default to 0@d
\@EAL: No
