Re: [ovs-dev] [PATCH ovn v2 2/3] ci: Build container image before very job

2023-11-22 Thread Ales Musil 
On Wed, Nov 22, 2023 at 2:28 PM Ales Musil  wrote:

> Build the image before every job to allow more
> fine-grained dependency pinning. This is especially
> useful for stable branches as they might need to stay
> on specific distribution or Python version.
>
> Signed-off-by: Ales Musil 
> ---
>  .cirrus.yml| 40 -
>  .github/workflows/test.yml | 46 +++---
>  2 files changed, 77 insertions(+), 9 deletions(-)
>
> diff --git a/.cirrus.yml b/.cirrus.yml
> index 71e9afbb6..894f953f3 100644
> --- a/.cirrus.yml
> +++ b/.cirrus.yml
> @@ -1,9 +1,25 @@
> -arm_unit_tests_task:
> +compute_engine_instance:
> +  image_project: ubuntu-os-cloud
> +  image: family/ubuntu-2304-amd64
>

there is still one issues, it should be:
  image: family/ubuntu-2304-arm64
  architecture: arm64


I'll wait with v3 for some reviews/next week.


> +  platform: linux
> +  memory: 4G
> +
> +build_image_task:
> +  install_dependencies_script:
> +- sudo apt update
> +- sudo apt install -y podman make
> +
> +  build_container_script:
> +- cd utilities/containers
> +- make ubuntu
> +- podman save -o /tmp/image.tar ovn-org/ovn-tests:ubuntu
> +
> +  upload_image_script:
> +- curl -s -X POST -T /tmp/image.tar http://
> $CIRRUS_HTTP_CACHE_HOST/${CIRRUS_CHANGE_IN_REPO}
>
> -  arm_container:
> -image: ghcr.io/ovn-org/ovn-tests:ubuntu
> -memory: 4G
> -cpu: 2
> +arm_unit_tests_task:
> +  depends_on:
> +- build_image
>
>env:
>  ARCH: aarch64
> @@ -11,6 +27,7 @@ arm_unit_tests_task:
>  PATH: ${HOME}/bin:${HOME}/.local/bin:${PATH}
>  RECHECK: yes
>  LANG: C
> +IMAGE_NAME: ovn-org/ovn-tests:ubuntu
>  matrix:
>- CC: gcc
>  TESTSUITE: test
> @@ -35,5 +52,16 @@ arm_unit_tests_task:
>
>name: ARM64 ${CC} ${TESTSUITE} ${TEST_RANGE}
>
> +  install_dependencies_script:
> +- sudo apt update
> +- sudo apt install -y podman
> +
> +  download_cache_script:
> +- curl http://$CIRRUS_HTTP_CACHE_HOST/${CIRRUS_CHANGE_IN_REPO} -o
> /tmp/image.tar
> +
> +  load_image_script:
> +- podman load -i /tmp/image.tar
> +- rm -rf /tmp/image.tar
> +
>build_script:
> -- ./.ci/linux-build.sh
> +- ./.ci/ci.sh --archive-logs
> diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
> index b3fcb57fc..fdbc8f5f5 100644
> --- a/.github/workflows/test.yml
> +++ b/.github/workflows/test.yml
> @@ -80,10 +80,38 @@ jobs:
>if: steps.dpdk_cache.outputs.cache-hit != 'true'
>run: ./.ci/dpdk-build.sh
>
> +  prepare-container:
> +env:
> +  DEPENDENCIES: podman
> +name: Prepare container
> +runs-on: ubuntu-22.04
> +
> +steps:
> +  - uses: actions/checkout@v3
> +
> +  - name: Update APT cache
> +run: sudo apt update
> +
> +  - name: Install dependencies
> +run: sudo apt install -y ${{ env.DEPENDENCIES }}
> +
> +  - name: Build container
> +run: make ubuntu
> +working-directory: utilities/containers
> +
> +  - name: Export image
> +run: podman save -o /tmp/image.tar ovn-org/ovn-tests
> +
> +  - name: Cache image
> +id: image_cache
> +uses: actions/cache@v3
> +with:
> +  path: /tmp/image.tar
> +  key: ${{ github.sha }}
> +
>build-linux:
> -needs: build-dpdk
> +needs: [build-dpdk, prepare-container]
>  env:
> -  IMAGE_NAME:  ghcr.io/ovn-org/ovn-tests:ubuntu
>ARCH:${{ matrix.cfg.arch }}
>CC:  ${{ matrix.cfg.compiler }}
>DPDK:${{ matrix.cfg.dpdk }}
> @@ -152,13 +180,25 @@ jobs:
>  sort -V | tail -1)
>working-directory: ovs
>
> -- name: cache
> +- name: cache dpdk
>if: matrix.cfg.dpdk != ''
>uses: actions/cache@v3
>with:
>  path: dpdk-dir
>  key: ${{ needs.build-dpdk.outputs.dpdk_key }}
>
> +- name: image cache
> +  uses: actions/cache@v3
> +  with:
> +path: /tmp/image.tar
> +key: ${{ github.sha }}
> +
> +- name: load image
> +  run: |
> +sudo podman load -i /tmp/image.tar
> +podman load -i /tmp/image.tar
> +rm -rf /tmp/image.tar
> +
>  - name: build
>if: ${{ startsWith(matrix.cfg.testsuite, 'system-test') }}
>run: sudo -E ./.ci/ci.sh --archive-logs
> --
> 2.41.0
>
>

-- 

Ales Musil

Senior Software Engineer - OVN Core

Red Hat EMEA 

amu...@redhat.com

___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

[ovs-dev] [PATCH v5] system-dpdk: Test with mlx5 devices.

2023-11-22 Thread David Marchand 
The DPDK unit test only runs if vfio or igb_uio kernel modules are loaded:
on systems with only mlx5, this test is always skipped.

Besides, the test tries to grab the first device listed by dpdk-devbind.py,
regardless of the PCI device status regarding kmod binding.

Remove dependency on this DPDK script and use a minimal script that
reads PCI sysfs.

This script is not perfect, as one can imagine PCI devices bound to
vfio-pci for virtual machines.
Plus, this script only tries to take over vfio-pci devices. mlx5 devices
can't be taken over blindly as it could mean losing connectivity to the
machine if the netdev was in use for this system.

For those two reasons, add a new environment variable DPDK_PCI_ADDR for
testers to select the PCI device of their liking.
For consistency and grep, the temporary file PCI_ADDR is renamed
to DPDK_PCI_ADDR.

Reviewed-by: Maxime Coquelin 
Acked-by: Eelco Chaudron 
Signed-off-by: David Marchand 
---
Changes since v4:
- separated from the original series,
- rebased,
- dropped mlx5 devices from the discovery script,
- documented DPDK_PCI_ADDR env variable,

Changes since v3:
- fixed nit from Maxime,

Changes since v2:
- sorted logs alphabetically,

---
 Documentation/topics/testing.rst | 11 ++---
 tests/automake.mk|  1 +
 tests/system-dpdk-find-device.py | 39 
 tests/system-dpdk-macros.at  | 10 ++--
 tests/system-dpdk.at | 14 ++--
 5 files changed, 57 insertions(+), 18 deletions(-)
 create mode 100755 tests/system-dpdk-find-device.py

diff --git a/Documentation/topics/testing.rst b/Documentation/topics/testing.rst
index 5f6940b84d..fb9b3e77b1 100644
--- a/Documentation/topics/testing.rst
+++ b/Documentation/topics/testing.rst
@@ -343,15 +343,20 @@ To see a list of all the available tests, run::
 
 These tests support a `DPDK supported NIC`_. The tests operate on a wider set 
of
 environments, for instance, when a virtual port is used.
-They do require proper DPDK variables (``DPDK_DIR`` and ``DPDK_BUILD``).
 Moreover you need to have root privileges to load the required modules and to 
bind
-the NIC to the DPDK-compatible driver.
+a PCI device to the DPDK-compatible driver.
 
 .. _DPDK supported NIC: https://core.dpdk.org/supported/#nics
 
+The phy test will skip if no suitable PCI device is found.
+It is possible to select which PCI device is used for this test by setting the
+DPDK_PCI_ADDR environment variable, which is especially useful when testing
+with a mlx5 device::
+
+# DPDK_PCI_ADDR=:82:00.0 make check-dpdk
+
 All tests are skipped if no hugepages are configured. User must look into the 
DPDK
 manual to figure out how to `Configure hugepages`_.
-The phy test will skip if no compatible physical device is available.
 
 .. _Configure hugepages: 
https://doc.dpdk.org/guides-22.11/linux_gsg/sys_reqs.html
 
diff --git a/tests/automake.mk b/tests/automake.mk
index f8a925012d..69117514cc 100644
--- a/tests/automake.mk
+++ b/tests/automake.mk
@@ -192,6 +192,7 @@ SYSTEM_OFFLOADS_TESTSUITE_AT = \
 
 SYSTEM_DPDK_TESTSUITE_AT = \
tests/system-common-macros.at \
+   tests/system-dpdk-find-device.py \
tests/system-dpdk-macros.at \
tests/system-dpdk-testsuite.at \
tests/system-dpdk.at
diff --git a/tests/system-dpdk-find-device.py b/tests/system-dpdk-find-device.py
new file mode 100755
index 00..b0997380bc
--- /dev/null
+++ b/tests/system-dpdk-find-device.py
@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+# Copyright (c) 2021 Red Hat, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from pathlib import Path
+import os
+import sys
+
+# The tester might want to select a PCI device, if so, trust it.
+if 'DPDK_PCI_ADDR' in os.environ:
+print(os.environ['DPDK_PCI_ADDR'])
+sys.exit(0)
+
+for device in sorted(Path('/sys/bus/pci/devices').iterdir()):
+class_path = device / 'class'
+# Only consider Network class devices
+if class_path.read_text().strip() != '0x02':
+continue
+kmod_path = device / 'driver' / 'module'
+kmod_name = kmod_path.resolve().name
+# Only care about devices bound to vfio_pci or igb_uio.
+if kmod_name not in ['vfio_pci', 'igb_uio']:
+continue
+print(device.resolve().name)
+sys.exit(0)
+
+sys.exit(1)
diff --git a/tests/system-dpdk-macros.at b/tests/system-dpdk-macros.at
index dcdfa55741..3b5a3512d4 100644
--- a/tests/system-dpdk-macros.at
+++ b/tests/system-dpdk-macros.at
@

[ovs-dev] [PATCH ovn v2 3/3] ci: Cover more container posibilities

2023-11-22 Thread Ales Musil 
Add more conditions to the image prepare process.
This allows us to test the prebuilt images for both
Fedora and Ubuntu.

Run the weekly image on main branch with the
use of Fedora for the weekly runs.

Reported-at: https://issues.redhat.com/browse/FDP-149
Signed-off-by: Ales Musil 
---
 .github/workflows/test.yml | 17 -
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index fdbc8f5f5..60afcba77 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -95,10 +95,25 @@ jobs:
   - name: Install dependencies
 run: sudo apt install -y ${{ env.DEPENDENCIES }}
 
+  - name: Choose image distro
+if: github.event_name == 'push' || github.event_name == 'pull_request'
+run: |
+  echo "IMAGE_DISTRO=ubuntu" >> $GITHUB_ENV
+
+  - name: Choose image distro
+if: github.event_name == 'schedule'
+run: |
+  echo "IMAGE_DISTRO=fedora" >> $GITHUB_ENV
+
   - name: Build container
-run: make ubuntu
+if: github.ref_name != 'main'
+run: make ${{ env.IMAGE_DISTRO }}
 working-directory: utilities/containers
 
+  - name: Download container
+if: github.ref_name == 'main'
+run: podman pull ghcr.io/ovn-org/ovn-tests:${{ env.IMAGE_DISTRO }}
+
   - name: Export image
 run: podman save -o /tmp/image.tar ovn-org/ovn-tests
 
-- 
2.41.0

___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

[ovs-dev] [PATCH ovn v2 2/3] ci: Build container image before very job

2023-11-22 Thread Ales Musil 
Build the image before every job to allow more
fine-grained dependency pinning. This is especially
useful for stable branches as they might need to stay
on specific distribution or Python version.

Signed-off-by: Ales Musil 
---
 .cirrus.yml| 40 -
 .github/workflows/test.yml | 46 +++---
 2 files changed, 77 insertions(+), 9 deletions(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index 71e9afbb6..894f953f3 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -1,9 +1,25 @@
-arm_unit_tests_task:
+compute_engine_instance:
+  image_project: ubuntu-os-cloud
+  image: family/ubuntu-2304-amd64
+  platform: linux
+  memory: 4G
+
+build_image_task:
+  install_dependencies_script:
+- sudo apt update
+- sudo apt install -y podman make
+
+  build_container_script:
+- cd utilities/containers
+- make ubuntu
+- podman save -o /tmp/image.tar ovn-org/ovn-tests:ubuntu
+
+  upload_image_script:
+- curl -s -X POST -T /tmp/image.tar 
http://$CIRRUS_HTTP_CACHE_HOST/${CIRRUS_CHANGE_IN_REPO}
 
-  arm_container:
-image: ghcr.io/ovn-org/ovn-tests:ubuntu
-memory: 4G
-cpu: 2
+arm_unit_tests_task:
+  depends_on:
+- build_image
 
   env:
 ARCH: aarch64
@@ -11,6 +27,7 @@ arm_unit_tests_task:
 PATH: ${HOME}/bin:${HOME}/.local/bin:${PATH}
 RECHECK: yes
 LANG: C
+IMAGE_NAME: ovn-org/ovn-tests:ubuntu
 matrix:
   - CC: gcc
 TESTSUITE: test
@@ -35,5 +52,16 @@ arm_unit_tests_task:
 
   name: ARM64 ${CC} ${TESTSUITE} ${TEST_RANGE}
 
+  install_dependencies_script:
+- sudo apt update
+- sudo apt install -y podman
+
+  download_cache_script:
+- curl http://$CIRRUS_HTTP_CACHE_HOST/${CIRRUS_CHANGE_IN_REPO} -o 
/tmp/image.tar
+
+  load_image_script:
+- podman load -i /tmp/image.tar
+- rm -rf /tmp/image.tar
+
   build_script:
-- ./.ci/linux-build.sh
+- ./.ci/ci.sh --archive-logs
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index b3fcb57fc..fdbc8f5f5 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -80,10 +80,38 @@ jobs:
   if: steps.dpdk_cache.outputs.cache-hit != 'true'
   run: ./.ci/dpdk-build.sh
 
+  prepare-container:
+env:
+  DEPENDENCIES: podman
+name: Prepare container
+runs-on: ubuntu-22.04
+
+steps:
+  - uses: actions/checkout@v3
+
+  - name: Update APT cache
+run: sudo apt update
+
+  - name: Install dependencies
+run: sudo apt install -y ${{ env.DEPENDENCIES }}
+
+  - name: Build container
+run: make ubuntu
+working-directory: utilities/containers
+
+  - name: Export image
+run: podman save -o /tmp/image.tar ovn-org/ovn-tests
+
+  - name: Cache image
+id: image_cache
+uses: actions/cache@v3
+with:
+  path: /tmp/image.tar
+  key: ${{ github.sha }}
+
   build-linux:
-needs: build-dpdk
+needs: [build-dpdk, prepare-container]
 env:
-  IMAGE_NAME:  ghcr.io/ovn-org/ovn-tests:ubuntu
   ARCH:${{ matrix.cfg.arch }}
   CC:  ${{ matrix.cfg.compiler }}
   DPDK:${{ matrix.cfg.dpdk }}
@@ -152,13 +180,25 @@ jobs:
 sort -V | tail -1)
   working-directory: ovs
 
-- name: cache
+- name: cache dpdk
   if: matrix.cfg.dpdk != ''
   uses: actions/cache@v3
   with:
 path: dpdk-dir
 key: ${{ needs.build-dpdk.outputs.dpdk_key }}
 
+- name: image cache
+  uses: actions/cache@v3
+  with:
+path: /tmp/image.tar
+key: ${{ github.sha }}
+
+- name: load image
+  run: |
+sudo podman load -i /tmp/image.tar
+podman load -i /tmp/image.tar
+rm -rf /tmp/image.tar
+
 - name: build
   if: ${{ startsWith(matrix.cfg.testsuite, 'system-test') }}
   run: sudo -E ./.ci/ci.sh --archive-logs
-- 
2.41.0

___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

[ovs-dev] [PATCH ovn v2 1/3] ci: Switch Cirrus CI to the Ubuntu image

2023-11-22 Thread Ales Musil 
Switch the Cirrus CI to use the Ubuntu image,
so it is unified with the GH actions. That is
useful for various dependency pinnings.

Signed-off-by: Ales Musil 
---
 .cirrus.yml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index 2d3824972..71e9afbb6 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -1,7 +1,7 @@
 arm_unit_tests_task:
 
   arm_container:
-image: ghcr.io/ovn-org/ovn-tests:fedora
+image: ghcr.io/ovn-org/ovn-tests:ubuntu
 memory: 4G
 cpu: 2
 
@@ -10,6 +10,7 @@ arm_unit_tests_task:
 CIRRUS_CLONE_SUBMODULES: true
 PATH: ${HOME}/bin:${HOME}/.local/bin:${PATH}
 RECHECK: yes
+LANG: C
 matrix:
   - CC: gcc
 TESTSUITE: test
-- 
2.41.0

___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

[ovs-dev] [PATCH ovn v2 0/3] Allow to use different container images per branch

2023-11-22 Thread Ales Musil 
This series allows different branches to use different images.
This is needed for stabikity reasons as it is hard to keep up
with new Ubuntu/Fedora releases for all stable branches.

With the last patch we will still use the prebuilt images
for main branch which is completely fine.

Ales Musil (3):
  ci: Switch Cirrus CI to the Ubuntu image
  ci: Build container image before very job
  ci: Cover more container posibilities

 .cirrus.yml| 41 +
 .github/workflows/test.yml | 61 --
 2 files changed, 93 insertions(+), 9 deletions(-)

-- 
2.41.0

___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [ovs-dev] [PATCH ovn 0/3] Allow to use different container images per branch

2023-11-22 Thread Ales Musil 
Sorry for the nosie, there is a wrong branch reference on 2/3 I'll post v2.

On Wed, Nov 22, 2023 at 2:25 PM Ales Musil  wrote:

> This series allows different branches to use different images.
> This is needed for stabikity reasons as it is hard to keep up
> with new Ubuntu/Fedora releases for all stable branches.
>
> With the last patch we will still use the prebuilt images
> for main branch which is completely fine.
>
> Ales Musil (3):
>   ci: Switch Cirrus CI to the Ubuntu image
>   ci: Build container image before very job
>   ci: Cover more container posibilities
>
>  .cirrus.yml| 41 +
>  .github/workflows/test.yml | 61 --
>  2 files changed, 93 insertions(+), 9 deletions(-)
>
> --
> 2.41.0
>
>

-- 

Ales Musil

Senior Software Engineer - OVN Core

Red Hat EMEA 

amu...@redhat.com

___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

[ovs-dev] [PATCH ovn 2/3] ci: Build container image before very job

2023-11-22 Thread Ales Musil 
Build the image before every job to allow more
fine-grained dependency pinning. This is especially
useful for stable branches as they might need to stay
on specific distribution or Python version.

Signed-off-by: Ales Musil 
---
 .cirrus.yml| 40 +-
 .github/workflows/test.yml | 51 +++---
 2 files changed, 82 insertions(+), 9 deletions(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index 71e9afbb6..894f953f3 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -1,9 +1,25 @@
-arm_unit_tests_task:
+compute_engine_instance:
+  image_project: ubuntu-os-cloud
+  image: family/ubuntu-2304-amd64
+  platform: linux
+  memory: 4G
+
+build_image_task:
+  install_dependencies_script:
+- sudo apt update
+- sudo apt install -y podman make
+
+  build_container_script:
+- cd utilities/containers
+- make ubuntu
+- podman save -o /tmp/image.tar ovn-org/ovn-tests:ubuntu
+
+  upload_image_script:
+- curl -s -X POST -T /tmp/image.tar 
http://$CIRRUS_HTTP_CACHE_HOST/${CIRRUS_CHANGE_IN_REPO}
 
-  arm_container:
-image: ghcr.io/ovn-org/ovn-tests:ubuntu
-memory: 4G
-cpu: 2
+arm_unit_tests_task:
+  depends_on:
+- build_image
 
   env:
 ARCH: aarch64
@@ -11,6 +27,7 @@ arm_unit_tests_task:
 PATH: ${HOME}/bin:${HOME}/.local/bin:${PATH}
 RECHECK: yes
 LANG: C
+IMAGE_NAME: ovn-org/ovn-tests:ubuntu
 matrix:
   - CC: gcc
 TESTSUITE: test
@@ -35,5 +52,16 @@ arm_unit_tests_task:
 
   name: ARM64 ${CC} ${TESTSUITE} ${TEST_RANGE}
 
+  install_dependencies_script:
+- sudo apt update
+- sudo apt install -y podman
+
+  download_cache_script:
+- curl http://$CIRRUS_HTTP_CACHE_HOST/${CIRRUS_CHANGE_IN_REPO} -o 
/tmp/image.tar
+
+  load_image_script:
+- podman load -i /tmp/image.tar
+- rm -rf /tmp/image.tar
+
   build_script:
-- ./.ci/linux-build.sh
+- ./.ci/ci.sh --archive-logs
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index b3fcb57fc..9fc4d3dbc 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -80,10 +80,43 @@ jobs:
   if: steps.dpdk_cache.outputs.cache-hit != 'true'
   run: ./.ci/dpdk-build.sh
 
+  prepare-container:
+env:
+  DEPENDENCIES: podman
+name: Prepare container
+runs-on: ubuntu-22.04
+
+steps:
+  - uses: actions/checkout@v3
+
+  - name: Update APT cache
+run: sudo apt update
+
+  - name: Install dependencies
+run: sudo apt install -y ${{ env.DEPENDENCIES }}
+
+  - name: Build container
+if: github.ref_name != 'ci_stabilization'
+run: make ubuntu
+working-directory: utilities/containers
+
+  - name: Download container
+if: github.ref_name == 'ci_stabilization'
+run: podman pull ghcr.io/ovn-org/ovn-tests:ubuntu
+
+  - name: Export image
+run: podman save -o /tmp/image.tar ovn-org/ovn-tests
+
+  - name: Cache image
+id: image_cache
+uses: actions/cache@v3
+with:
+  path: /tmp/image.tar
+  key: ${{ github.sha }}
+
   build-linux:
-needs: build-dpdk
+needs: [build-dpdk, prepare-container]
 env:
-  IMAGE_NAME:  ghcr.io/ovn-org/ovn-tests:ubuntu
   ARCH:${{ matrix.cfg.arch }}
   CC:  ${{ matrix.cfg.compiler }}
   DPDK:${{ matrix.cfg.dpdk }}
@@ -152,13 +185,25 @@ jobs:
 sort -V | tail -1)
   working-directory: ovs
 
-- name: cache
+- name: cache dpdk
   if: matrix.cfg.dpdk != ''
   uses: actions/cache@v3
   with:
 path: dpdk-dir
 key: ${{ needs.build-dpdk.outputs.dpdk_key }}
 
+- name: image cache
+  uses: actions/cache@v3
+  with:
+path: /tmp/image.tar
+key: ${{ github.sha }}
+
+- name: load image
+  run: |
+sudo podman load -i /tmp/image.tar
+podman load -i /tmp/image.tar
+rm -rf /tmp/image.tar
+
 - name: build
   if: ${{ startsWith(matrix.cfg.testsuite, 'system-test') }}
   run: sudo -E ./.ci/ci.sh --archive-logs
-- 
2.41.0

___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

[ovs-dev] [PATCH ovn 1/3] ci: Switch Cirrus CI to the Ubuntu image

2023-11-22 Thread Ales Musil 
Switch the Cirrus CI to use the Ubuntu image,
so it is unified with the GH actions. That is
useful for various dependency pinnings.

Signed-off-by: Ales Musil 
---
 .cirrus.yml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index 2d3824972..71e9afbb6 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -1,7 +1,7 @@
 arm_unit_tests_task:
 
   arm_container:
-image: ghcr.io/ovn-org/ovn-tests:fedora
+image: ghcr.io/ovn-org/ovn-tests:ubuntu
 memory: 4G
 cpu: 2
 
@@ -10,6 +10,7 @@ arm_unit_tests_task:
 CIRRUS_CLONE_SUBMODULES: true
 PATH: ${HOME}/bin:${HOME}/.local/bin:${PATH}
 RECHECK: yes
+LANG: C
 matrix:
   - CC: gcc
 TESTSUITE: test
-- 
2.41.0

___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

[ovs-dev] [PATCH ovn 3/3] ci: Cover more container posibilities

2023-11-22 Thread Ales Musil 
Add more conditions to the image prepare process.
This allows us to test the prebuilt images for both
Fedora and Ubuntu.

Run the weekly image on main branch with the
use of Fedora for the weekly runs.

Reported-at: https://issues.redhat.com/browse/FDP-149
Signed-off-by: Ales Musil 
---
 .github/workflows/test.yml | 18 ++
 1 file changed, 14 insertions(+), 4 deletions(-)

diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 9fc4d3dbc..60afcba77 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -95,14 +95,24 @@ jobs:
   - name: Install dependencies
 run: sudo apt install -y ${{ env.DEPENDENCIES }}
 
+  - name: Choose image distro
+if: github.event_name == 'push' || github.event_name == 'pull_request'
+run: |
+  echo "IMAGE_DISTRO=ubuntu" >> $GITHUB_ENV
+
+  - name: Choose image distro
+if: github.event_name == 'schedule'
+run: |
+  echo "IMAGE_DISTRO=fedora" >> $GITHUB_ENV
+
   - name: Build container
-if: github.ref_name != 'ci_stabilization'
-run: make ubuntu
+if: github.ref_name != 'main'
+run: make ${{ env.IMAGE_DISTRO }}
 working-directory: utilities/containers
 
   - name: Download container
-if: github.ref_name == 'ci_stabilization'
-run: podman pull ghcr.io/ovn-org/ovn-tests:ubuntu
+if: github.ref_name == 'main'
+run: podman pull ghcr.io/ovn-org/ovn-tests:${{ env.IMAGE_DISTRO }}
 
   - name: Export image
 run: podman save -o /tmp/image.tar ovn-org/ovn-tests
-- 
2.41.0

___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

[ovs-dev] [PATCH ovn 0/3] Allow to use different container images per branch

2023-11-22 Thread Ales Musil 
This series allows different branches to use different images.
This is needed for stabikity reasons as it is hard to keep up
with new Ubuntu/Fedora releases for all stable branches.

With the last patch we will still use the prebuilt images
for main branch which is completely fine.

Ales Musil (3):
  ci: Switch Cirrus CI to the Ubuntu image
  ci: Build container image before very job
  ci: Cover more container posibilities

 .cirrus.yml| 41 +
 .github/workflows/test.yml | 61 --
 2 files changed, 93 insertions(+), 9 deletions(-)

-- 
2.41.0

___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

[ovs-dev] [dpdk-latest 2/2] ci: Reduce optional libraries in DPDK.

2023-11-22 Thread David Marchand 
Since DPDK v23.11, it is possible to select more easily which optional
library is enabled.

OVS needs the vhost library (and its dependencies).
The net/tap DPDK driver needs the gso library.
Other optional library can be disabled.

This reduces the cache entry for DPDK from ~7MB to ~4MB.

Signed-off-by: David Marchand 
---
 .ci/dpdk-build.sh | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.ci/dpdk-build.sh b/.ci/dpdk-build.sh
index 8380c48637..698b9e1b14 100755
--- a/.ci/dpdk-build.sh
+++ b/.ci/dpdk-build.sh
@@ -42,6 +42,9 @@ function build_dpdk()
 DPDK_OPTS="$DPDK_OPTS -Denable_apps=test-pmd"
 enable_drivers="net/null,net/af_xdp,net/tap,net/virtio"
 DPDK_OPTS="$DPDK_OPTS -Denable_drivers=$enable_drivers"
+# OVS depends on the vhost library (and its dependencies).
+# net/tap depends on the gso library.
+DPDK_OPTS="$DPDK_OPTS -Denable_libs=cryptodev,dmadev,gso,vhost"
 
 # Install DPDK using prefix.
 DPDK_OPTS="$DPDK_OPTS --prefix=$DPDK_INSTALL_DIR"
-- 
2.41.0

___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

[ovs-dev] [dpdk-latest 1/2] ci: Cache DPDK installed libraries only.

2023-11-22 Thread David Marchand 
Rather than save the whole DPDK sources and build artefacts, checkout
sources in a separate directory and build DPDK there.
Only the installed artefacts are then going to the cache.
Example sources in the share/dpdk installed directory can be pruned too.

This makes a (v23.11-rc3) DPDK cache entry size go from ~120MB to ~7MB.

Signed-off-by: David Marchand 
---
 .ci/dpdk-build.sh  | 25 +++--
 .ci/linux-build.sh |  9 +
 2 files changed, 20 insertions(+), 14 deletions(-)

diff --git a/.ci/dpdk-build.sh b/.ci/dpdk-build.sh
index aa83e44643..8380c48637 100755
--- a/.ci/dpdk-build.sh
+++ b/.ci/dpdk-build.sh
@@ -5,25 +5,27 @@ set -x
 
 function build_dpdk()
 {
-local VERSION_FILE="dpdk-dir/cached-version"
 local DPDK_VER=$1
 local DPDK_OPTS=""
+local DPDK_INSTALL_DIR="$(pwd)/dpdk-dir"
+local VERSION_FILE="$DPDK_INSTALL_DIR/cached-version"
 
-rm -rf dpdk-dir
+rm -rf dpdk-src
+rm -rf $DPDK_INSTALL_DIR
 
 if [ "${DPDK_VER##refs/*/}" != "${DPDK_VER}" ]; then
-git clone --single-branch $DPDK_GIT dpdk-dir -b "${DPDK_VER##refs/*/}"
-pushd dpdk-dir
+git clone --single-branch $DPDK_GIT dpdk-src -b "${DPDK_VER##refs/*/}"
+pushd dpdk-src
 git log -1 --oneline
 else
 wget https://fast.dpdk.org/rel/dpdk-$1.tar.xz
 tar xvf dpdk-$1.tar.xz > /dev/null
 DIR_NAME=$(tar -tf dpdk-$1.tar.xz | head -1 | cut -f1 -d"/")
-mv ${DIR_NAME} dpdk-dir
-pushd dpdk-dir
+mv ${DIR_NAME} dpdk-src
+pushd dpdk-src
 fi
 
-# Switching to 'default' machine to make dpdk-dir cache usable on
+# Switching to 'default' machine to make the dpdk cache usable on
 # different CPUs. We can't be sure that all CI machines are exactly same.
 DPDK_OPTS="$DPDK_OPTS -Dmachine=default"
 
@@ -42,14 +44,17 @@ function build_dpdk()
 DPDK_OPTS="$DPDK_OPTS -Denable_drivers=$enable_drivers"
 
 # Install DPDK using prefix.
-DPDK_OPTS="$DPDK_OPTS --prefix=$(pwd)/build"
+DPDK_OPTS="$DPDK_OPTS --prefix=$DPDK_INSTALL_DIR"
 
 meson $DPDK_OPTS build
 ninja -C build
 ninja -C build install
-
-echo "Installed DPDK in $(pwd)"
 popd
+
+# Remove examples sources.
+rm -rf $DPDK_INSTALL_DIR/share/dpdk/examples
+
+echo "Installed DPDK in $DPDK_INSTALL_DIR"
 echo "${DPDK_VER}" > ${VERSION_FILE}
 }
 
diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index 170f4730f3..ea1e6599f4 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -12,8 +12,9 @@ EXTRA_OPTS="--enable-Werror"
 
 function install_dpdk()
 {
-local VERSION_FILE="dpdk-dir/cached-version"
-local DPDK_LIB=$(pwd)/dpdk-dir/build/lib/x86_64-linux-gnu
+local DPDK_INSTALL_DIR="$(pwd)/dpdk-dir"
+local VERSION_FILE="${DPDK_INSTALL_DIR}/cached-version"
+local DPDK_LIB=${DPDK_INSTALL_DIR}/lib/x86_64-linux-gnu
 
 if [ "$DPDK_SHARED" ]; then
 EXTRA_OPTS="$EXTRA_OPTS --with-dpdk=shared"
@@ -29,13 +30,13 @@ function install_dpdk()
 export PATH=$(pwd)/dpdk-dir/build/bin:$PATH
 
 if [ ! -f "${VERSION_FILE}" ]; then
-echo "Could not find DPDK in $(pwd)/dpdk-dir"
+echo "Could not find DPDK in $DPDK_INSTALL_DIR"
 return 1
 fi
 
 # Update the library paths.
 sudo ldconfig
-echo "Found cached DPDK $(cat ${VERSION_FILE}) build in $(pwd)/dpdk-dir"
+echo "Found cached DPDK $(cat ${VERSION_FILE}) build in $DPDK_INSTALL_DIR"
 }
 
 function configure_ovs()
-- 
2.41.0

___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [ovs-dev] [PATCH v9 1/3] netdev-linux: Use ethtool to detect offload support.

2023-11-22 Thread Simon Horman 
On Tue, Nov 21, 2023 at 02:26:50PM -0500, Mike Pattrick wrote:
> Currently when userspace-tso is enabled, netdev-linux interfaces will
> indicate support for all offload flags regardless of interface
> configuration. This patch checks for which offload features are enabled
> during netdev construction.
> 
> Signed-off-by: Mike Pattrick 
> 
> --
> 
> v6:
>  - Removed legacy comment
>  - Reverse xmas
> v7:
>  - Replaced comment in netdev_linux_construct
>  - Corrected indentation, allocation functions
>  - Changed loop logic

Thanks Mike,

as far as I can tell this patch addresses the most recent review, of v6.

Acked-by: Simon Horman 
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [ovs-dev] [PATCH v9 2/3] userspace: Respect tso/gso segment size.

2023-11-22 Thread Simon Horman 
On Tue, Nov 21, 2023 at 02:26:51PM -0500, Mike Pattrick wrote:
> From: Flavio Leitner 
> 
> Currently OVS will calculate the segment size based on the
> MTU of the egress port. That usually happens to be correct
> when the ports share the same MTU, but that is not always true.
> 
> Therefore, if the segment size is provided, then use that and
> make sure the over sized packets are dropped.
> 
> Signed-off-by: Flavio Leitner 
> Co-authored-by: Mike Pattrick 
> Signed-off-by: Mike Pattrick 
> 
> ---
> v7:
>  - Removed redundant content from comment
>  - Expanded logging message when packets are dropped

Thanks Mike,

as far as I can tell this patch addresses the most recent review, of v6.

Acked-by: Simon Horman 
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [ovs-dev] [PATCH v9 3/3] userspace: Add Generic Segmentation Offloading.

2023-11-22 Thread Simon Horman 
On Tue, Nov 21, 2023 at 02:26:52PM -0500, Mike Pattrick wrote:
> From: Flavio Leitner 
> 
> This provides a software implementation in the case
> the egress netdev doesn't support segmentation in hardware.
> 
> The challenge here is to guarantee packet ordering in the
> original batch that may be full of TSO packets. Each TSO
> packet can go up to ~64kB, so with segment size of 1440
> that means about 44 packets for each TSO. Each batch has
> 32 packets, so the total batch amounts to 1408 normal
> packets.
> 
> The segmentation estimates the total number of packets
> and then the total number of batches. Then allocate
> enough memory and finally do the work.
> 
> Finally each batch is sent in order to the netdev.
> 
> Signed-off-by: Flavio Leitner 
> Co-authored-by: Mike Pattrick 
> Signed-off-by: Mike Pattrick 
> 
> ---
> 
> v4:
>  - Various formatting changes
>  - Fixed memory leak in soft-gso code if packet is flagged
>for GSO but incorrectly lacks segment size.
> 
> v5:
>  - Corrected indentation and comments
>  - Fixed pointer arithmatic issue
>  - Added a test to exercise software gso code
> 
> v7:
>  - Removed redundant parenthesis
>  - Improved system test
> 
> v8:
>  - Corrected system test
> 
> v9:
>  - Output from ethtool command across CI environments was not reliable.
>  - The test will now ignore ethtool output

Hi Mike,

I notice several updates in this patchset targeted ad the system test.
Have those changes settled down now?

Other than the above, I do not have any questions regarding this patch(set).
And I do see that the most recent review, of v6, has been addressed as of v7.

Acked-by: Simon Horman 

...
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev