Public bug reported: [ Impact ]
The currently included aardvark-dns version 1.4.0 contains a bug (https://github.com/containers/aardvark-dns/issues/389) that makes podman pretty much unusable as executor for GitLab runner. When running multiple GitLab CI jobs concurrently, a significant percentage of DNS requests experiences package loss, which leads to random job failures. In context of CI, where you want reproducible results, this is unacceptable. The bug was fixed in 04/2024 (https://github.com/containers/aardvark- dns/pull/449) and released in version 1.11.0 (and there was a bugfix release 1.10.1 for RHEL - https://github.com/containers/aardvark- dns/pull/513). - https://github.com/containers/aardvark-dns/releases/tag/v1.11.0 - https://github.com/containers/aardvark-dns/releases/tag/v1.10.1-rhel There are workarounds like switching to CNI or even switching from podman to docker as container engine, but because netavark and aardvark- dns are configured as the default when installing podman 4.9.3 on noble, this should get fixed imo. [ Test Plan ] detailed instructions how to reproduce the bug - install gitlab-runner (https://docs.gitlab.com/runner/install/linux-repository/#install-gitlab-runner) - install podman and setup as rootless (adding subuid's and subgid's for gitlab-runner user - https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md) - use podman as gitlab-runner executor (https://docs.gitlab.com/runner/executors/docker/#use-podman-to-run-docker-commands) - configure podman socket for gitlab-runner user - enable socket for gitlab-runner user: `sudo systemctl --user -M gitlab-runner@ enable --now podman.socket` - copy the socket listen string from `sudo systemctl --user -M gitlab-runner@ status podman.socket` - enable lingering `sudo loginctl enable-linger gitlab-runner` - register runner via GitLab web interface - edit gitlab-runner config.toml - paste socket listen string from earlier - enable FF_NETWORK_PER_BUILD (environment = ["FF_NETWORK_PER_BUILD=1"]) - configure gitlab-runner to allow running 20 or more jobs concurrently (concurrent = 20) - trigger a gitlab pipeline with multiple jobs running in parallel performing DNS requests - check for job failures because of DNS [ Where problems could occur ] I am not sure if a new aardvark-dns version also requires a newer netavark version. [ Other Info ] There is also a gitlab forum post regarding this: https://forum.gitlab.com/t/podman-runner-cannot-resolve-the-repository-hostname/104343 The user jwillikers reported that a manually installed newer version of aardvark-dns worked without issues with the podman gitlab-runner setup. ** Affects: aardvark-dns (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2115594 Title: [SRU] Update aardvark-dns to 1.10.1 or 1.11.0 or later in noble for bugfix To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/aardvark-dns/+bug/2115594/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
