Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20469 )
Change subject: IMPALA-12436: Create multi-platform builder images ...................................................................... IMPALA-12436: Create multi-platform builder images Updates the toolchain build image creation process to support publishing multi-platform (aarch64 and amd64) images. Uses docker buildx to produce multi-platform images. This requires QEMU on the builder; these builds are quick, so the overhead of emulating arm is minimal. Currently assumes images are built on amd64 as not all builds specify `--platform`. Moves the publishing step to happen within buildx, because buildx does not support loading multi-platform images in the local Docker registry. This does mean that a build failure could result in publishing only some of the images. These are not updated frequently and are usually preceded by a test run, so this risk seems minimal. The jenkins/build.sh script avoids this issue by first running a build without publishing, then when it succeeds running again with publishing (which will largely pull image layers from the build cache so should be very fast). Python 2.7 ran into build issues installing awscli on ARM, so OS targeting ARM were updated to ensure they include Python 3.8+ and install a newer version of awscli in Python 3. New options were added to buildall.py: - registry: publish images to the specified registry - multi: enables multi-platform builds on RedHat 8 and Ubuntu 20/22 Multi-platform builds require QEMU and a non-default buildx builder. Jenkins job is updated to install QEMU, create a builder, and build multi-platform images. Multi-platform images can't be loaded into a local Docker context (where they would be listed via docker images); we continue to load them there by default with --load, but disable it when doing multi-platform builds. The only way to get images out of the builder for multi-platform builds is to specify a registry. Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Reviewed-on: http://gerrit.cloudera.org:8080/20469 Reviewed-by: Joe McDonnell <[email protected]> Tested-by: Michael Smith <[email protected]> --- M docker/README.md M docker/all/postinstall.sh M docker/buildall.py M docker/redhat8.df M docker/ubuntu2004.df M docker/ubuntu2204.df M jenkins/build.sh 7 files changed, 87 insertions(+), 32 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Michael Smith: Verified -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 8 Gerrit-Owner: Michael Smith <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Laszlo Gaal <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]>
