This is an automated email from the ASF dual-hosted git repository. vatamane pushed a commit to branch cleanup-unused-commands in repository https://gitbox.apache.org/repos/asf/couchdb-ci.git
commit b253111b57139d66f8369b6a2a94ed61efb48eb3 Author: Nick Vatamaniuc <[email protected]> AuthorDate: Fri Dec 30 00:17:24 2022 -0500 Update default Erlang and cleanup unused commands Some of those haven't worked in more than 2 years and we've only used buildx to build both the x86 and the multi-arch images. Ideally we wouldn't want to use proprietary Docker bits but that's the only one that works well with multi-arch builds. Perhaps Podman or other newer things can replace it soon enough. --- README.md | 54 ++++++----------- bin/install-dependencies.sh | 14 +---- build.sh | 144 ++------------------------------------------ 3 files changed, 25 insertions(+), 187 deletions(-) diff --git a/README.md b/README.md index d30e43f..8e0a01d 100644 --- a/README.md +++ b/README.md @@ -49,20 +49,14 @@ On the other hand, some OSes won't run older Erlangs because of library changes, Just specify on the command line any of the `ERLANGVERSION`, `NODEVERSION`, or `ELIXIRVERSION` environment variables: ``` -NODEVERSION=14 ELIXIRVERSION=v1.13.4 ERLANGVERSION=24.3.4.2 ./build.sh platform debian-jessie +NODEVERSION=14 ELIXIRVERSION=v1.13.4 ERLANGVERSION=24.3.4.7 ./build.sh platform debian-jessie ``` -The tool also recognizes a special `ERLANGVERSION=all` value for the `debian-buster` -platform. This builds the lowest, default, and highest versions of Erlang using -the [kerl](https://github.com/kerl/kerl) build system, and installs them to -`/usr/local/kerl` for activation before builds. This version is intended for use -in standard CI runs, such as for pull requests. - ## Building images for other architectures ### Multi-arch images with Docker Buildx -We can use Docker's +Use Docker's [Buildx](https://docs.docker.com/buildx/working-with-buildx/) plugin to generate multi-architecture container images with a single command invocation. Docker Desktop ships with buildx support, but you'll need to create a new builder to @@ -83,36 +77,38 @@ example: The `$BUILDX_PLATFORMS` environment variable can be used to override the default set of target platforms that will be supplied to the buildx builder. -### Cross-building with $CONTAINERARCH - -Alternatively, we can build individual images for each architecture. This only works from an `x86_64` build host. +# Useful things you can do -First, configure your machine with the correct dependencies to build multi-arch binaries: +## Update images used for package releases with new Erlang versions ``` -docker run --privileged --rm tonistiigi/binfmt --install all +ERLANGVERSION=24.3.4.7 ./build.sh buildx-platform-release ``` -This is a one-time setup step. This docker container run will install the correct qemu static binaries necessary for running foreign architecture binaries on your host machine. It includes special magic to ensure `sudo` works correctly inside a container, too. +This will build all the Debian and RHEL-clone OS images on x86-64 with that version of Erlang -Then, override the `CONTAINERARCH` environment variable when starting `build.sh`: +## Update images used for CI with new Erlang versions ``` -CONTAINERARCH=arm64v8 ./build.sh platform debian-bullseye +ERLANGVERSION=24.3.4.7 ./build.sh buildx-platform debian-bullseye ``` -## Publishing a container +This will update Debian Bullseye OS image for all architectures (x86, +ppc64le, arm64) with that version of Erlang. Do this for the images +which are used with non-x86 architectures. -If you built a single-architecture container image and did not supply `--push` -as a build arg to upload it automatically you can upload the image using +If the same image is used for release package building and CI, run +this command after `buildx-platform-release` to ensure that the +debian-bullseye will be a rebuilt as a multi-arch image. Otherwise, +buildx-platform-release creates as x86 only. + +## Update Debian Bullseye image with 25.2 ``` -./build.sh platform-upload <distro>-<version> +BUILDX_PLATFORMS=linux/amd64 ERLANGVERSION=25.2 ./build.sh buildx-platform debian-bullseye ``` ---- - -# Useful things you can do +In this case, since we're not using 25.2 for multi-arch testing, opt to build it only for x86. ## Full `build.sh` options @@ -123,18 +119,8 @@ Recognized commands: clean <plat> Removes all images for <plat>. clean-all Removes all images for all platforms. - *buildx-base <plat> Builds a multi-architecture base image. *buildx-platform <plat> Builds a multi-architecture image with Erlang & JS support. - - base <plat> Builds the image for <plat> without Erlang or JS support. - base-all Builds all images without Erlang or JS support. - *base-upload <plat> Uploads the apache/couchdbci-{os} base images to Docker Hub. - *base-upload-all Uploads all the apache/couchdbci base images to Docker Hub. - - platform <plat> Builds the image for <plat> with Erlang & JS support. - platform-all Builds all images with Erlang and JS support. - *platform-upload <plat> Uploads the apache/couchdbci-{os} images to Docker Hub. - *platform-upload-all Uploads all the apache/couchdbci images to Docker Hub. + *buildx-platform-release <plat> Builds x86-64 images with default Erlang for all supported release OSes couch <plat> Builds and tests CouchDB for <plat>. couch-all Builds and tests CouchDB on all platforms. diff --git a/bin/install-dependencies.sh b/bin/install-dependencies.sh index 1bbf010..ecc69fc 100755 --- a/bin/install-dependencies.sh +++ b/bin/install-dependencies.sh @@ -30,7 +30,7 @@ set -e # Defaults updated 2019-10-14 NODEVERSION=${NODEVERSION:-14} -ERLANGVERSION=${ERLANGVERSION:-23.3.4.15} +ERLANGVERSION=${ERLANGVERSION:-24.3.4.7} ELIXIRVERSION=${ELIXIRVERSION:-v1.13.4} @@ -97,18 +97,6 @@ case "${OSTYPE}" in run_scripts ${EXTRA_SCRIPTS_DIR} 'yum-' elif [[ ${ID} =~ ${debians} ]]; then - # Catching this early, so as to avoid user frustration - if [[ ${ERLANGVERSION%%.*} -le 19 ]] && [[ ${VERSION_CODENAME} =~ ${latest} ]]; then - if [[ $ARCH =~ $arms ]] && [[ ! ${SKIPERLANG} ]]; then - echo "" - echo "Recent versions of Linux (Stretch, Bionic, etc) provide a version of libssl" - echo "which is too new to complile earlier (<=19) versions of Erlang. Please" - echo "either choose an earlier distro release or a more rencent version of Erlang." - echo "" - exit 1 - fi - fi - NODEVERSION=${NODEVERSION} ERLANGVERSION=${ERLANGVERSION} \ ${SCRIPTPATH}/apt-dependencies.sh ${JSINSTALL} if [[ ! ${SKIPERLANG} ]]; then diff --git a/build.sh b/build.sh index 4c9aaa5..87aafcc 100755 --- a/build.sh +++ b/build.sh @@ -84,49 +84,6 @@ split-os-ver() { version=${tokens[1]} } -build-base-platform() { - check-envs - split-os-ver $1 - # invoke as build-base <plat> - # base images never get JavaScript, nor Erlang - docker build -f dockerfiles/${os}-${version} \ - --build-arg js=nojs \ - --build-arg erlang=noerlang \ - $buildargs \ - --tag apache/couchdbci-${os}:${CONTAINERARCH}${version}-base \ - ${SCRIPTPATH} -} - -buildx-base-platform() { - check-envs - split-os-ver $1 - # invoke as build-base <plat> - # base images never get JavaScript, nor Erlang - if [ "$os" == "rockylinux" ]; then - repo="centos" - else - repo="$os" - fi - docker buildx build -f dockerfiles/${os}-${version} \ - --build-arg js=nojs \ - --build-arg erlang=noerlang \ - $buildargs \ - --platform ${BUILDX_PLATFORMS} \ - --tag apache/couchdbci-${repo}:${version}-base \ - --push \ - ${SCRIPTPATH} -} - -upload-base() { - if [[ ! ${DOCKER_ID_USER} ]]; then - echo "Please set your Docker credentials before using this command:" - echo " export DOCKER_ID_USER=<username>" - echo " docker login" - exit 1 - fi - docker push couchdbdev/$1-base -} - find-erlang-version() { if [ -z "${ERLANGVERSION}" ] then @@ -139,19 +96,6 @@ pull-os-image() { docker pull $image_name } -build-platform() { - check-envs - find-erlang-version $1 - pull-os-image $1 - split-os-ver $1 - docker build -f dockerfiles/${os}-${version} \ - $buildargs \ - --no-cache \ - --tag apache/couchdbci-${os}:${CONTAINERARCH}${version}-erlang-${ERLANGVERSION} \ - ${SCRIPTPATH} - unset ERLANGVERSION -} - buildx-platform() { check-envs find-erlang-version $1 @@ -169,7 +113,6 @@ buildx-platform() { --tag apache/couchdbci-${repo}:${version}-erlang-${ERLANGVERSION} \ --push \ ${SCRIPTPATH} - unset ERLANGVERSION } clean() { @@ -178,19 +121,6 @@ clean() { docker rmi apache/couchdbci-${os} -f || true } -upload-platform() { - if [[ ! ${DOCKER_ID_USER} ]]; then - echo "Please set your Docker credentials before using this command:" - echo " export DOCKER_ID_USER=<username>" - echo " docker login" - exit 1 - fi - find-erlang-version $1 - check-envs - split-os-ver $1 - docker push apache/couchdbci-${os}:${CONTAINERARCH}${version}-erlang-${ERLANGVERSION} -} - build-test-couch() { find-erlang-version $1 docker run \ @@ -215,74 +145,18 @@ case "$1" in clean $plat done ;; - buildx-base) - # Build and upload multi-arch base image using Docker Buildx - shift - buildx-base-platform $1 - ;; - base) - # Build base image for requested target platform - shift - build-base-platform $1 - ;; - base-all) - # build all base images - shift - for plat in $DEBIANS $UBUNTUS $CENTOSES; do - build-base-platform $plat $* - done - ;; - base-upload) - shift - upload-base $plat $* - ;; - base-upload-all) - shift - for plat in $DEBIANS $UBUNTUS $CENTOSES; do - upload-base $plat $* - done - ;; buildx-platform) # Build and upload multi-arch platform with JS and Erlang support shift buildx-platform $1 ;; - platform) - # build platform with JS and Erlang support - shift - build-platform $1 - ;; - platform-foreign) - # makes only foreign arch platforms - shift - for arch in $XPLAT_ARCHES; do - CONTAINERARCH=$arch build-platform $XPLAT_BASE - done - ;; - platform-all) - # build all platforms with JS and Erlang support - shift - for plat in $DEBIANS $UBUNTUS $CENTOSES; do - build-platform $plat $* - done - for arch in $XPLAT_ARCHES; do - CONTAINERARCH=$arch build-platform $XPLAT_BASE - done - ERLANGVERSION=all build-platform $ERLANGALL_BASE - ;; - platform-upload) - shift - upload-platform $plat $* - ;; - platform-upload-all) + buildx-platform-release) + # Build and upload multi-arch platform with JS and Erlang support + # For all platforms shift for plat in $DEBIANS $UBUNTUS $CENTOSES; do - upload-platform $plat $* - done - for arch in $XPLAT_ARCHES; do - CONTAINERARCH=$arch upload-platform $XPLAT_BASE $* + BUILDX_PLATFORMS=linux/amd64 buildx-platform $plat done - ERLANGVERSION=all upload-platform $ERLANGALL_BASE ;; couch) # build and test CouchDB on <plat> @@ -312,16 +186,6 @@ Recognized commands: *buildx-base <plat> Builds a multi-architecture base image. *buildx-platform <plat> Builds a multi-architecture image with Erlang & JS support. - base <plat> Builds the image for <plat> without Erlang or JS support. - base-all Builds all images without Erlang or JS support. - *base-upload <plat> Uploads the apache/couchdbci-{os} base images to Docker Hub. - *base-upload-all Uploads all the apache/couchdbci base images to Docker Hub. - - platform <plat> Builds the image for <plat> with Erlang & JS support. - platform-all Builds all images with Erlang and JS support. - *platform-upload <plat> Uploads the apache/couchdbci-{os} images to Docker Hub. - *platform-upload-all Uploads all the apache/couchdbci images to Docker Hub. - couch <plat> Builds and tests CouchDB for <plat>. couch-all Builds and tests CouchDB on all platforms.
