geido commented on code in PR #45755:
URL: https://github.com/apache/superset/pull/45755#discussion_r4400566498
##########
.github/workflows/docker.yml:
##########
@@ -503,6 +693,68 @@ jobs:
sleep 30
done
+ # supersetbot tagged the linux/amd64 push above with the unsuffixed tags
+ # (<sha>, <short sha>, master, each with the preset chunk) and the
+ # linux/arm64 push in docker-build-arm64 with the same tags plus an
+ # `-arm` suffix. Fold each pair into one manifest list under the
+ # unsuffixed tag, which is what the single multi-platform build used
+ # to publish. The run-scoped GHA-* tags stay per platform (IMAGE_TAG
+ # carries a -arm64 suffix in the arm64 job) since nothing pulls them
+ # cross-platform.
+ - name: Publish the multi-platform manifest lists
+ if: >-
+ steps.publish_guard.outputs.publish == 'true' &&
+ github.event_name == 'push' && github.ref == 'refs/heads/master'
+ shell: bash
+ env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ BUILD_PRESET: ${{ matrix.build_preset }}
+ run: |
+ set -euo pipefail
+ eval "$(bash scripts/docker-build-plan.sh "$GITHUB_EVENT_NAME"
"$GITHUB_REF" "${GITHUB_REF_NAME:-}" "$BUILD_PLATFORM")"
+ mapfile -t tags < <(
+ DOCKERHUB_TOKEN= supersetbot docker \
+ --preset "$BUILD_PRESET" \
+ --context "$BUILD_CONTEXT" \
+ --context-ref "$BUILD_CONTEXT_REF" \
+ --platform linux/amd64 \
+ --dry-run |
+ grep -oE -- '-t apache/superset:[^[:space:]\\]+' | cut -d' ' -f2
| sort -u
+ )
+ # Merge from the per-commit tags (the full-sha tag of this preset
+ # and its `-arm` twin) rather than from the mutable `master*` and
+ # `master*-arm` tags, so a newer commit's push landing mid-run
+ # cannot pair this commit's amd64 image with another commit's arm64.
+ sha_tag=""
+ for tag in "${tags[@]}"; do
+ case "$tag" in
"apache/superset:${GITHUB_SHA}"|"apache/superset:${GITHUB_SHA}-"*)
sha_tag="$tag"; break ;; esac
+ done
+ if [ -z "$sha_tag" ]; then
+ echo "::error::no per-commit tag found among: ${tags[*]}"
+ exit 1
+ fi
+ merged=0
+ for tag in "${tags[@]}"; do
+ case "$tag" in *:GHA-*) continue ;; esac
+ for attempt in 1 2 3; do
+ if docker buildx imagetools create -t "$tag" "$sha_tag"
"${sha_tag}-arm"; then
+ merged=$((merged + 1))
+ break
+ fi
+ if [ "$attempt" -eq 3 ]; then
+ echo "::error::could not merge ${tag}-arm into $tag"
+ exit 1
+ fi
+ echo "::warning::manifest merge attempt ${attempt} for $tag
failed; retrying in 15s..."
+ sleep 15
+ done
+ docker buildx imagetools inspect "$tag"
Review Comment:
Not sure, but this `imagetools inspect` sits outside the retry, so with `set
-e` one transient Docker Hub error ends the step after that tag merged. Since
`master*` sorts last, it would be the one left amd64-only. Would `|| true`, or
moving it inside the retry loop, make sense here?
##########
.github/workflows/docker.yml:
##########
@@ -330,28 +377,168 @@ jobs:
assert_cache_push_for_master lean "3.11-slim-trixie-lean"
assert_cache_push_for_master py311 "3.11-slim-bookworm"
assert_cache_push_for_master py312 "3.12-slim-bookworm"
+ assert_platform_cache_refs superset linux/amd64 "3.11-slim-trixie"
+ assert_platform_cache_refs superset linux/arm64
"3.11-slim-trixie-arm64"
+ assert_platform_cache_refs dev linux/arm64
"3.11-slim-trixie-dev-arm64"
+ assert_platform_cache_refs py312 linux/arm64
"3.12-slim-bookworm-arm64"
+ assert_platform_master_tag superset linux/amd64 master
+ assert_platform_master_tag superset linux/arm64 master-arm
+ assert_platform_master_tag dev linux/amd64 master-dev
+ assert_platform_master_tag dev linux/arm64 master-dev-arm
+ assert_platform_master_tag py311 linux/arm64 master-py311-arm
assert_release_tag superset "6.1.0"
assert_release_tag dev "6.1.0-dev"
assert_release_tag lean "6.1.0-lean"
assert_release_tag py311 "6.1.0-py311"
assert_release_tag py312 "6.1.0-py312"
assert_release_ref_swap
- echo "Docker build command verified for Python versions, cache refs,
cache export, publish guard, master tags, release tags, and the release ref
swap"
+ echo "Docker build command verified for Python versions, cache refs,
cache export, publish guard, master tags, per-platform plans/cache refs/tags,
release tags, and the release ref swap"
+
+ # Master publishes are multi-platform. Building linux/arm64 under QEMU on
+ # an amd64 runner took most of a 30+ minute build, so each platform is now
+ # built natively in its own job: this one pushes the linux/arm64 image
+ # under supersetbot's `-arm` suffixed tags (see getDockerTags in
+ # apache-superset/supersetbot), and docker-build below, which runs after
+ # it, pushes linux/amd64 under the unsuffixed tags and then folds both
+ # into one multi-platform manifest list per tag. Pull requests and version
+ # branches only validate a linux/amd64 build, so this job is skipped there.
+ docker-build-arm64:
+ name: docker-build (arm64)
+ needs: [setup_matrix, changes, check-freshness]
+ if: >-
+ needs.check-freshness.outputs.is-current == 'true' &&
+ github.event_name == 'push' &&
+ github.ref == 'refs/heads/master' &&
+ needs.setup_matrix.outputs.has-secrets &&
+ (needs.changes.outputs.python == 'true' ||
+ needs.changes.outputs.frontend == 'true' ||
+ needs.changes.outputs.docker == 'true')
+ runs-on: ubuntu-26.04-arm
+ timeout-minutes: 60
+ permissions:
+ contents: read
+ strategy:
+ matrix:
+ build_preset: ${{fromJson(needs.setup_matrix.outputs.matrix_config)}}
+ fail-fast: false
+ # Same per-preset publish queue as docker-build, kept separate so the two
+ # platforms of one commit build concurrently.
+ concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}-${{ matrix.build_preset
}}-arm64
+ cancel-in-progress: false
+ env:
+ DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
+ DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
+ BUILD_PLATFORM: linux/arm64
+ IMAGE_TAG: apache/superset:GHA-${{ matrix.build_preset }}-${{
github.run_id }}-arm64
+
+ steps:
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #
v7.0.1
+ with:
+ persist-credentials: false
+
+ - name: Free up disk space
+ shell: bash
+ run: |
+ echo "Disk before cleanup:"; df -h /
+ sudo rm -rf \
+ /usr/share/dotnet \
+ /usr/local/lib/android \
+ /opt/ghc \
+ /usr/local/.ghcup \
+ /opt/hostedtoolcache/CodeQL \
+ /usr/local/share/boost || true
+ echo "Disk after cleanup:"; df -h /
+
+ - name: Setup Docker Environment
+ uses: $/.github/actions/setup-docker
+ with:
+ dockerhub-user: ${{ secrets.DOCKERHUB_USER }}
+ dockerhub-token: ${{ secrets.DOCKERHUB_TOKEN }}
+ build: "true"
+
+ - name: Setup supersetbot
+ uses: $/.github/actions/setup-supersetbot/
+
+ # Same stale-commit check as docker-build; see the comment there.
+ - name: Guard against publishing a superseded master commit
+ id: publish_guard
+ shell: bash
+ run: |
+ set -euo pipefail
+ TIP_SHA=""
+ for attempt in 1 2 3; do
+ if TIP_SHA="$(git ls-remote
"$GITHUB_SERVER_URL/$GITHUB_REPOSITORY" refs/heads/master | cut -f1)" &&
+ [ -n "$TIP_SHA" ]; then
+ break
+ fi
+ if [ "$attempt" -eq 3 ]; then
+ echo "::error::could not read the master tip to validate
$GITHUB_SHA"
+ exit 1
+ fi
+ echo "::warning::ls-remote attempt ${attempt} failed; retrying in
10s..."
+ sleep 10
+ done
+ eval "$(bash scripts/docker-publish-guard.sh "$GITHUB_EVENT_NAME"
"$GITHUB_REF" "$GITHUB_SHA" "$TIP_SHA")"
+ if [ "$PUBLISH" != "true" ]; then
+ echo "::notice::master advanced to ${TIP_SHA} while ${GITHUB_SHA}
was queued; skipping this build"
+ fi
+ echo "publish=$PUBLISH" >> "$GITHUB_OUTPUT"
+
+ - name: Build Docker Image (linux/arm64)
+ if: steps.publish_guard.outputs.publish == 'true'
+ shell: bash
+ env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ BUILD_PRESET: ${{ matrix.build_preset }}
+ run: |
+ set -euo pipefail
+ eval "$(bash scripts/docker-build-plan.sh "$GITHUB_EVENT_NAME"
"$GITHUB_REF" "${GITHUB_REF_NAME:-}" "$BUILD_PLATFORM")"
+ export PUBLISH_DOCKER_CACHE
+ # The platform argument selects the arm64 registry cache ref; see
+ # scripts/docker-build-extra-flags.sh.
+ EXTRA_FLAGS="$(scripts/docker-build-extra-flags.sh "$BUILD_PRESET"
"$IMAGE_TAG" ci "$BUILD_PLATFORM")"
+
+ for attempt in 1 2 3; do
+ if DOCKERHUB_TOKEN= supersetbot docker \
+ $PUSH_OR_LOAD \
+ --preset "$BUILD_PRESET" \
+ --context "$BUILD_CONTEXT" \
+ --context-ref "$BUILD_CONTEXT_REF" \
+ --extra-flags "$EXTRA_FLAGS" \
+ $PLATFORM_ARG; then
+ break
+ fi
+ if [ "$attempt" -eq 3 ]; then
+ echo "::error::supersetbot docker build failed after 3 attempts"
+ exit 1
+ fi
+ echo "::warning::Build attempt ${attempt} failed; retrying in
30s..."
+ sleep 30
+ done
docker-build:
name: docker-build
- needs: [setup_matrix, changes, check-freshness]
+ needs: [setup_matrix, changes, check-freshness, docker-build-arm64]
Review Comment:
IIUC this `needs` makes every amd64 leg, and its publish guard, wait for the
slowest arm64 leg, so the platforms run in series, not concurrently as the
arm64 concurrency comment says. A docs-only commit landing in that wait would
then leave all `master*` tags on the previous commit until the next code push.
Is that intended, or could only the merge wait on arm64?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
