This is an automated email from the ASF dual-hosted git repository.
dbtsai pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new c94ecaf [SPARK-60917][CONNECT][FOLLOWUP] Check ARM release wheel
builds in CI
c94ecaf is described below
commit c94ecafc15e11e077cb7913d78db58351e2604eb
Author: DB Tsai <[email protected]>
AuthorDate: Thu Nov 5 13:12:30 2026 -0800
[SPARK-60917][CONNECT][FOLLOWUP] Check ARM release wheel builds in CI
### What changes were proposed in this pull request?
Add `wheel-build-checks.yml` for x86 Linux to aarch64 Linux wheel builds
with Maturin/Zig and macOS universal2 wheel builds. Both checks use the same
reusable `build-wheel.yml` recipe as all four release wheel targets.
Run the checks for Cargo manifests/lockfile, build scripts, protocol
definitions under `crates/spark-connect-proto/proto/`, packaged Python files
under `python/pyspark/`, packaging/toolchain configuration, and the wheel
workflows. Ordinary Rust source changes do not start these focused jobs. The
scheduled release dry run builds only `master`, so `branch-*` maintenance
branches have no scheduled backstop for Rust source changes. Validate those
branches by manually dispatching `release. [...]
The shared recipe uses no dependency or compiled-artifact cache. Its
timeout is configurable: checks use 45 minutes and releases explicitly retain
their previous 360-minute limit. Check artifacts expire after one day; release
artifacts retain the repository default. In-progress checks are canceled only
for PR events.
Resolve the canonical source repository from `env.UPSTREAM_REPO` through
the `resolve` job's `repo` output, alongside the immutable source SHA. The
reusable workflow is loaded from the caller's revision, so it can build an
older maintenance branch without requiring that source branch to contain the
helper workflow.
Checkout now sets `persist-credentials: false` for wheel builds; the old
release wheel checkout used the action's default. The wheel build commands,
four-target matrix, artifact names, release modes and publishing jobs are
retained. Document the build-only checks, master-only schedule, and manual
maintenance-branch validation in `dev/design/ci.md`. Check names identify their
target; the shared job uses a short `Build` label so release checks do not
repeat `Build wheel (target)`. The Z [...]
Follow-up to #155/#163 and
[SPARK-60917](https://issues.apache.org/jira/browse/SPARK-60917). Native ARM
core tests remain complementary runtime coverage.
### Why are the changes needed?
Native GCC compilation does not exercise the release's Zig cross-build, and
a native macOS build does not compile both universal2 slices. Sharing the
recipe keeps those checks aligned with releases, while narrow triggers limit
macOS runner use.
### Does this PR introduce _any_ user-facing change?
No. This changes CI wheel-build coverage and shares the release build
recipe.
### How was this patch tested?
- `actionlint` 1.7.12 and whitespace checks pass.
- Configuration checks verify timeout wiring, canonical repository and SHA
outputs, cache removal, unchanged build commands/matrix and publishing modes,
and the narrowed path filter. The review cleanup passes `actionlint` and
structural checks that limit behavior changes to job labels and the added proto
path filter.
- The Rust source tree is unchanged from the previously validated revision:
107 transport and 414 Rust API release tests passed, all integration targets
compiled with and without `wasm-udf`, and formatting passed. Clippy reports
pre-existing errors in unchanged source files.
- On revision `6d8754a`, the [macOS universal2 wheel
build](https://github.com/apache/spark-connect-rust/actions/runs/42103740961/job/127513368414)
and [native aarch64 core
tests](https://github.com/apache/spark-connect-rust/actions/runs/42103740912/job/127513015130)
passed. The Linux aarch64 Zig check remained queued at the latest snapshot. No
manual ASF reruns were requested.
- A full fork release dry run remains pending. The fork's Actions API
reports `enabled: true` but zero registered workflows, despite nine workflow
YAML files (including `release.yml` with manual dispatch) on its default
`master` branch and admin access for the authenticated account. Direct workflow
lookup, targeted enable, and dry-run dispatch all return HTTP 404. One off/on
reset did not restore registration; the original Actions policy was verified
restored exactly. The signed-in Ch [...]
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex
This pull request and its description were written by Isaac.
Closes #165 from dbtsai/codex/release-wheel-ci.
Authored-by: DB Tsai <[email protected]>
Signed-off-by: DB Tsai <[email protected]>
Generated-by: OpenAI Codex
Co-authored-by: Isaac <[email protected]>
---
.github/workflows/build-wheel.yml | 118 +++++++++++++++++++++++++++++++
.github/workflows/release.yml | 59 ++++------------
.github/workflows/wheel-build-checks.yml | 69 ++++++++++++++++++
dev/design/ci.md | 29 +++++++-
4 files changed, 226 insertions(+), 49 deletions(-)
diff --git a/.github/workflows/build-wheel.yml
b/.github/workflows/build-wheel.yml
new file mode 100644
index 0000000..4732427
--- /dev/null
+++ b/.github/workflows/build-wheel.yml
@@ -0,0 +1,118 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# Shared wheel recipe for releases and focused PR build checks. The workflow
+# comes from the caller's revision; source_repository/source_ref select only
the
+# source checkout, so a release can still build older maintenance branches.
+name: Build wheel
+
+on:
+ workflow_call:
+ inputs:
+ runner:
+ description: GitHub-hosted runner label
+ required: true
+ type: string
+ target:
+ description: Rust or maturin wheel target
+ required: true
+ type: string
+ zig:
+ description: Use Zig for Linux cross-linking
+ default: false
+ type: boolean
+ timeout_minutes:
+ description: Maximum build duration in minutes
+ default: 360
+ type: number
+ artifact_retention_days:
+ description: Days to retain wheel artifacts (0 uses the repository
default)
+ default: 0
+ type: number
+ source_repository:
+ description: Repository containing the wheel source
+ required: true
+ type: string
+ source_ref:
+ description: Pinned commit containing the wheel source
+ required: true
+ type: string
+
+permissions:
+ contents: read
+
+jobs:
+ build:
+ name: Build
+ runs-on: ${{ inputs.runner }}
+ timeout-minutes: ${{ inputs.timeout_minutes }}
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ repository: ${{ inputs.source_repository }}
+ ref: ${{ inputs.source_ref }}
+ persist-credentials: false
+ - uses: actions/setup-python@v5
+ with:
+ python-version: "3.11"
+ - name: Install Rust toolchain
+ shell: bash
+ run: |
+ rustup toolchain install stable --profile minimal
+ rustup default stable
+ - name: Add Rust target(s)
+ shell: bash
+ env:
+ WHEEL_TARGET: ${{ inputs.target }}
+ run: |
+ case "$WHEEL_TARGET" in
+ universal2-apple-darwin)
+ rustup target add x86_64-apple-darwin aarch64-apple-darwin ;;
+ *)
+ rustup target add "$WHEEL_TARGET" ;;
+ esac
+ - name: Install protoc
+ shell: bash
+ run: |
+ case "$RUNNER_OS" in
+ Linux) sudo apt-get update && sudo apt-get install -y
protobuf-compiler ;;
+ macOS) brew install protobuf ;;
+ Windows) choco install protoc -y ;;
+ esac
+ - name: Build abi3 wheel with maturin
+ # abi3-py39 produces one wheel per (platform, arch) valid for CPython
3.9+.
+ # --zig cross-links Linux wheels against glibc 2.17 (manylinux2014),
+ # including aarch64 builds from an x86_64 host.
+ shell: bash
+ env:
+ WHEEL_TARGET: ${{ inputs.target }}
+ WHEEL_USE_ZIG: ${{ inputs.zig }}
+ run: |
+ python -m pip install --upgrade maturin
+ args=(--release --out dist --target "$WHEEL_TARGET")
+ if [[ "$WHEEL_USE_ZIG" == "true" ]]; then
+ python -m pip install ziglang
+ args+=(--zig)
+ fi
+ maturin build "${args[@]}"
+ - uses: actions/upload-artifact@v4
+ with:
+ name: wheels-${{ inputs.target }}
+ path: dist/*.whl
+ retention-days: ${{ inputs.artifact_retention_days }}
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index 8a9d5ca..b628f49 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -133,6 +133,7 @@ jobs:
github.repository != 'apache/spark-connect-rust' ||
(inputs.release-version == '' && inputs.rc-count == '')
outputs:
+ repo: ${{ env.UPSTREAM_REPO }}
mode: ${{ steps.r.outputs.mode }}
version: ${{ steps.r.outputs.version }}
rc_tag: ${{ steps.r.outputs.rc_tag }}
@@ -247,7 +248,6 @@ jobs:
name: Build wheel (${{ matrix.target }})
needs: resolve
if: needs.resolve.outputs.mode == 'dryrun' || needs.resolve.outputs.mode
== 'rc'
- runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
@@ -256,52 +256,17 @@ jobs:
- { os: ubuntu-latest, target: aarch64-unknown-linux-gnu, zig: true
}
- { os: macos-latest, target: universal2-apple-darwin }
- { os: windows-latest, target: x86_64-pc-windows-msvc }
- steps:
- - uses: actions/checkout@v4
- with:
- repository: ${{ env.UPSTREAM_REPO }}
- ref: ${{ needs.resolve.outputs.sha }}
- - uses: actions/setup-python@v5
- with:
- python-version: "3.11"
- - name: Install Rust toolchain
- shell: bash
- run: |
- rustup toolchain install stable --profile minimal
- rustup default stable
- - name: Add Rust target(s)
- shell: bash
- run: |
- case "${{ matrix.target }}" in
- universal2-apple-darwin)
- rustup target add x86_64-apple-darwin aarch64-apple-darwin ;;
- *)
- rustup target add "${{ matrix.target }}" ;;
- esac
- - name: Install protoc
- shell: bash
- run: |
- case "$RUNNER_OS" in
- Linux) sudo apt-get update && sudo apt-get install -y
protobuf-compiler ;;
- macOS) brew install protobuf ;;
- Windows) choco install protoc -y ;;
- esac
- - name: Build abi3 wheel with maturin
- # abi3-py39 produces one wheel per (platform, arch) valid for CPython
3.9+.
- # `--zig` cross-links the Linux wheels against an old glibc
(manylinux2014).
- shell: bash
- run: |
- python -m pip install --upgrade maturin
- args=(--release --out dist --target "${{ matrix.target }}")
- if [[ "${{ matrix.zig }}" == "true" ]]; then
- python -m pip install ziglang
- args+=(--zig)
- fi
- maturin build "${args[@]}"
- - uses: actions/upload-artifact@v4
- with:
- name: wheels-${{ matrix.target }}
- path: dist/*.whl
+ uses: ./.github/workflows/build-wheel.yml
+ with:
+ runner: ${{ matrix.os }}
+ target: ${{ matrix.target }}
+ zig: ${{ matrix.zig || false }}
+ # Preserve the previous release job's six-hour limit.
+ timeout_minutes: 360
+ # Keep the release source canonical and pinned by resolve, including when
+ # the workflow is dispatched from a committer's fork.
+ source_repository: ${{ needs.resolve.outputs.repo }}
+ source_ref: ${{ needs.resolve.outputs.sha }}
build-sdist:
name: Build source distribution
diff --git a/.github/workflows/wheel-build-checks.yml
b/.github/workflows/wheel-build-checks.yml
new file mode 100644
index 0000000..23661e9
--- /dev/null
+++ b/.github/workflows/wheel-build-checks.yml
@@ -0,0 +1,69 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# Build-only checks complement the native ARM transport tests in rust.yml.
+name: Release wheel build checks
+
+on:
+ pull_request:
+ branches: [master, "branch-*"]
+ paths: &wheel-build-paths
+ - Cargo.lock
+ - '**/Cargo.toml'
+ - '**/build.rs'
+ - crates/spark-connect-proto/proto/**
+ - python/pyspark/**
+ - pyproject.toml
+ - .cargo/**
+ - rust-toolchain*
+ - .github/workflows/release.yml
+ - .github/workflows/build-wheel.yml
+ - .github/workflows/wheel-build-checks.yml
+ push:
+ branches: [master, "branch-*"]
+ paths: *wheel-build-paths
+ workflow_dispatch:
+
+permissions:
+ contents: read
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+ wheel-builds:
+ name: Check (${{ matrix.target }})
+ strategy:
+ fail-fast: false
+ matrix:
+ include:
+ # Exercise Linux ARM assembly through zig cc on an x86 host.
+ - { os: ubuntu-latest, target: aarch64-unknown-linux-gnu, zig: true }
+ # Exercise both the x86_64 and aarch64 slices.
+ - { os: macos-latest, target: universal2-apple-darwin }
+ uses: ./.github/workflows/build-wheel.yml
+ with:
+ runner: ${{ matrix.os }}
+ target: ${{ matrix.target }}
+ zig: ${{ matrix.zig || false }}
+ timeout_minutes: 45
+ artifact_retention_days: 1
+ source_repository: ${{ github.repository }}
+ source_ref: ${{ github.sha }}
diff --git a/dev/design/ci.md b/dev/design/ci.md
index 15e9887..cf9bd57 100644
--- a/dev/design/ci.md
+++ b/dev/design/ci.md
@@ -6,12 +6,13 @@ This document describes the GitHub Actions CI pipeline for
the Spark Connect Rus
## Overview
-The CI consists of two main workflows:
+The CI consists of three main workflows:
1. **Rust CI** (`.github/workflows/rust.yml`) - Build and test the pure-Rust
crates
2. **Python Connect Tests** (`.github/workflows/build_python_connect.yml`) -
Build the PyO3 wheel and run Spark Connect integration tests
+3. **Release wheel build checks** (`.github/workflows/wheel-build-checks.yml`)
- Build Linux aarch64 and macOS universal2 wheels using the release recipe
-Both workflows run on every push to main/master and on all pull requests.
+Rust CI and Python Connect Tests run on pushes and pull requests targeting
`master` and `branch-*`. Release wheel build checks run for changes to packaged
Python sources, protocol definitions, dependencies and build configuration, and
can also be dispatched manually.
## Rust CI (`rust.yml`)
@@ -126,6 +127,30 @@ Each failing test is a **gap in parity**. The fix process:
5. **Verify locally**: Run `scripts/run_official_connect_tests.py` with
`--target-pyspark ./python`
6. **Push & verify CI**: The workflow will show the test passing on the next
run
+## Release wheel build checks (`wheel-build-checks.yml`)
+
+These checks use the same reusable `build-wheel.yml` recipe as `release.yml`.
+An x86_64 Linux runner cross-builds the Linux aarch64 wheel with Maturin and
Zig.
+A macOS runner builds a universal2 wheel containing both x86_64 and aarch64
+slices. These are build-only checks; Rust CI separately executes the native ARM
+core tests.
+
+The workflow runs on pushes and pull requests targeting `master` or `branch-*`
+when packaged sources under `python/pyspark/`, protocol definitions under
+`crates/spark-connect-proto/proto/`, Cargo manifests or the lockfile, build
scripts,
+packaging/toolchain configuration, or the wheel workflows change.
+Ordinary Rust source changes do not trigger these focused checks. The scheduled
+`release.yml` dry run builds only `master`, so `branch-*` maintenance branches
have
+no scheduled backstop for Rust source changes. To validate a maintenance
branch,
+manually dispatch `release.yml` with `branch` set to that branch, empty
+`release-version` and `rc-count` inputs, and `finalize: false`.
+
+The shared recipe uses no dependency or build caches, so each run is a clean
build
+matching the release. Checks have a 45-minute timeout;
+releases retain their six-hour limit. Check artifacts expire after one day;
+release artifacts retain the repository default. Checkout credentials are not
+persisted in either build.
+
## Continuous improvement
The parity ledger (`dev/parity/inventory.csv`) is the source of truth:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]