This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 04be01a [SPARK-59077][INFRA] Make the release workflow safe for a
direct release
04be01a is described below
commit 04be01ab5e52e451aa10268f96581581ba8e9c2b
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Aug 28 16:52:10 2026 +0900
[SPARK-59077][INFRA] Make the release workflow safe for a direct release
## What changes were proposed in this pull request?
Hardens `.github/workflows/release.yml` so a committer can cut a release
directly (via a `v*` tag push or a `dry_run=false` dispatch) without the risk
of a botched, irreversible publish.
- **Dry-run now actually rehearses the crate publish.** Previously a
`dry_run=true` dispatch only built wheels — the crate publish was never
exercised, so the first proof that `cargo publish` worked was the real,
irreversible release. `publish-crates` now always runs and, in dry-run,
executes `cargo publish --dry-run -p apache-spark-connect-proto -p
apache-spark-connect-core -p apache-spark-connect`. cargo (>= 1.90) orders and
verify-builds these together against the local workspace, [...]
- **Idempotent re-runs.** The real crate publish skips any crate whose
current version is already on crates.io, and wheels/sdist upload with `twine
upload --skip-existing`. A re-run after a partial failure no longer aborts on
"already exists".
- **Portable, complete artifacts.** Wheels are built from a target matrix:
Linux `x86_64` + `aarch64` cross-linked against an old glibc via `--zig`
(manylinux2014), macOS `universal2`, and Windows `x64`, plus a source
distribution (sdist) as the install fallback. This replaces the single
`ubuntu-latest` wheel whose glibc floor tracked the runner (and which PyPI
could reject), and the arm64-only macOS wheel.
- **`concurrency` guard** so two releases never overlap and a publish in
flight is never cancelled.
- **Stronger version gate.** `validate-version` now also asserts the
internal path-dep version pins (`spark-connect-core`/`spark-connect` → `proto`)
track the workspace version.
- **Crate metadata.** The three published crates inherit
`repository`/`homepage`/`authors` from the workspace, fixing the `manifest has
no ... repository` warning so the crates.io pages link back to the repo.
The workflow stays first-party-actions-only (`actions/*` +
`twine`/`maturin` via pip), consistent with the existing ASF Actions-policy
note in the file.
## Why are the changes needed?
A "dry run" that skips the crate publish gives false confidence: crates.io
versions are permanent (yank-only), and the previous workflow offered no
rehearsal and no safe recovery from a partial publish. The Linux wheel was also
tied to the runner's glibc with no aarch64/sdist coverage.
## Does this PR introduce _any_ user-facing change?
No. This only changes the release/CI workflow and crate packaging metadata;
it does not change any client API.
## How was this patch tested?
Locally rehearsed the crate side offline against a populated cargo cache
(network to crates.io is the only piece a runner adds):
- `cargo package -p apache-spark-connect-proto -p apache-spark-connect-core
-p apache-spark-connect` packaged all three and verify-built each in dependency
order, resolving `core`/`connect` against the **locally-packaged** siblings —
the same resolution `cargo publish --dry-run` uses.
- Inspected the shipped `Cargo.toml`: path deps stripped, `version` pins
kept, and `repository`/`homepage`/`authors` now present (no more "manifest has
no repository" warning).
- `validate-version` version/tag/path-pin logic checked against the current
`4.2.0`.
- `release.yml` validated as well-formed YAML.
Wheel/sdist build steps use standard `maturin` invocations and are
exercised by the workflow's own dry-run.
This pull request and its description were written by Isaac.
Closes #79 from HyukjinKwon/release-actions-review.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.github/workflows/release.yml | 138 +++++++++++++++++++++++++++++-----
crates/spark-connect-core/Cargo.toml | 3 +
crates/spark-connect-proto/Cargo.toml | 3 +
crates/spark-connect/Cargo.toml | 3 +
pyproject.toml | 8 +-
5 files changed, 133 insertions(+), 22 deletions(-)
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index 0f99673..fede7a5 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -30,8 +30,11 @@
# 4. Dispatch this workflow with dry_run=true to rehearse (default), then
push
# a matching `v<version>` tag (or dispatch with dry_run=false) to publish.
#
-# A dry run builds and validates artifacts without publishing. A tag push (or
an
-# explicit dry_run=false dispatch) publishes to crates.io and PyPI.
+# A dry run fully rehearses BOTH sides without publishing: it packages and
+# verify-builds every crate in dependency order (`cargo publish --dry-run`) and
+# builds every wheel and the sdist. A tag push (or an explicit dry_run=false
+# dispatch) additionally publishes to crates.io and PyPI. Both publish steps
are
+# idempotent, so a re-run after a partial failure skips what already landed.
name: Release
@@ -48,6 +51,12 @@ on:
permissions:
contents: read
+# Only one release may run at a time, and a publish in flight is never
cancelled
+# (cancelling mid-publish is what leaves crates.io/PyPI half-updated).
+concurrency:
+ group: release
+ cancel-in-progress: false
+
jobs:
validate-version:
name: Validate version
@@ -56,6 +65,7 @@ jobs:
- uses: actions/checkout@v4
- name: Cargo and pyproject versions agree (and match the tag, if any)
run: |
+ set -euo pipefail
CARGO_VER=$(grep -m1 '^version' Cargo.toml | awk -F'"' '{print $2}')
PYPROJECT_VER=$(grep -m1 '^version' pyproject.toml | awk -F'"'
'{print $2}')
echo "Cargo.toml=${CARGO_VER} pyproject.toml=${PYPROJECT_VER}"
@@ -63,6 +73,17 @@ jobs:
echo "::error::Cargo.toml (${CARGO_VER}) and pyproject.toml
(${PYPROJECT_VER}) versions differ"
exit 1
fi
+ # Internal path-dependency version pins (e.g. spark-connect-core ->
proto)
+ # must track the workspace version, or `cargo publish` embeds a stale
+ # requirement into the published crate.
+ BAD=$(grep -rEn 'path = "\.\./spark-connect[^"]*"'
crates/*/Cargo.toml \
+ | grep -E 'version = "' \
+ | grep -v "version = \"${CARGO_VER}\"" || true)
+ if [[ -n "${BAD}" ]]; then
+ echo "::error::internal path-dep version pin(s) do not match
${CARGO_VER}:"
+ echo "${BAD}"
+ exit 1
+ fi
if [[ "${GITHUB_REF}" == refs/tags/* ]]; then
TAG="${GITHUB_REF##*/}"
if [[ "${TAG}" != "v${CARGO_VER}" ]]; then
@@ -74,7 +95,6 @@ jobs:
publish-crates:
name: Publish Rust crates to crates.io
needs: validate-version
- if: github.event_name == 'push' || (github.event_name ==
'workflow_dispatch' && inputs.dry_run == false)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
@@ -84,36 +104,88 @@ jobs:
rustup default stable
- name: Install protoc
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
- - name: cargo publish (dependency order)
+ - name: cargo publish (dependency-ordered; dry-run unless releasing)
env:
CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }}
+ # true only for an explicit dry-run dispatch; tag pushes and an
explicit
+ # dry_run=false dispatch both evaluate to "false" and publish for
real.
+ DRY_RUN: ${{ github.event_name == 'workflow_dispatch' &&
inputs.dry_run }}
run: |
- # Publish leaf-first so each crate's path+version dependencies
already
- # exist on crates.io. Allow the index a moment to catch up between
crates.
- for crate in apache-spark-connect-proto apache-spark-connect-core
apache-spark-connect; do
- echo "::group::cargo publish -p ${crate}"
- cargo publish -p "${crate}"
+ set -euo pipefail
+ # Leaf-first; cargo (>=1.90) orders and verify-builds these together,
+ # resolving siblings from the local workspace.
+ CRATES=(apache-spark-connect-proto apache-spark-connect-core
apache-spark-connect)
+ VER=$(grep -m1 '^version' Cargo.toml | awk -F'"' '{print $2}')
+
+ if [[ "${DRY_RUN}" == "true" ]]; then
+ # Rehearse the real publish. Because siblings resolve from the
local
+ # workspace, this works even before anything exists on crates.io.
+ pkgs=(); for c in "${CRATES[@]}"; do pkgs+=(-p "$c"); done
+ echo "::group::cargo publish --dry-run ${pkgs[*]}"
+ cargo publish --dry-run "${pkgs[@]}"
echo "::endgroup::"
- sleep 30
+ echo "Dry run only — nothing published."
+ exit 0
+ fi
+
+ # Real publish. Skip any crate whose current version is already on
+ # crates.io so a re-run after a partial failure is idempotent.
+ pkgs=()
+ for c in "${CRATES[@]}"; do
+ if curl -sf -H "User-Agent: spark-connect-rust-release
([email protected])" \
+ "https://crates.io/api/v1/crates/${c}/${VER}"; >/dev/null; then
+ echo "${c} ${VER} is already on crates.io — skipping"
+ else
+ pkgs+=(-p "$c")
+ fi
done
+ if [[ ${#pkgs[@]} -eq 0 ]]; then
+ echo "All crates already published at ${VER}; nothing to do."
+ exit 0
+ fi
+ echo "::group::cargo publish ${pkgs[*]}"
+ # cargo publishes in dependency order and waits for the index between
+ # crates, so no manual sleep is needed.
+ cargo publish "${pkgs[@]}"
+ echo "::endgroup::"
build-wheels:
- name: Build wheel (${{ matrix.os }})
+ name: Build wheel (${{ matrix.target }})
needs: validate-version
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
- os: [ubuntu-latest, macos-latest, windows-latest]
+ include:
+ - os: ubuntu-latest
+ target: x86_64-unknown-linux-gnu
+ zig: true
+ - os: ubuntu-latest
+ target: aarch64-unknown-linux-gnu
+ zig: true
+ - os: macos-latest
+ target: universal2-apple-darwin
+ - os: windows-latest
+ target: x86_64-pc-windows-msvc
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install Rust toolchain
+ shell: bash
run: |
rustup toolchain install stable --profile minimal
rustup default stable
+ - name: Add Rust target(s)
+ shell: bash
+ run: |
+ case "${{ matrix.target }}" in
+ universal2-apple-darwin)
+ rustup target add x86_64-apple-darwin aarch64-apple-darwin ;;
+ *)
+ rustup target add "${{ matrix.target }}" ;;
+ esac
- name: Install protoc
shell: bash
run: |
@@ -123,19 +195,47 @@ jobs:
Windows) choco install protoc -y ;;
esac
- name: Build abi3 wheel with maturin
- # abi3-py39 produces a single wheel per platform valid for CPython
3.9+.
+ # abi3-py39 produces a single wheel per (platform, arch) valid for
+ # CPython 3.9+. `--zig` cross-links the Linux wheels against an old
glibc
+ # (manylinux2014), so they install on essentially any modern Linux
rather
+ # than only on the runner's glibc.
shell: bash
run: |
python -m pip install --upgrade maturin
- maturin build --release --out dist
+ args=(--release --out dist --target "${{ matrix.target }}")
+ if [[ "${{ matrix.zig }}" == "true" ]]; then
+ python -m pip install ziglang
+ args+=(--zig)
+ fi
+ maturin build "${args[@]}"
- uses: actions/upload-artifact@v4
with:
- name: wheels-${{ matrix.os }}
+ name: wheels-${{ matrix.target }}
path: dist/*.whl
+ build-sdist:
+ name: Build source distribution
+ needs: validate-version
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v4
+ - uses: actions/setup-python@v5
+ with:
+ python-version: "3.11"
+ - name: Build sdist with maturin
+ # sdist packages source only (no compile), so protoc is not needed
here.
+ # It is the install fallback for platforms without a prebuilt wheel.
+ run: |
+ python -m pip install --upgrade maturin
+ maturin sdist --out dist
+ - uses: actions/upload-artifact@v4
+ with:
+ name: wheels-sdist
+ path: dist/*.tar.gz
+
publish-wheels:
- name: Publish wheel to PyPI
- needs: build-wheels
+ name: Publish wheel + sdist to PyPI
+ needs: [build-wheels, build-sdist]
if: github.event_name == 'push' || (github.event_name ==
'workflow_dispatch' && inputs.dry_run == false)
runs-on: ubuntu-latest
steps:
@@ -144,6 +244,7 @@ jobs:
python-version: "3.11"
- uses: actions/download-artifact@v4
with:
+ # Matches every wheels-<target> artifact plus wheels-sdist.
pattern: wheels-*
path: dist
merge-multiple: true
@@ -155,4 +256,5 @@ jobs:
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
run: |
python -m pip install --upgrade twine
- twine upload dist/*
+ # --skip-existing makes a re-run after a partial upload safe.
+ twine upload --skip-existing dist/*
diff --git a/crates/spark-connect-core/Cargo.toml
b/crates/spark-connect-core/Cargo.toml
index cd714c3..6f4c1d8 100644
--- a/crates/spark-connect-core/Cargo.toml
+++ b/crates/spark-connect-core/Cargo.toml
@@ -3,6 +3,9 @@ name = "apache-spark-connect-core"
version.workspace = true
edition.workspace = true
license.workspace = true
+repository.workspace = true
+homepage.workspace = true
+authors.workspace = true
description = "Spark Connect client transport: gRPC channel, retries,
reattach, artifacts, config, errors"
[lib]
diff --git a/crates/spark-connect-proto/Cargo.toml
b/crates/spark-connect-proto/Cargo.toml
index 2ba3f8d..604ff89 100644
--- a/crates/spark-connect-proto/Cargo.toml
+++ b/crates/spark-connect-proto/Cargo.toml
@@ -3,6 +3,9 @@ name = "apache-spark-connect-proto"
version.workspace = true
edition.workspace = true
license.workspace = true
+repository.workspace = true
+homepage.workspace = true
+authors.workspace = true
description = "Generated Spark Connect protobuf/gRPC types (spark.connect.*)"
[lib]
diff --git a/crates/spark-connect/Cargo.toml b/crates/spark-connect/Cargo.toml
index 5042122..92ad84f 100644
--- a/crates/spark-connect/Cargo.toml
+++ b/crates/spark-connect/Cargo.toml
@@ -3,6 +3,9 @@ name = "apache-spark-connect"
version.workspace = true
edition.workspace = true
license.workspace = true
+repository.workspace = true
+homepage.workspace = true
+authors.workspace = true
description = "Pure-Rust Spark Connect DataFrame client mirroring the PySpark
API surface"
[lib]
diff --git a/pyproject.toml b/pyproject.toml
index fb9f434..0ec2245 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -32,10 +32,10 @@ dependencies = [
]
# cloudpickle is vendored at pyspark/cloudpickle (matching Apache Spark's own
# bundling), so it is intentionally NOT an external dependency.
-
-[project.optional-dependencies]
-grpc = ["grpcio>=1.48.0"]
-all = ["grpcio>=1.48.0"]
+#
+# No optional-dependencies: unlike upstream PySpark's `connect` extra, this
+# package has no pure-Python gRPC client — the Rust extension
(pyspark._pyspark)
+# speaks gRPC via tonic, so grpcio is never imported at runtime.
[project.urls]
Homepage = "https://spark.apache.org/";
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
