This is an automated email from the ASF dual-hosted git repository.
dbtsai pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new cdf3aa8 [SPARK-60917][CONNECT][FOLLOWUP] Test aarch64 SHA builds in CI
cdf3aa8 is described below
commit cdf3aa887a5fb69b06b06f0ec1a4d6a3bf7fea3e
Author: DB Tsai <[email protected]>
AuthorDate: Thu Nov 5 11:43:13 2026 -0800
[SPARK-60917][CONNECT][FOLLOWUP] Test aarch64 SHA builds in CI
### What changes were proposed in this pull request?
Follow-up to #155 and
[SPARK-60917](https://issues.apache.org/jira/browse/SPARK-60917).
Add a focused `ubuntu-24.04-arm` job to Rust CI. It installs
`build-essential` and `protobuf-compiler`, then runs locked release transport
unit tests with the explicit `aarch64-unknown-linux-gnu` target. Its
Cargo/target cache includes the runner architecture.
Document the C compiler/assembler and archiver requirement for non-Windows
aarch64 source builds, including target tools for cross-compilation, and link
it from the Python and contributor build instructions. Add a short comment that
artifact upload retries must rebuild the consumed request stream.
### Why are the changes needed?
#155 enabled `sha2-asm` on non-Windows aarch64, but the existing PR jobs
run on x86_64. A native ARM job compiles the assembly dependency and runs the
transport tests on each PR and master/maintenance-branch push, exposing ARM
build failures before the release workflow needs the dependency.
### Does this PR introduce _any_ user-facing change?
The source-build documentation now describes the required target C tools.
Prebuilt Python wheel installation requires no C toolchain. There is no API or
upload behavior change.
### How was this patch tested?
- `actionlint` 1.7.12 passed for `.github/workflows/rust.yml`.
- On macOS aarch64: 107 transport and 414 Rust API release unit tests
passed. All client integration targets compiled with and without `wasm-udf`;
formatting and whitespace checks passed.
- 539 offline/mock Python tests passed, with eight live-server tests
skipped in that invocation. Focused native Arrow and official
creation/collection parity checks passed against Spark 4.3.0. API introspection
found only the intentionally absent `Column.to_plan`.
- Workspace Clippy reports existing errors in unchanged API/Python/test
files; none point into the files changed by this PR.
- The new [native Linux ARM
job](https://github.com/apache/spark-connect-rust/actions/runs/42088937091/job/127461503950)
passed on this PR: `sha2-asm` 0.6.4 compiled and 107 transport tests passed.
The release workflow retains its separate wheel builds.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (application version unavailable).
This pull request and its description were written by Isaac.
Closes #163 from dbtsai/codex/arm-sha-ci.
Authored-by: DB Tsai <[email protected]>
Signed-off-by: DB Tsai <[email protected]>
Co-authored-by: Isaac <[email protected]>
---
.github/workflows/rust.yml | 31 +++++++++++++++++++++++++++++++
crates/spark-connect-core/src/client.rs | 2 ++
docs/contributing.md | 4 +++-
docs/installation.md | 9 +++++++++
4 files changed, 45 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
index a064179..82c00df 100644
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -89,6 +89,37 @@ jobs:
# Enforce the "100% coverage, no followups" rule
run: bash scripts/audit_no_stubs.sh
+ aarch64-core:
+ name: Core transport tests (aarch64 Linux)
+ runs-on: ubuntu-24.04-arm
+ timeout-minutes: 20
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Install Rust toolchain
+ run: |
+ rustup toolchain install stable --profile minimal
+ rustup default stable
+
+ - name: Cache cargo & target
+ uses: actions/cache@v4
+ with:
+ path: |
+ ~/.cargo/registry
+ ~/.cargo/git
+ target
+ key: ${{ runner.os }}-${{ runner.arch }}-cargo-aarch64-${{
hashFiles('**/Cargo.lock') }}
+ restore-keys: ${{ runner.os }}-${{ runner.arch }}-cargo-aarch64-
+
+ - name: Install system dependencies
+ run: |
+ sudo apt-get update
+ sudo apt-get install -y build-essential protobuf-compiler
+
+ - name: Test native aarch64 transport
+ # Compile sha2-asm with the target C toolchain before release builds
need it.
+ run: cargo test --locked -p apache-spark-connect-core --release --lib
--target aarch64-unknown-linux-gnu
+
wasm-udf:
name: WASM UDF feature
runs-on: ubuntu-latest
diff --git a/crates/spark-connect-core/src/client.rs
b/crates/spark-connect-core/src/client.rs
index 03b94af..f0e03df 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -888,6 +888,8 @@ impl SparkConnectClient {
/// Send artifact frames and validate the response.
/// Cache uploads also require an acknowledgement for every requested
artifact.
+ /// The stream is consumed once; retries must rebuild the frames and
restart
+ /// AddArtifacts rather than resume a partially consumed stream.
async fn upload_artifact_requests(
&self,
requests: impl futures::Stream<Item =
spark_connect_proto::AddArtifactsRequest> + Send + 'static,
diff --git a/docs/contributing.md b/docs/contributing.md
index 9b5851d..07cb106 100644
--- a/docs/contributing.md
+++ b/docs/contributing.md
@@ -12,7 +12,9 @@ disabled. Reference the JIRA key in your pull request title,
e.g.
## Building and testing
-Requires a Rust toolchain and `protobuf-compiler`.
+Requires a Rust toolchain and `protobuf-compiler`. Non-Windows aarch64 builds
+also need the target C tools described in the
+[source build requirements](installation.md#build-from-source).
```bash
cargo build # build the library crates
diff --git a/docs/installation.md b/docs/installation.md
index 311ed30..b7aaf9a 100644
--- a/docs/installation.md
+++ b/docs/installation.md
@@ -38,6 +38,11 @@ cargo build --release
**Build requirements:**
- Rust toolchain (`rustup` / `cargo`)
- `protobuf-compiler`
+- For non-Windows aarch64 targets, a C compiler/assembler and archiver that
+ support the target, required by `sha2-asm`. Install `build-essential` on
+ Debian/Ubuntu or Xcode Command Line Tools on macOS. Cross-compilation needs
+ the corresponding target tools, configured for the `cc` crate (for example,
+ through its target-specific `CC_<target>` and `AR_<target>` variables).
## Python (Drop-in Replacement)
@@ -88,6 +93,10 @@ pip install dist/pyspark_client_rust-*.whl
- Python 3.9 or later
- Rust toolchain (`rustup` / `cargo`)
- `protobuf-compiler`
+- For non-Windows aarch64 targets, the C toolchain described in the
+ [Rust source build requirements](#build-from-source).
+
+Installing a prebuilt Python wheel does not require a C toolchain.
## Version and Compatibility
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]