This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new b1c2af9 [SPARK-59071][INFRA] Measure Rust coverage against a live
Connect server
b1c2af9 is described below
commit b1c2af938e3fc09779d412593f5c157bb4f4e466
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Aug 28 15:07:24 2026 +0900
[SPARK-59071][INFRA] Measure Rust coverage against a live Connect server
### What changes were proposed in this pull request?
The Coverage workflow (`.github/workflows/coverage.yml`) ran `cargo
llvm-cov -p apache-spark-connect` with **no Spark Connect server** and no
`SPARK_REMOTE` set. The behavioral `e2e_*.rs` tests self-gate on `SPARK_REMOTE`:
```rust
fn should_run() -> bool { std::env::var("SPARK_REMOTE").is_ok() }
#[test] fn dataframe_transforms_and_setops() { if !should_run() { return; }
... }
```
So in the coverage job those tests **no-op**, and the runtime execution
paths (collect/count/RPC-backed DataFrame, functions, session, catalog, ...)
are never exercised. Only unit + golden (plan-serialization) tests run, so the
Rust coverage badge reported **~73.3%**, well below the real coverage. Those
same e2e tests already run in `build_python_connect.yml`, which downloads Spark
4.2.0, starts a Connect server, and sets `SPARK_REMOTE`.
This PR makes the coverage job start a Spark 4.2.0 Connect server
(mirroring `build_python_connect.yml`) and measure coverage in two accumulated
`cargo-llvm-cov` passes that are then combined:
1. the default unit + golden tests, run in parallel;
2. the behavioral e2e suite against the live server (`SPARK_REMOTE` set),
run serially since the tests share one server session.
`--no-report` accumulates raw coverage from each pass; `cargo llvm-cov
report` merges both into one JSON summary consumed by the existing badge parser.
Supporting changes: JDK 17 + the Python deps the server's workers need, a
shared Spark-dist cache key (reused from the parity job), a 45-minute job
timeout, and a server-log artifact for debugging. The two duplicate Python
setup steps are consolidated into one.
### Why are the changes needed?
So the Rust coverage badge reflects the coverage actually achieved when the
runtime paths are exercised, instead of undercounting because the e2e tests
silently no-op without a server.
### Does this PR introduce _any_ user-facing change?
No. CI-only change.
### How was this patch tested?
CI (the coverage workflow itself).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
Closes #73 from HyukjinKwon/SPARK-59071.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.github/workflows/coverage.yml | 106 ++++++++++++++++++++++++++++++++++++-----
1 file changed, 93 insertions(+), 13 deletions(-)
diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml
index 83b327f..9140644 100644
--- a/.github/workflows/coverage.yml
+++ b/.github/workflows/coverage.yml
@@ -32,11 +32,15 @@ on:
env:
CARGO_TERM_COLOR: always
+ SPARK_VERSION: "4.2.0"
jobs:
coverage:
name: Measure and publish coverage badges
runs-on: ubuntu-latest
+ # Bound the job: coverage now runs the behavioral e2e suite against a live
Spark
+ # Connect server, so a hung test fails the job instead of running to the
6-hour default.
+ timeout-minutes: 45
permissions:
contents: write # Needed to push to badges branch
steps:
@@ -67,10 +71,89 @@ jobs:
- name: Install cargo-llvm-cov
uses: taiki-e/install-action@cargo-llvm-cov
+ - name: Set up JDK 17
+ uses: actions/setup-java@v4
+ with:
+ distribution: temurin
+ java-version: "17"
+
+ - name: Set up Python 3.11
+ uses: actions/setup-python@v5
+ with:
+ python-version: "3.11"
+ cache: pip
+
+ - name: Install Python dependencies
+ # Two sets: (1) build deps for the Python-coverage half
(maturin/pytest/coverage);
+ # (2) the runtime deps the Connect server's Python workers need for
arrow/pandas
+ # execution (mirrors build_python_connect.yml).
+ run: |
+ python -m pip install --upgrade pip
+ python -m pip install maturin pytest coverage
+ python -m pip install pandas pyarrow numpy grpcio grpcio-status
googleapis-common-protos cloudpickle zstandard py4j
+
+ - name: Cache Spark ${{ env.SPARK_VERSION }} distribution
+ uses: actions/cache@v4
+ with:
+ path: ~/spark-dist
+ key: spark-dist-${{ env.SPARK_VERSION }}-hadoop3
+
+ - name: Download Apache Spark ${{ env.SPARK_VERSION }} (Connect server)
+ id: spark-download
+ run: |
+ PKG="spark-${SPARK_VERSION}-bin-hadoop3"
+ if [ ! -d "${HOME}/spark-dist/${PKG}" ]; then
+
URL="https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/${PKG}.tgz";
+ mkdir -p "${HOME}/spark-dist"
+ echo "Downloading ${URL}"
+ wget -q -O "${HOME}/spark-dist/${PKG}.tgz" "${URL}"
+ tar -xzf "${HOME}/spark-dist/${PKG}.tgz" -C "${HOME}/spark-dist"
+ rm -f "${HOME}/spark-dist/${PKG}.tgz"
+ else
+ echo "Using cached Spark distribution."
+ fi
+ echo "SPARK_HOME=${HOME}/spark-dist/${PKG}" >> "$GITHUB_OUTPUT"
+
+ - name: Start Spark Connect server
+ env:
+ # The server spawns Python workers for arrow/pandas execution; they
need
+ # pandas + pyarrow (installed above) plus pyspark (from
SPARK_HOME/python).
+ PYSPARK_PYTHON: python
+ PYSPARK_DRIVER_PYTHON: python
+ run: |
+ SPARK_HOME="${{ steps.spark-download.outputs.SPARK_HOME }}"
+ echo "Starting Connect server on sc://localhost:15002"
+ "${SPARK_HOME}/sbin/start-connect-server.sh" \
+ --packages "org.apache.spark:spark-connect_2.13:${SPARK_VERSION}" \
+ --driver-memory 3g \
+ --conf spark.sql.session.localRelationSizeLimit=1073741824 \
+ > /tmp/connect-server.log 2>&1 &
+ for i in $(seq 1 90); do
+ if (echo > /dev/tcp/localhost/15002) 2>/dev/null; then
+ echo "Connect server is up (after ${i}s)."; break
+ fi
+ if [ "$i" -eq 90 ]; then echo "Server failed to start:"; cat
/tmp/connect-server.log; exit 1; fi
+ sleep 1
+ done
+
- name: Generate Rust coverage report
+ # The behavioral e2e_* tests self-gate on SPARK_REMOTE and no-op
without a live
+ # server, so a server-less coverage run undercounts the runtime
execution paths
+ # (collect/count/RPC-backed DataFrame, functions, session, catalog,
...) and the
+ # badge showed only ~73%. Run coverage in two accumulated passes and
combine:
+ # 1. the default unit + golden (plan-serialization) tests, run in
parallel;
+ # 2. the behavioral e2e suite against the live server (SPARK_REMOTE
set), run
+ # serially since the tests share one server session.
+ # `--no-report` accumulates raw coverage; `report` merges both passes
into one JSON.
run: |
- cargo llvm-cov --no-cfg-coverage -p apache-spark-connect \
- --json --output-path coverage-rust-llvm.json
+ cargo llvm-cov clean --workspace
+ cargo llvm-cov --no-cfg-coverage --no-report -p apache-spark-connect
+ SPARK_REMOTE=sc://localhost:15002 \
+ cargo llvm-cov --no-cfg-coverage --no-report -p
apache-spark-connect \
+ --test e2e_functions --test e2e_columns --test e2e_dataframe \
+ --test e2e_coverage --test e2e_integration \
+ -- --test-threads=1
+ cargo llvm-cov report --json --output-path coverage-rust-llvm.json
- name: Parse Rust coverage and create badge JSON
# Extract line coverage % from llvm-cov JSON output and convert to
shields.io
@@ -127,17 +210,6 @@ jobs:
print(f"Rust coverage: {coverage_pct:.1f}% ({color})")
PYTHON_EOF
- - name: Install Python and dependencies
- uses: actions/setup-python@v5
- with:
- python-version: "3.11"
- cache: pip
-
- - name: Install Python build dependencies
- run: |
- python -m pip install --upgrade pip
- python -m pip install maturin pytest coverage
-
- name: Build Python extension
run: |
cargo build -p pyspark-rs --release
@@ -228,3 +300,11 @@ jobs:
coverage-rust-badge.json
coverage-python-badge.json
retention-days: 5
+
+ - name: Upload Connect server log
+ if: always()
+ uses: actions/upload-artifact@v4
+ with:
+ name: coverage-connect-server-log
+ path: /tmp/connect-server.log
+ if-no-files-found: warn
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
