HyukjinKwon opened a new pull request, #99: URL: https://github.com/apache/spark-connect-rust/pull/99
### What changes were proposed in this pull request? Prepare `master` to track **Apache Spark 4.3.0** (currently at **RC1**). One PR covering: - **Version strings → 4.3.0:** `SPARK_VERSION` (`channel.rs`), `__version__` (`version.py`), `PROTO_VERSION.txt` (`Cargo.toml`/`pyproject.toml` were already 4.3.0). - **CI retarget (5 workflows) to the RC staging** — since 4.3.0 is not released to Maven Central / `archive.apache.org` / PyPI, the workflows now: download the **connect-bundled** tarball `spark-4.3.0-bin-hadoop3-connect.tgz` from `dist/dev/spark/v4.3.0-rc1-bin/` and start the server **without `--packages`** (jar bundled); clone official tests at the **`v4.3.0-rc1`** tag; install the reference pyspark from the staging sdist. `SPARK_HOME` is resolved by glob to tolerate the extracted dir name. - **Re-vendor upstream to `v4.3.0-rc1`:** `pandas`, `sql/worker`, `profiler.py`, `sql/pandas/serializers.py` re-synced byte-for-byte; **new `pyspark.messages` package** added; drift `SPARK_TAG` bumped (verified `diff -r` matches the tag). - **New 4.3.0 SQL functions:** `collect_union`, `to_base32`, `from_base32` (builder + dispatch arm + Python wrapper); **`DataFrame.zip`** exposed in PyO3 (the core `Zip` relation already existed). - **Docs/examples** bumped to 4.3.0. ### Why are the changes needed? Spark 4.3.0 adds new client APIs and refactors vendored modules; master already carried the 4.3.0 proto but still *targeted* 4.2.0 everywhere (CI, vendored tag, docs). This retargets the whole tracked surface to 4.3.0-rc1 so CI exercises the client against the actual RC. ### Does this PR introduce _any_ user-facing change? Yes — the client now targets Spark 4.3.0, adds `collect_union`/`to_base32`/`from_base32` and `DataFrame.zip`, and ships the vendored `pyspark.messages` package. ### How was this patch tested? Locally (this environment can't run a live 4.3.0 server): `cargo fmt --all --check` clean; `cargo build`/`cargo test --no-run` for `apache-spark-connect` + `pyspark-rs` compile clean; vendored re-sync verified `diff -r` byte-identical to `v4.3.0-rc1`. The server-gated parity/e2e/coverage/pandas/pipelines gates run on CI against the RC (retargeted here). ### Remaining follow-ups (need the live 4.3.0 parity environment; tracked under Epic SPARK-59102) - `variant_strip_nulls` (bool `include_arrays` arg): the generic column dispatch drops trailing scalar args (same pre-existing limitation as `first(..., ignorenulls)`), so it needs a dedicated binding + parity verification rather than a subtly-wrong generic arm. - `TimestampNTZNanos`/`TimestampLTZNanos` datatype semantics (currently mapped to `Time`/`Timestamp` in `from_proto`) — confirm against real pyspark 4.3.0. - Golden capture for the new functions and the full introspection-diff parity proof (AGENTS.md's mandatory gate), which need a running 4.3.0 server + pyspark 4.3.0. > Note: RC-staging URLs are transient (valid through the vote); this PR tracks 4.3.0-rc1 and should be re-pointed to the final artifacts once 4.3.0 GA is released. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
