HyukjinKwon opened a new pull request, #90:
URL: https://github.com/apache/spark-connect-rust/pull/90
### What changes were proposed in this pull request?
`pyspark-client-rust` installs the same `pyspark` API surface as the
reference
`pyspark-client`, backed by the native Rust engine (tonic). Because it is a
drop-in
under the `pyspark` import name, this PR makes it easy to tell which client
is in use
— at runtime and on PyPI:
- **Runtime markers**: `pyspark.__rust_client__` (`True`) and
`pyspark.__engine__`
(`"rust"`).
- **One-time connect log**: the first time a session connects, an `INFO`
line is
emitted on the `pyspark` logger naming the Rust-backed client and linking
the docs
(best-effort; never fails the connect).
- **New docs page** "Which client am I using?": `pyspark` vs
`pyspark-client` vs
`pyspark-client-rust`, how to tell them apart, where to file issues, and
server
compatibility — linked from the docs home and the Compatibility page.
- **PyPI metadata**: clearer description, more keywords, and a Documentation
URL.
No change to the API surface or to the behavior of existing Spark Connect
code.
### Why are the changes needed?
Follow-up from the dev@ discussion on the Python wrapper of the Rust Connect
client.
Because the package installs under the `pyspark` name as a drop-in, it
should be
unmistakable which client is active — so behavior and bug reports are
attributed to
the right project (`apache/spark-connect-rust`) rather than the reference
client, and
so the two clients' relationship is documented. Intended for RC2.
### Does this PR introduce _any_ user-facing change?
Yes, additive only: new `pyspark.__rust_client__` / `pyspark.__engine__`
attributes, a
one-time `INFO` log on first connect, a new documentation page, and updated
PyPI
package metadata. No existing API or result changes.
### How was this patch tested?
- `cargo check -p pyspark-rs` and `cargo fmt -p pyspark-rs -- --check` pass.
- `python -m py_compile` on `pyspark/__init__.py`; verified
`pyspark.__rust_client__`
/ `pyspark.__engine__` are present.
- `pyproject.toml` validated with `tomllib`.
- Docs nav, the new page, and its internal links reviewed (the site is built
in CI).
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
