Hyukjin Kwon created SPARK-58960:
------------------------------------
Summary: Rust core Spark Connect client with a drop-in Python
(pyspark) wrapper
Key: SPARK-58960
URL: https://issues.apache.org/jira/browse/SPARK-58960
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Restructure spark-connect-rust so the Rust core is the Spark Connect client and
a thin Python wrapper (published to PyPI as pyspark-client-rust, import package
pyspark) is layered over it via PyO3.
The repository becomes a four-crate workspace:
- spark-connect-proto: generated spark.connect.* gRPC/protobuf types
- spark-connect-core: transport (channel, retries, reattach, artifacts, errors)
- spark-connect: pure-Rust, synchronous DataFrame API mirroring PySpark
- pyspark-rs: PyO3 bindings that build the _pyspark extension module
plus python/pyspark/**, the drop-in pyspark package.
Also: versioned 4.2.0 (crates + wheel) to track the Spark release; adds
release.yml publishing to crates.io and PyPI using the committer-fork release
model; disables GitHub Issues in .asf.yaml in favor of JIRA; restores optional
to_datafusion()/to_polars() conversions.
PR: https://github.com/apache/spark-connect-rust/pull/49
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
