[
https://issues.apache.org/jira/browse/SPARK-58960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-58960:
---------------------------------
Epic Link: SPARK-59035
Affects Version/s: connect-rust-0.1.0
(was: 5.0.0)
> Rust core Spark Connect client with a drop-in Python (pyspark) wrapper
> ----------------------------------------------------------------------
>
> Key: SPARK-58960
> URL: https://issues.apache.org/jira/browse/SPARK-58960
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-0.1.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> Restructure spark-connect-rust so the Rust core is the Spark Connect client
> and a thin Python wrapper (published to PyPI as pyspark-client-rust, import
> package pyspark) is layered over it via PyO3.
> The repository becomes a four-crate workspace:
> - spark-connect-proto: generated spark.connect.* gRPC/protobuf types
> - spark-connect-core: transport (channel, retries, reattach, artifacts,
> errors)
> - spark-connect: pure-Rust, synchronous DataFrame API mirroring PySpark
> - pyspark-rs: PyO3 bindings that build the _pyspark extension module
> plus python/pyspark/**, the drop-in pyspark package.
> Also: versioned 4.2.0 (crates + wheel) to track the Spark release; adds
> release.yml publishing to crates.io and PyPI using the committer-fork release
> model; disables GitHub Issues in .asf.yaml in favor of JIRA; restores
> optional to_datafusion()/to_polars() conversions.
> PR: https://github.com/apache/spark-connect-rust/pull/49
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
