HyukjinKwon opened a new pull request, #50: URL: https://github.com/apache/spark-connect-rust/pull/50
## Summary **Part 1 of 2.** Introduces the native Rust client for Apache Spark Connect as a Cargo workspace of three library crates, replacing the previous single `spark-connect-rs` crate. The drop-in Python (`pyspark`) wrapper is added in a follow-up PR stacked on this one. ## What's in this PR (Rust core only) - **`spark-connect-proto`** — generated `spark.connect.*` gRPC/protobuf types. - **`spark-connect-core`** — transport: channel, retries, reattach, artifacts, errors. - **`spark-connect`** — pure-Rust, **synchronous** DataFrame API mirroring PySpark (session, dataframe, column, functions, plan, group, catalog, window, readwriter, streaming, types). Optional `datafusion` / `polars` features add `to_datafusion()` / `to_polars()` conversions. - **`examples/`** — SQL, readers/writers, streaming, Delta Lake, Databricks. - **Golden-proto tests** (`tests/golden/`) — plans, expressions, and all SQL functions asserted byte-for-byte against captured reference protos. - **CI** — `cargo build`/`test`, `rustfmt`/`clippy`, no-stub audit; a crates.io `release.yml` using the Apache Spark committer-fork model. - **`.asf.yaml`** — GitHub Issues disabled in favor of ASF JIRA. - Versioned **4.2.0** to track the Apache Spark release; developed and validated against Spark 4.2.0. ## Verification - `cargo build`, `cargo build -p examples`, and `cargo test` (21 suites, 0 failures) all pass; no-stub audit passes; `datafusion` feature compiles. JIRA: https://issues.apache.org/jira/browse/SPARK-58960 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
