HyukjinKwon commented on PR #51: URL: https://github.com/apache/spark-connect-rust/pull/51#issuecomment-5394462146
To ground the plan-building coverage against a concrete Spark version: the golden-proto tests that validate `plan.rs` are captured from the **Apache Spark 4.2.0** reference client, so every plan / expression / function they cover is asserted **byte-for-byte** against what pyspark 4.2.0 emits on the wire. So the two paths are complementary rather than one substituting for the other: - **golden-proto tests** — validate our plan building against Spark 4.2.0's exact protobufs (this is where `plan.rs` is covered), and - **official connect suite** (via `rust_transport_plugin.py`) — validates the transport + Arrow round-trip against a live 4.2.0 server. You're right that the official-suite path routes plan-building through upstream pyspark, so it isn't what covers `plan.rs` — the golden tests are. The honest remaining gap is that the 4.2.0 golden capture doesn't yet cover the newly-added items (the +100 functions and the closure/UDTF/streaming methods added in this stack); extending the capture to those is the way to close it, and that's tracked. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
