HyukjinKwon opened a new pull request, #223: URL: https://github.com/apache/spark-connect-rust/pull/223
### What changes were proposed in this pull request? [SPARK-61022](https://issues.apache.org/jira/browse/SPARK-61022): `df.isStreaming` now asks the server, by an `IsStreaming` AnalyzePlan request of the plan, as PySpark Connect's does. It is true for a DataFrame with a streaming source anywhere in its plan. In the Rust API, `DataFrame::is_streaming` returns the server's answer, or the error of the request, as a `Result<bool>`. ### Why are the changes needed? `isStreaming` checked only whether the root of the plan was a streaming read. So it was False for `spark.readStream.format("rate").load().select("value")`, and for the filters, unions and joins of a streaming DataFrame, which PySpark 4.3.0 reports as streaming. ### Does this PR introduce _any_ user-facing change? Yes. `isStreaming` is True for every DataFrame of a stream, as in PySpark 4.3.0. In the Rust API, `DataFrame::is_streaming` returns a `Result<bool>`. ### How was this patch tested? - A new live test checks `isStreaming` of a stream, of a select, a filter and a union of it, and of a join with it. It passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks the `IsStreaming` requests sent to a mock Connect server, which now answers them. - The e2e coverage test checks a select of a stream. A rate source with a schema, and a named source, are now errors of `is_streaming`, as the server rejects them when it analyzes the plan, as for PySpark. The test creates the directories of its file sources. - The lib tests, every integration test against a local Spark 4.3.0 server, the offline Python suites, and the live drop-in and Arrow suites and live cases. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
