HyukjinKwon opened a new pull request, #229: URL: https://github.com/apache/spark-connect-rust/pull/229
### What changes were proposed in this pull request? [SPARK-61034](https://issues.apache.org/jira/browse/SPARK-61034): `transformWithState` now sends its output schema, as PySpark Connect does. PySpark sends the `outputStructType` of `df.groupBy(...).transformWithState(statefulProcessor, outputStructType, outputMode, timeMode)` as the `output_schema` of the `TransformWithStateInfo` of its `GroupMap`, as it does for `transformWithStateInPandas`. The server reads it as the schema of the rows that the processor returns. The Rust API `GroupedData::transform_with_state` now sends the return type of its UDF, of `outputStructType`, there. ### Why are the changes needed? The client sent no output schema for `transformWithState`, so the server failed to analyze the DataFrame: ``` [CONNECT_INVALID_PLAN.DATA_TYPE_UNSUPPORTED_PROTO_TO_CATALYST] The Spark Connect plan is invalid. Does not support convert KIND_NOT_SET to catalyst types. ``` ### Does this PR introduce _any_ user-facing change? Yes. `transformWithState` works, as in PySpark 4.3.0. ### How was this patch tested? - A new live test runs a streaming query of `transformWithState`, of an output schema given as a DDL string and as a StructType, and checks its rows. It passes with official PySpark 4.3.0 and fails without this change. - A new lib test checks the output schema of the `TransformWithStateInfo` that `transform_with_state` sends. - The lib tests of both crates (569 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1797 passed, 8 skipped), and the live drop-in and Arrow suites (105 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
