HyukjinKwon opened a new pull request, #232: URL: https://github.com/apache/spark-connect-rust/pull/232
### What changes were proposed in this pull request? [SPARK-61038](https://issues.apache.org/jira/browse/SPARK-61038): `DataStreamWriter.trigger` now takes its triggers, including `realTime`, as PySpark Connect does. `df.writeStream.trigger(...)`: - takes exactly one of `processingTime`, `once`, `continuous`, `availableNow` and `realTime`, and raises `ONLY_ALLOW_SINGLE_TRIGGER` for none or more than one - takes an interval or duration as a non-empty string, which it strips, else `VALUE_NOT_NON_EMPTY_STR` - takes `once` and `availableNow` as `True`, else `VALUE_NOT_ALLOWED` - sets the `real_time_batch_duration` of the `WriteStreamOperationStart` for `trigger(realTime="5 seconds")`, the real-time mode of the query The core `Trigger` has a `RealTime` variant, and `Trigger::to_proto` serializes each trigger. ### Why are the changes needed? The client: - raised `NotImplementedError: the realTime trigger is not yet supported by this client` - took the first trigger of several - raised plain `ValueError`s for no trigger and for `once=False`, and `TypeError`s for intervals that are not strings - sent empty and unstripped intervals ### Does this PR introduce _any_ user-facing change? Yes. `trigger` takes the triggers, and raises the errors, of PySpark 4.3.0. ### How was this patch tested? - A new live test starts a query of the real-time trigger, which the server rejects for the memory sink with `STREAMING_REAL_TIME_MODE.OPERATOR_OR_SINK_NOT_IN_ALLOWLIST`, as with official PySpark 4.3.0. It fails without this change. - New offline tests check the errors of `trigger`, and the triggers that `start` sends to a mock Connect server. - A new lib test checks the proto of each trigger. - On a local server, 16 cases of `trigger` behave as in official PySpark 4.3.0. - The lib tests of both crates (569 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1807 passed, 8 skipped), and the live drop-in and Arrow suites (106 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
