tkaymak opened a new pull request, #41106: URL: https://github.com/apache/beam/pull/41106
Closes #40950, part of #36841. `Redistribute.byKey()` expands to a GroupByKey with `ReshuffleTrigger`, which the Spark 4 streaming translator rejects as a custom trigger. `RedistributeByKey` is now registered next to `Reshuffle` in `PipelineTranslatorCommon` and translated the same way, a repartition on the key, in batch and streaming. `Redistribute.arbitrarily()` builds on it. Windows, timestamps and panes are kept, since the rows move unchanged. The 7 `RedistributeTest` cases excluded from the streaming ValidatesRunner suite failed with "Custom triggers (ReshuffleTrigger())" and pass now. Locally the streaming suite runs 198 tests with 0 failures, the batch Structured Streaming suites on Spark 4 and 3 pass unchanged, as do the full `:runners:spark:4:test` and `:runners:spark:3:test`. The trigger files run both ValidatesRunner PostCommits here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
