tkaymak opened a new pull request, #41220: URL: https://github.com/apache/beam/pull/41220
Part of #40949, part of #36841. The streaming translator now accepts a stateless ParDo with additional outputs. It reuses the batch translation, the DoFn writes wide rows with one column per tag and each output filters its column. The wide rows are persisted only in batch, since `persist` on a streaming Dataset throws, the same guard as `PipelineTranslator`. A stateful ParDo with additional outputs is still rejected, `BeamStatefulProcessor` would merge every tag into the main output, so #40949 stays open for that case. `Combine.perKey().withHotKeyFanout()` is rejected in streaming for now. With this change it would translate, but `CombineTest$BasicTests.testHotKeyCombining` then computes a wrong mean (2.0645 instead of 2.0), tracked in #41199. In streaming each consumed tag reruns the DoFn, as any shared upstream does today, tracked in #41198 with measured counts. Of the 17 tests excluded from the streaming ValidatesRunner suite under #40949, which failed with "ParDo with additional outputs", 14 pass now, among them all 6 `MetricsTest$AttemptedMetricTests`. The two hot key tests stay excluded under #41199 and `testValueStateTaggedOutput` under #40949. Locally the streaming suite runs 212 tests with 0 failures, the batch Structured Streaming suites on Spark 4 and 3 and the full `:runners:spark:4:test` and `:runners:spark:3:test` pass. The trigger files run both ValidatesRunner PostCommits here. The streaming suite has a known flake on master, addressed by #41137. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
