tkaymak opened a new pull request, #41426: URL: https://github.com/apache/beam/pull/41426
Closes #40949, part of #36841. The streaming translator now accepts a stateful ParDo with additional outputs. It mirrors `StatefulParDoTranslatorBatch`: with more than one consumed output the `transformWithState` processor emits wide rows with one column per tag, encoded with `oneOfEncoder`, and each output filters its column. The batch helpers `skipUnconsumedOutputs`, `tagsColumnIndex` and `createEncoders` are reused, so they and `ParDoTranslatorBatch` are now `@Internal public`. `ParDoStatefulProcessor` gets `singleOutput` and `multiOutput` factories, and `BeamStatefulProcessor` is generic on the row type with an abstract output receiver, like `StatefulDoFnGroupFunction`. The single output path and `GroupByKeyStatefulProcessor` behave as before. Like the stateless case in #41220, streaming datasets are not persisted, so each consumed tag runs its own `transformWithState` with its own state. This is not new with tags, one output with two consumers already does the same on master. The copies can see a key's elements in a different order or in different micro batches, and fire timers at different points, so the tags agree only for a DoFn whose output per tag does not depend on that. A local probe with a dedup DoFn (first element per key to main, the rest to a duplicates tag) gave consistent tags in 6 of 6 runs, local mode reads deterministically. An exchange at the fan out does not help, the copies get different operator ids, so this is tracked in #41425 and is not fixed by #41198. `ParDoTest$StateTests.testValueStateTaggedOutput` failed with the rejection before and passes now, its exclusion is removed. The new `StatefulParDoStreamingTest` case flattens a main output from `@ProcessElement` and an additional output from `@OnTimer` into one query and declares a third tag that is never consumed. Locally the streaming suite runs 213 tests with 0 failures, the batch Structured Streaming suites on Spark 4 and 3 and the full `:runners:spark:4:test` and `:runners:spark:3:test` pass. The trigger files run both ValidatesRunner PostCommits here. Merge order: it bumps the same Spark4 trigger line as #41256, whichever lands second rebases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
