tkaymak opened a new pull request, #41426:
URL: https://github.com/apache/beam/pull/41426

   Closes #40949, part of #36841.
   
   The streaming translator now accepts a stateful ParDo with additional 
outputs. It mirrors `StatefulParDoTranslatorBatch`: with more than one consumed 
output the `transformWithState` processor emits wide rows with one column per 
tag, encoded with `oneOfEncoder`, and each output filters its column. The batch 
helpers `skipUnconsumedOutputs`, `tagsColumnIndex` and `createEncoders` are 
reused, so they and `ParDoTranslatorBatch` are now `@Internal public`. 
`ParDoStatefulProcessor` gets `singleOutput` and `multiOutput` factories, and 
`BeamStatefulProcessor` is generic on the row type with an abstract output 
receiver, like `StatefulDoFnGroupFunction`. The single output path and 
`GroupByKeyStatefulProcessor` behave as before.
   
   Like the stateless case in #41220, streaming datasets are not persisted, so 
each consumed tag runs its own `transformWithState` with its own state. This is 
not new with tags, one output with two consumers already does the same on 
master. The copies can see a key's elements in a different order or in 
different micro batches, and fire timers at different points, so the tags agree 
only for a DoFn whose output per tag does not depend on that. A local probe 
with a dedup DoFn (first element per key to main, the rest to a duplicates tag) 
gave consistent tags in 6 of 6 runs, local mode reads deterministically. An 
exchange at the fan out does not help, the copies get different operator ids, 
so this is tracked in #41425 and is not fixed by #41198.
   
   `ParDoTest$StateTests.testValueStateTaggedOutput` failed with the rejection 
before and passes now, its exclusion is removed. The new 
`StatefulParDoStreamingTest` case flattens a main output from `@ProcessElement` 
and an additional output from `@OnTimer` into one query and declares a third 
tag that is never consumed. Locally the streaming suite runs 213 tests with 0 
failures, the batch Structured Streaming suites on Spark 4 and 3 and the full 
`:runners:spark:4:test` and `:runners:spark:3:test` pass. The trigger files run 
both ValidatesRunner PostCommits here.
   
   Merge order: it bumps the same Spark4 trigger line as #41256, whichever 
lands second rebases.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to