tkaymak opened a new issue, #41425:
URL: https://github.com/apache/beam/issues/41425

   ### What happened?
   
   In streaming mode the Spark 4 Structured Streaming runner does not persist 
datasets (`persist` on a streaming Dataset throws), so a stateful ParDo whose 
output is consumed more than once runs one `transformWithState` per consumer, 
each with its own state store. This applies to one output with two consumers, 
which `StatefulParDoStreamingTest` already documents (the stateful DoFn counts 
10 attempted elements for 5 inputs), and to a stateful ParDo with additional 
outputs where more than one tag is consumed (#40949).
   
   For a DoFn whose output per tag depends on the order of elements per key, on 
micro batch boundaries or on when timers fire, the copies can disagree, for 
example a dedup that sends the first element per key to the main output and the 
rest to a duplicates tag could put an element in both outputs or in neither. A 
local probe with that dedup DoFn (2 shards, 100 elements on one key, micro 
batches of 20, both tags consumed through one Flatten and as two leaves, 3 runs 
each) gave consistent tags in all 6 runs, since local mode reads 
deterministically. Batch boundaries that depend on time or rate, and shuffle 
order across executors, are not covered by that probe.
   
   A shuffle at the fan out point, as planned in #41198 for stateless 
upstreams, does not help here, the copies get different operator ids and Spark 
does not reuse them. No fix with public Spark API is known yet.
   
   Part of #36841.
   
   ### Issue Priority
   
   Priority: 2 (default / most bugs should be filed as P2)
   
   ### Issue Components
   
   - [x] Component: Spark Runner
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to