tkaymak opened a new issue, #41425: URL: https://github.com/apache/beam/issues/41425
### What happened? In streaming mode the Spark 4 Structured Streaming runner does not persist datasets (`persist` on a streaming Dataset throws), so a stateful ParDo whose output is consumed more than once runs one `transformWithState` per consumer, each with its own state store. This applies to one output with two consumers, which `StatefulParDoStreamingTest` already documents (the stateful DoFn counts 10 attempted elements for 5 inputs), and to a stateful ParDo with additional outputs where more than one tag is consumed (#40949). For a DoFn whose output per tag depends on the order of elements per key, on micro batch boundaries or on when timers fire, the copies can disagree, for example a dedup that sends the first element per key to the main output and the rest to a duplicates tag could put an element in both outputs or in neither. A local probe with that dedup DoFn (2 shards, 100 elements on one key, micro batches of 20, both tags consumed through one Flatten and as two leaves, 3 runs each) gave consistent tags in all 6 runs, since local mode reads deterministically. Batch boundaries that depend on time or rate, and shuffle order across executors, are not covered by that probe. A shuffle at the fan out point, as planned in #41198 for stateless upstreams, does not help here, the copies get different operator ids and Spark does not reuse them. No fix with public Spark API is known yet. Part of #36841. ### Issue Priority Priority: 2 (default / most bugs should be filed as P2) ### Issue Components - [x] Component: Spark Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
