tkaymak opened a new issue, #41254: URL: https://github.com/apache/beam/issues/41254
### What happened? The Spark 4 Structured Streaming runner starts one streaming query per leaf dataset and checkpoints leaf `i` under `<checkpointDir>/<i>` (`StreamingEvaluationContext`). The leaves are collected in a `HashSet` in `PipelineTranslator.TranslatingVisitor`, and `TranslationResult` has no `equals` or `hashCode`, so the iteration order, and with it each leaf's index, follows identity hash codes and can change between JVM runs. After a restart a pipeline with more than one leaf can resume a leaf from another leaf's checkpoint, with the wrong source offsets and state. Spark does not detect it, since every Beam stateful operator has the same operator name and state variables. The fix is to keep the leaves in translation order, for example with a `LinkedHashSet`, so the index of a leaf is stable for an unchanged pipeline. This should land before the first release that announces Spark 4 streaming. Part of #36841. ### Issue Priority Priority: 1 (data loss / total loss of function) ### Issue Components - [x] Component: Spark Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
