tkaymak opened a new pull request, #41256: URL: https://github.com/apache/beam/pull/41256
Fixes #41254, part of #36841. The Spark 4 streaming runner starts one query per leaf and checkpoints leaf `i` under `<checkpointDir>/<i>`. The leaves were collected in a `HashSet` of `TranslationResult`, which has no `equals` or `hashCode`, so their order followed identity hash codes. For the same 8 leaf pipeline three JVM runs gave two different orders, so after a restart a leaf could resume from another leaf's checkpoint without an error. The leaves now keep translation order in a `LinkedHashSet`. `PipelineTranslatorTest.leavesFollowTranslationOrder` translates 8 leaves and asserts the order. It failed before the change and passes on Spark 3 and 4. Locally gates, the streaming and both batch Structured Streaming ValidatesRunner suites and the full `:runners:spark:4:test` and `:runners:spark:3:test` pass. The trigger files run the Spark 4 and SparkStructuredStreaming ValidatesRunner PostCommits here, the batch suites also evaluate leaves in this order now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
