tkaymak commented on PR #39576: URL: https://github.com/apache/beam/pull/39576#issuecomment-5356251845
Update for reviewers, two rounds of changes since the review request. First, the four remarks from my own review pass are addressed: the Kryo registrations for Spark's streaming internals moved to the end of the registrator so auto assigned ids stay identical across Spark 3 and 4, the fallback temporary checkpoint directory is cleaned up after evaluation, the throwing checks in the stateful streaming translator now live in a named rejectUnsupported method with the contract documented, and awaitTermination polls the leaf queries round robin so a failing query surfaces immediately and stops its siblings. Second, the biggest known gap is closed: checkpoint recovery is now durable. The source id is derived deterministically from the read transform's full name, the first run pins its split list under the checkpoint location (Beam sources do not guarantee deterministic splitting), each partition reader persists its checkpoint mark per batch epoch atomically through the Hadoop filesystem, and a restarted run resumes from the newest mark at or before its batch start epoch. Semantics are at least once, a mark is written when a batch finishes reading rather than transactionally with Spark's commit, that caveat is documented on BeamReaderCache. A new end to end test restarts a pipeline against the same checkpoint location with a cleared reader cache and proves it resumes instead of re reading. Full suite locally: 234 tests, 0 failures, 5 pre existing skips. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
