GitHub user cshuo added a comment to the discussion: RocksDB as The Replica of MDT/RLI
> This is still best effort right.. Is there no way to make hudi commit first, > and then flink proceeds to checkpoint. like a pre-checkpoint hook? in flink? > is this a flink limitation? @vinothchandar Yeah, that's a limitation or transactional sink contract for flink. A Flink checkpoint is completed only after all operators asynchronously finish and acknowledge their snapshots. Even if the sink successfully commits to Hudi, a failure in any other operator can still abort the overall checkpoint. Moreover, an aborted checkpoint does not necessarily fail or restart the job—Flink may simply continue and let a later checkpoint cover a longer processing interval. This is why Flink’s transactional sink model uses checkpoint-coordinated two-phase commit: prepare during the checkpoint and make the external commit visible only after the global checkpoint completes. Flink writers for Iceberg, Delta etc, follow the same pattern. GitHub link: https://github.com/apache/hudi/discussions/18296#discussioncomment-18261200 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
