vamshipasunuru1 opened a new pull request, #20290: URL: https://github.com/apache/hudi/pull/20290
### Summary Today `hoodie.prewrite.cleaner.policy=ROLLBACK_FAILED_WRITES` rolls back failed writes inline inside `startCommit`, which runs on the Flink JobManager while a write task blocks on `Correspondent.requestInstantTime()`. Marker listing and file deletion therefore sit directly on the instant request path. Tuning timeouts is not reliable as any throttling from file-system can cause checkpoint failures . This adds a policy that keeps the rollback on the JobManager but moves it off the critical path. ### Changelog - HoodiePreWriteCleanerPolicy: new ASYNC_ROLLBACK_FAILED_WRITES value, with `isAsyncRollbackFailedWrites()` and `fromString` support for the lowercase form, matching how the existing values are parsed. - BaseHoodieWriteClient#runPreWriteCleanerPolicy: the new policy is a logged no-op, so starting a commit never blocks on rollback. Engines that do not drive the rollback themselves perform no pre-write rollback under it. - StreamWriteOperatorCoordinator: when the policy is set, builds a dedicated `prewrite-rollback` executor and write client, and submits a rollback at startup and after each completed checkpoint. Fire-and-forget with an in-flight guard, so neither the instant request path nor the checkpoint path ever waits; a skipped round is retried on the next checkpoint. The rollback write client is deliberately separate from the coordinator's: the rollback runs through the table service client, whose TransactionManager tracks a single state-change owner, so driving it from two threads can leave the table lock held. This mirrors what CleanFunction already does. The rollback client honours the configured `hoodie.rollback.parallelism` as-is; nothing is inferred or overridden. The rollback runs inside the JobManager JVM via HoodieFlinkEngineContext, so that value is the knob to tune if the default is too wide for the JobManager. Safety relies on the LAZY failed-writes policy selecting instants by expired heartbeat, so a live writer's instant is never a candidate and running late is harmless. ### Impact Reliable checkpointing without unbounded I/O work in the critical path. ### Risk Level Low. ### Documentation Update To enable this, set hoodie.prewrite.cleaner.policy=ASYNC_ROLLBACK_FAILED_WRITES. The value of this property is NONE by default. ### Contributor's checklist - [ X] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ X] Enough context is provided in the sections above - [ X] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
