wombatu-kun opened a new issue, #20388: URL: https://github.com/apache/hudi/issues/20388
### Bug Description **What happened:** With blocking instant generation (a Flink COPY_ON_WRITE table written with upsert, the default), `StreamWriteOperatorCoordinator` can start the instant of the next checkpoint while the previous instant is still being committed. Starting an instant runs the EAGER failed-writes rollback (`BaseHoodieWriteClient#startCommit` -> `rollbackFailedWrites`), which rolls back every inflight instant, including the one being committed. The guard that should prevent this, `EventBuffers#awaitAllInstantsToCompleteIfNecessary` on the `instant-request` thread, lets the creation through in three ways: 1. It only waits when some buffer already holds a write metadata event (`nonEmpty()`). The events are applied on the `meta-event-handle` thread, so a request handled before the events of the previous instant are applied does not wait at all, although every writer sends those events before it requests the next instant. 2. `CommitGuard#blockFor(String)` awaits the condition once, so the reset of any other buffer releases it while the previous instant is still pending. 3. The check runs outside the guard's lock, so a reset that lands between the check and the await is lost, and the creation stalls for `write.commit.ack.timeout` before failing the job. `ITTestHoodieDataSource#testLookupJoin[5]` (`COPY_ON_WRITE, rocksdb, false`) has failed this way on four unrelated PRs: - https://github.com/apache/hudi/actions/runs/36477043088/job/109114283145 (2026-09-28) - https://github.com/apache/hudi/actions/runs/36977257406/job/110743700678 (2026-10-02) - https://github.com/apache/hudi/actions/runs/37730045047/job/113156896886 (2026-10-08) - https://github.com/apache/hudi/actions/runs/39052223751/job/117452975075 (2026-10-17, on #20366) In each run the commit of instant A waits for the table lock held by the `instant-request` thread and then fails because `A.inflight` is gone. The blocked commit had already written A's files to the metadata table, which keeps serving them: A is no longer pending on the data timeline, and the completed rollback lists it. The test's retry (#19093) then commits a new instant X, A's file slice now counts as before the timeline start, and the next read fails with `FileNotFoundException` on A's deleted base file. A coordinator unit test on master hits a second interleaving: the rollback of A starts while A is inflight, A's commit completes, and the rollback still runs, cleaning A's base files and un-publishing A's completed deltacommit in the metadata table, with no error. From the code (not reproduced): the commit failure fails the job, and after the restart the writers do not resend A's write metadata because A is no longer pending, so A's rows are lost. A related window at job restart is not covered here: bootstrap buffers are created on `meta-event-handle`, so an instant request handled before the first bootstrap event is applied can still roll back the instant being recovered. **What you expected:** In blocking mode, no instant starts until the instants of all earlier checkpoints are committed, whether or not their write metadata events have been applied yet. **Steps to reproduce:** 1. Start a `StreamWriteOperatorCoordinator` for a COPY_ON_WRITE upsert table with synchronous executors, as `TestStreamWriteOperatorCoordinator` does. 2. Request the instant of checkpoint 1. 3. Before any write metadata event of checkpoint 1 reaches the coordinator, request the instant of checkpoint 2. It rolls back the instant of checkpoint 1 (one completed rollback on the timeline) and starts a new instant. ### Environment **Hudi version:** master (`9b7a7139bffb`). The guard has had this shape since #13464 (1.1.0); the releases were not checked end to end. **Query engine:** Flink **Relevant configs:** COPY_ON_WRITE table, upsert operation (blocking instant generation), default failed-writes clean policy (EAGER), metadata table enabled. ### Logs and Stack Trace From job 117452975075: ``` [meta-event-handle] WARN org.apache.hudi.common.util.RetryHelper [] - Task [acquire lock] failed. current retry number 1, will retry after 200 ms. "lockCreateTime" : "2026-10-17 17:28:19.205", "lockThreadName" : "instant-request", [meta-event-handle] WARN org.apache.hudi.common.util.RetryHelper [] - Task [acquire lock] failed. current retry number 4, will retry after 200 ms. "lockCreateTime" : "2026-10-17 17:28:19.659", "lockThreadName" : "instant-request", [meta-event-handle] ERROR org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Executor executes action [commits the instant 20261017172818599] error java.lang.IllegalArgumentException: File /tmp/junit-892835510902829725/t2/.hoodie/timeline/20261017172818599.inflight does not exist! at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42) at org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.transitionStateToComplete(ActiveTimelineV2.java:566) ... at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:324) ... at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:693) [main] WARN org.apache.hudi.table.ITTestHoodieDataSource [] - testLookupJoin collected 0 of 8 rows on attempt 1/3; a teardown race produced an empty lookup join. Retrying. ... Caused by: java.io.FileNotFoundException: File /tmp/junit-892835510902829725/t2/par1/8b54a29b-6f4f-4cc0-88e0-c633e8f1fa86_1-4-0_20261017172818599.parquet does not exist ... at org.apache.hudi.table.format.cow.CopyOnWriteInputFormat.createInputSplits(CopyOnWriteInputFormat.java:188) ``` The second interleaving, from the coordinator unit test on master: ``` [main] INFO org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Create instant [20261018040357423] for table [TestHoodieTable] with type [COPY_ON_WRITE] [pool-6-thread-1] INFO org.apache.hudi.client.BaseHoodieTableServiceClient [] - Begin rollback of instant 20261018040357423 for table /tmp/junit-1862800489165909157 [main] INFO org.apache.hudi.sink.StreamWriteOperatorCoordinator [] - Commit instant [20261018040357423] success! [pool-6-thread-1] INFO org.apache.hudi.table.action.rollback.CopyOnWriteRollbackActionExecutor [] - Clean out all base files generated for commit: [==>20261018040357423__commit__INFLIGHT] [pool-6-thread-1] INFO org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor [] - Rolled back inflight instant 20261018040357423 [pool-6-thread-1] INFO org.apache.hudi.table.action.rollback.MergeOnReadRollbackActionExecutor [] - Rolling back instant [20261018040357423__20261018040359027__deltacommit__COMPLETED] ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
