wenbingshen commented on PR #10767: URL: https://github.com/apache/hudi/pull/10767#issuecomment-1968124802
> What is exactly the instant time of the bootstrap instant (should be the inflight instant if the metadata file not committed or empty if it is), and why the coordinator rollsback the instant then if the instant has been committed successfully? @danny0405 1. Hudi StreamWriteOperatorCoordinator has a single-threaded thread pool executor to serially execute some checkpoint and operator events; 2. When notifyCheckpointComplete is executed in the executor, the execution is asynchronous. At this time, Flink will consider that the ck has been executed. Assume that the instant submitted by this ck is Instant A. Because reading HDFS is stuck for a long time, we encounter One situation I encountered is that the scheduleCompaction was stuck for 15 minutes, and the ck timeout was 5 minutes, which exceeded the timeout of flink checkpoint. The status at this time was that the StreamWriteOperatorCoordinator completed the submission of the deltacommit file complete, and the ckp_meta file completed the submission complete. , Flink subTask will time out in the instantToWrite method because the new Instant is not initialized. If the task is configured to retry, the Task will restart; 3. After the Task restarts, because the deltacommit file has been completed and the ckp_meta file has been completed, the uncommitted write metadata event will not be sent to the coordinator, but the bootstrap write metadata event will be sent to the coordinator. At this time, all bootstrap events sent by the task are in the StreamWriteOperatorCoordinator Queued in the executor queue, because the StreamWriteOperatorCoordinator is stuck on scheduleCompaction at this time, the thread worker is still running and cannot handle subsequent bootstrap events; 4. When StreamWriteOperatorCoordinator finishes processing scheduleCompaction and creates new Instant B, subTask knows the creation of new Instant B based on ckp_meta coordination. When subTask processElement, a new parquet file will be created to write data (in Insert mode). The executor of StreamWriteOperatorCoordinator then obtains the bootstrap event in the queue for processing and generates a new Instant C. In startInstant function, it will Rollback Instant B because B is now only in the Inflight state; 5. When C is generated and B is rolled back by the Coordinator, the subTask still uses the parquet file corresponding to B to write. When the next flink checkpoint is executed, the data of B will be snapshotted on the timeline of C. 6. As a result, the StreamWriteOperatorCoordinator and the subTask Operator are incoordinated, violating the three-phase submission protocol in Hudi, as I said above, including data snapshot, submit commit file, and submit ckp_meta file -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
