CaesarWangX opened a new issue, #11202: URL: https://github.com/apache/hudi/issues/11202
**_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at [email protected]. - If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly. **Describe the problem you faced** After upgrading from Hudi 0.11 to Hudi 0.14, we occasionally encounter this error. Once it occurs, the task continues to fail even after restarting, and the error message remains the same with the FileID. I have reviewed the following two issues, but unfortunately, there is no effective solution available. https://github.com/apache/hudi/issues/5886 https://github.com/apache/hudi/issues/8890 Fortunately, by examining the source code, I have successfully replicated this issue and attempted to modify the source code to prevent task failures. Any information or suggestions are helpful, thank you. **To Reproduce** Steps to reproduce the behavior: 1. Filegroup has only one data file. 2. Delete the deltacommit corresponding to this filegroup (only delete deltacommit, keep request and inflight). 3. New data will continue to be written to this filegroup (determined by getSmallFiles). BWT, Under normal circumstances, the task fails due to this reason when these three conditions are coincidentally met at the same time. How challenging it is indeed! Therefore, reproducing this error is quite difficult. Essentially, it occurs when the job fails after completing data writing to the filegroup but before committing the deltacommit.😅 **Expected behavior** A clear and concise description of what you expected to happen. **Environment Description** * Hudi version : 0.14.0, MOR, INSERT * Spark version : 3.4.1 * Storage (HDFS/S3/GCS..) : * Running on Docker? (yes/no) : **Additional context** **Not finished yet, will provide more information later.** After examining the code, the key difference between Hudi 0.11 and 0.14 is as follows: **AbstractTableFileSystemView** Hudi0.11: <img width="1186" alt="image" src="https://github.com/apache/hudi/assets/12985552/78f519f6-b77f-4798-ac03-69647a374a7b";> <img width="1227" alt="image" src="https://github.com/apache/hudi/assets/12985552/adf624b0-1901-4cba-b61d-baed3725f590";> Hudi0.14: **Stacktrace** Caused by: java.util.NoSuchElementException: FileID xxxxx of partition path dt=2019-02-20 does not exist. at org.apache.hudi.io.HoodieMergeHandle.getLatestBaseFile(HoodieMergeHandle.java:159) at org.apache.hudi.io.HoodieMergeHandle.<init>(HoodieMergeHandle.java:121) at org.apache.hudi.io.FlinkMergeHandle.<init>(FlinkMergeHandle.java:70) at org.apache.hudi.io.FlinkConcatHandle.<init>(FlinkConcatHandle.java:53) at org.apache.hudi.client.HoodieFlinkWriteClient.getOrCreateWriteHandle(HoodieFlinkWriteClient.java:557) at org.apache.hudi.client.HoodieFlinkWriteClient.insert(HoodieFlinkWriteClient.java:175) at org.apache.hudi.sink.StreamWriteFunction.lambda$initWriteFunction$0(StreamWriteFunction.java:181) at org.apache.hudi.sink.StreamWriteFunction.lambda$flushRemaining$7(StreamWriteFunction.java:461) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
