bvaradar opened a new pull request #722: HUDI-148 Small File selection logic for MOR must skip fileIds selected for pending compaction correctly URL: https://github.com/apache/incubator-hudi/pull/722 The current small file-handling selection could sometimes throw exception similar to com.uber.hoodie.exception.HoodieUpsertException: Failed to upsert for commit time 20190608082556 at com.uber.hoodie.HoodieWriteClient.upsert(HoodieWriteClient.java:175) ...... ...... at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.NoSuchElementException: No value present at java.util.Optional.get(Optional.java:135) at com.uber.hoodie.table.HoodieMergeOnReadTable$MergeOnReadUpsertPartitioner.lambda$getSmallFiles$0(HoodieMergeOnReadTable.java:417) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) at .... java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1359) at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) at com.uber.hoodie.table.HoodieMergeOnReadTable$MergeOnReadUpsertPartitioner.getSmallFiles(HoodieMergeOnReadTable.java:421) at com.uber.hoodie.table.HoodieCopyOnWriteTable$UpsertPartitioner.assignInserts(HoodieCopyOnWriteTable.java:635) at com.uber.hoodie.table.HoodieCopyOnWriteTable$UpsertPartitioner.<init>(HoodieCopyOnWriteTable.java:598) at com.uber.hoodie.table.HoodieMergeOnReadTable$MergeOnReadUpsertPartitioner.<init>(HoodieMergeOnReadTable.java:387) at com.uber.hoodie.table.HoodieMergeOnReadTable.getUpsertPartitioner(HoodieMergeOnReadTable.java:101) at com.uber.hoodie.HoodieWriteClient.getPartitioner(HoodieWriteClient.java:469) at com.uber.hoodie.HoodieWriteClient.upsertRecordsInternal(HoodieWriteClient.java:453) at com.uber.hoodie.HoodieWriteClient.upsert(HoodieWriteClient.java:170) ... 10 more ========= This is because the initial stream of file-slices considered for Small files also includes those in pending compaction.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
