openinx commented on a change in pull request #1704:
URL: https://github.com/apache/iceberg/pull/1704#discussion_r522591211
##########
File path:
core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java
##########
@@ -226,11 +226,16 @@ public RewriteDataFilesActionResult execute() {
return TableScanUtil.planTasks(splitTasks, targetSizeInBytes,
splitLookback, splitOpenFileCost);
})
.flatMap(Streams::stream)
+ .filter(task -> task.files().size() > 1)
.collect(Collectors.toList());
+ if (combinedScanTasks.isEmpty()) {
+ return RewriteDataFilesActionResult.empty();
+ }
+
List<DataFile> addedDataFiles = rewriteDataForTasks(combinedScanTasks);
- List<DataFile> currentDataFiles = filteredGroupedTasks.values().stream()
- .flatMap(tasks -> tasks.stream().map(FileScanTask::file))
+ List<DataFile> currentDataFiles = combinedScanTasks.stream()
Review comment:
Is this change correct ? If we have two `file1` (200MB), `file2`
(1MB), and the target file size is 128MB, then it will be planned into three
combined tasks: 128MB, 72MB, 1MB. Finally the `file1` will occur twice in the
currentDataFiles ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
