openinx commented on a change in pull request #1704:
URL: https://github.com/apache/iceberg/pull/1704#discussion_r522811571
##########
File path:
core/src/main/java/org/apache/iceberg/actions/BaseRewriteDataFilesAction.java
##########
@@ -226,11 +226,16 @@ public RewriteDataFilesActionResult execute() {
return TableScanUtil.planTasks(splitTasks, targetSizeInBytes,
splitLookback, splitOpenFileCost);
})
.flatMap(Streams::stream)
+ .filter(task -> task.files().size() > 1)
.collect(Collectors.toList());
+ if (combinedScanTasks.isEmpty()) {
+ return RewriteDataFilesActionResult.empty();
+ }
+
List<DataFile> addedDataFiles = rewriteDataForTasks(combinedScanTasks);
- List<DataFile> currentDataFiles = filteredGroupedTasks.values().stream()
- .flatMap(tasks -> tasks.stream().map(FileScanTask::file))
+ List<DataFile> currentDataFiles = combinedScanTasks.stream()
Review comment:
It's right that we may don't have to rewrite those files whose size
greater than target file size. But in this patch, I don't understand why we
need to change it , could you revert this pls ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
