RussellSpitzer commented on a change in pull request #3292:
URL: https://github.com/apache/iceberg/pull/3292#discussion_r746110292
##########
File path: core/src/main/java/org/apache/iceberg/util/TableScanUtil.java
##########
@@ -51,6 +51,31 @@ public static boolean hasDeletes(FileScanTask task) {
return CloseableIterable.combine(splitTasks, tasks);
}
+ /**
+ * Split files into FileScanTasks which only contain a single offset
(rowGroup). For files which do not
+ * expose the offsets, use the normal split code.
+ * @param tasks Scan tasks, one per whole file to be split
+ * @param fallbackSplitSize the splitSize to use when the file does not
contain explicit offsets to use
+ * @return Scan tasks, one per offset
+ */
+ public static CloseableIterable<FileScanTask>
splitOnOffsets(CloseableIterable<FileScanTask> tasks,
Review comment:
@rdblue Changed implementation here, didn't recombine splits for the
same file yet, but wanted to make sure you are ok with this. We could
theoretically change this for all splitFiles but I didn't want to change that
api yet ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
