aokolnychyi commented on code in PR #7714:
URL: https://github.com/apache/iceberg/pull/7714#discussion_r1220364643
##########
core/src/main/java/org/apache/iceberg/util/TableScanUtil.java:
##########
@@ -246,6 +246,34 @@ public static <T extends ScanTask> List<T>
mergeTasks(List<T> tasks) {
return mergedTasks;
}
+ public static long computeSplitSize(
+ long scanSize, int parallelism, long minSplitSize, long maxSplitSize) {
+
+ Preconditions.checkArgument(
+ minSplitSize < maxSplitSize,
+ "Min split size (%s) must be < max split size (%s)",
+ minSplitSize,
+ maxSplitSize);
+
+ // aim for a split per slot by default
+ int splitCount = parallelism;
Review Comment:
It is probably to better target 2x splits per slot by default.
##########
core/src/main/java/org/apache/iceberg/util/TableScanUtil.java:
##########
@@ -246,6 +246,34 @@ public static <T extends ScanTask> List<T>
mergeTasks(List<T> tasks) {
return mergedTasks;
}
+ public static long computeSplitSize(
+ long scanSize, int parallelism, long minSplitSize, long maxSplitSize) {
+
+ Preconditions.checkArgument(
+ minSplitSize < maxSplitSize,
+ "Min split size (%s) must be < max split size (%s)",
+ minSplitSize,
+ maxSplitSize);
+
+ // aim for a split per slot by default
+ int splitCount = parallelism;
Review Comment:
It is probably better target 2x splits per slot by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
