gianm commented on code in PR #13955:
URL: https://github.com/apache/druid/pull/13955#discussion_r1149889000
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/input/SlicerUtils.java:
##########
@@ -98,6 +98,66 @@ public static <T> List<List<T>> makeSlices(
return slicesList;
}
+ /**
+ * Creates up to "maxNumSlices" lists from "iterator".
+ *
+ * This method creates as few slices as possible, while keeping each slice
under the provided limits.
+ *
+ * If there is a conflict between maxNumSlices and maxFilesPerSlice or
maxWeightPerSlice, then maxNumSlices wins.
+ * This means that for small values of maxNumSlices, slices may have more
than maxFilesPerSlice files, or more
+ * than maxWeightPerSlice weight.
+ */
+ public static <T> List<List<T>> makeSlicesDynamic(
+ final Iterator<T> iterator,
+ final ToLongFunction<T> weightFunction,
+ final int maxNumSlices,
+ final int maxFilesPerSlice,
+ final long maxWeightPerSlice
+ )
+ {
Review Comment:
I'll add these comments to hopefully clarify things:
```
* This function uses a greedy algorithm that starts out with a single
empty slice. Items are assigned to the slice
* that has the lowest weight at the time that item is encountered. New
slices are created, up to maxNumSlices, when
* the lightest existing slice exceeds either maxFilesPerSlice or
maxWeightPerSlice.
*
* Slices may have more than maxFilesPerSlice files, or more than
maxWeightPerSlice weight, if necessary to
* keep the total number of slices at or below maxNumSlices.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
