JingsongLi commented on code in PR #1301:
URL: https://github.com/apache/incubator-paimon/pull/1301#discussion_r1218785822
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/source/StaticFileStoreSource.java:
##########
@@ -53,11 +68,43 @@ public SplitEnumerator<FileStoreSourceSplit,
PendingSplitsCheckpoint> restoreEnu
PendingSplitsCheckpoint checkpoint) {
Collection<FileStoreSourceSplit> splits =
checkpoint == null ? getSplits() : checkpoint.splits();
- return new StaticFileStoreSplitEnumerator(context, null, splits,
splitBatchSize);
+ SplitAssigner splitAssigner =
+ createSplitAssigner(context, splitBatchSize, splitAssignMode,
splits);
+ return new StaticFileStoreSplitEnumerator(context, null,
splitAssigner);
}
private List<FileStoreSourceSplit> getSplits() {
FileStoreSourceSplitGenerator splitGenerator = new
FileStoreSourceSplitGenerator();
return splitGenerator.createSplits(readBuilder.newScan().plan());
}
+
+ public static SplitAssigner createSplitAssigner(
+ SplitEnumeratorContext<FileStoreSourceSplit> context,
+ int splitBatchSize,
+ SplitAssignMode splitAssignMode,
+ Collection<FileStoreSourceSplit> splits) {
+ switch (splitAssignMode) {
+ case FAIR:
+ return new FairSplitAssigner(
+ splitBatchSize,
+ createSplitAssignment(splits,
context.currentParallelism()));
+ case PREEMPTIVE:
+ return new PreemptiveSplitAssigner(new LinkedList<>(splits));
+ default:
+ throw new UnsupportedOperationException(
+ "Unsupported assign mode " +
splitAssignMode.toString());
+ }
+ }
+
+ private static Map<Integer, LinkedList<FileStoreSourceSplit>>
createSplitAssignment(
Review Comment:
move this method into FairSplitAssigner. It is a part of `FAIR`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@paimon.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
