jihoonson commented on a change in pull request #8570: Auto compaction based on
parallel indexing
URL: https://github.com/apache/incubator-druid/pull/8570#discussion_r332990496
##########
File path:
server/src/main/java/org/apache/druid/server/coordinator/helper/DruidCoordinatorSegmentCompactor.java
##########
@@ -151,6 +160,26 @@ public DruidCoordinatorRuntimeParams
run(DruidCoordinatorRuntimeParams params)
.build();
}
+ /**
+ * Each compaction task can run a parallel indexing task. When we count the
number of current running
+ * compaction tasks, we should count the sub tasks of parallel indexing task
as well. However, we currently
+ * don't have a way to get the number of current running sub tasks except
poking each supervisor task,
+ * which is complex to handle all kinds of failures.
+ * Here, instead, we compute a rough number of running sub tasks by summing
maxNumConcurrentSubTasks
+ * in tuningConfig of each supervisor task. The result will be a
conservatively estimated number of sub tasks
+ * which should be ok since it won't affect to the performance of other
ingestion types.
+ */
+ private int findNumMaxConcurrentSubTasks(@Nullable
ClientCompactQueryTuningConfig tuningConfig)
+ {
+ if (tuningConfig != null && tuningConfig.getMaxNumConcurrentSubTasks() !=
null) {
+ // The actual number of subtasks might be smaller than the configured
max.
+ // However, we use the max to simplify the estimation here.
+ return tuningConfig.getMaxNumConcurrentSubTasks();
+ } else {
+ return 0;
+ }
Review comment:
If `maxNumConcurrentSubTasks` is 1, the supervisor task runs in the
sequential mode and processes data by itself instead of spawning sub tasks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
