AmatyaAvadhanula commented on a change in pull request #12221:
URL: https://github.com/apache/druid/pull/12221#discussion_r799903838
##########
File path:
indexing-service/src/main/java/org/apache/druid/indexing/overlord/ImmutableWorkerInfo.java
##########
@@ -119,12 +142,33 @@ public boolean isValidVersion(String minVersion)
return worker.getVersion().compareTo(minVersion) >= 0;
}
- public boolean canRunTask(Task task)
+ public boolean canRunTask(Task task, double parallelIndexWorkerRatio)
{
return (worker.getCapacity() - getCurrCapacityUsed() >=
task.getTaskResource().getRequiredCapacity()
+ && canRunParallelIndexTask(task, parallelIndexWorkerRatio)
&&
!getAvailabilityGroups().contains(task.getTaskResource().getAvailabilityGroup()));
}
+ public boolean canRunParallelIndexTask(Task task, double
parallelIndexWorkerRatio)
+ {
+ if (!task.getType().equals(ParallelIndexSupervisorTask.TYPE)) {
+ return true;
+ }
+ return getWorkerParallelIndexCapacity(parallelIndexWorkerRatio) -
getCurrParallelIndexCapacityUsed()
Review comment:
Yes, that could be an issue with the default of 0.1
We could set the default value to 1, in which case the behaviour would be
the same as it is today.
A default value of 0.5 might be the best? Deadlocks won't occur and subtask
concurrency is not affected except in the cases where a lot of supervising
tasks are running
The value can be configured to be lower when supervising tasks are
configured to have a lot of subtasks and "fast" progress of a few ingestion
tasks is preferred over "slow" progress of all the submitted ones
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
