cryptoe commented on code in PR #16889:
URL: https://github.com/apache/druid/pull/16889#discussion_r1772905471
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/ImmutableWorkerInfo.java:
##########
@@ -225,6 +253,89 @@ private int getWorkerParallelIndexCapacity(double
parallelIndexTaskSlotRatio)
return workerParallelIndexCapacity;
}
+ public boolean canRunTask(Task task, Map<String, Number> taskLimits)
Review Comment:
Can you please add java docs for this method.
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/ImmutableWorkerInfo.java:
##########
@@ -57,6 +63,7 @@ public ImmutableWorkerInfo(
@JsonProperty("worker") Worker worker,
@JsonProperty("currCapacityUsed") int currCapacityUsed,
@JsonProperty("currParallelIndexCapacityUsed") int
currParallelIndexCapacityUsed,
+ @JsonProperty("currTypeSpecificCapacityUsed") Map<String, Integer>
typeSpecificCapacityMap,
Review Comment:
This should be nullable no ?
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/ZkWorker.java:
##########
@@ -167,6 +167,29 @@ private int getCurrParallelIndexCapacityUsed(Map<String,
TaskAnnouncement> tasks
return currParallelIndexCapacityUsed;
}
+ @JsonProperty("currTypeSpecificCapacityUsed")
+ public Map<String, Integer> getCurrTypeSpecificCapacityUsed()
Review Comment:
I thought we had deprecated Zk based runner in favour of http.
@kfaraz Does this change still make sense ?
##########
docs/configuration/index.md:
##########
@@ -1135,6 +1135,7 @@ The following configs only apply if the Overlord is
running in remote mode. For
|`druid.indexer.runner.taskAssignmentTimeout`|How long to wait after a task
has been assigned to a Middle Manager before throwing an error.|`PT5M`|
|`druid.indexer.runner.minWorkerVersion`|The minimum Middle Manager version to
send tasks to. The version number is a string. This affects the expected
behavior during certain operations like comparison against
`druid.worker.version`. Specifically, the version comparison follows dictionary
order. Use ISO8601 date format for the version to accommodate date comparisons.
|"0"|
| `druid.indexer.runner.parallelIndexTaskSlotRatio`| The ratio of task slots
available for parallel indexing supervisor tasks per worker. The specified
value must be in the range `[0, 1]`. |1|
+|`druid.indexer.runner.taskSlotLimits`| A map where each key is a task type,
and the corresponding value represents the limit on the number of task slots
that a task of that type can occupy on a worker. The key is a `String` that
specifies the task type. The value can either be a Double or Integer. A
`Double` in the range [0, 1], representing a ratio of the available task slots
that tasks of this type can occupy. An `Integer` that is greater than or equal
to 0, representing an absolute limit on the number of task slots that tasks of
this type can occupy.|Empty map|
Review Comment:
How does this interact with compaction slots ?
##########
docs/configuration/index.md:
##########
@@ -1135,6 +1135,7 @@ The following configs only apply if the Overlord is
running in remote mode. For
|`druid.indexer.runner.taskAssignmentTimeout`|How long to wait after a task
has been assigned to a Middle Manager before throwing an error.|`PT5M`|
|`druid.indexer.runner.minWorkerVersion`|The minimum Middle Manager version to
send tasks to. The version number is a string. This affects the expected
behavior during certain operations like comparison against
`druid.worker.version`. Specifically, the version comparison follows dictionary
order. Use ISO8601 date format for the version to accommodate date comparisons.
|"0"|
| `druid.indexer.runner.parallelIndexTaskSlotRatio`| The ratio of task slots
available for parallel indexing supervisor tasks per worker. The specified
value must be in the range `[0, 1]`. |1|
+|`druid.indexer.runner.taskSlotLimits`| A map where each key is a task type,
and the corresponding value represents the limit on the number of task slots
that a task of that type can occupy on a worker. The key is a `String` that
specifies the task type. The value can either be a Double or Integer. A
`Double` in the range [0, 1], representing a ratio of the available task slots
that tasks of this type can occupy. An `Integer` that is greater than or equal
to 0, representing an absolute limit on the number of task slots that tasks of
this type can occupy.|Empty map|
Review Comment:
Could you please provide an example as well ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
