maytasm commented on a change in pull request #12228:
URL: https://github.com/apache/druid/pull/12228#discussion_r800158757
##########
File path:
server/src/main/java/org/apache/druid/client/indexing/HttpIndexingServiceClient.java
##########
@@ -194,26 +194,39 @@ public String cancelTask(String taskId)
@Override
public int getTotalWorkerCapacity()
+ {
+ return getWorkers().stream().mapToInt(workerInfo ->
workerInfo.getWorker().getCapacity()).sum();
+ }
+
+ @Override
+ public int getTotalWorkerCapacityWithAutoScale()
{
try {
final StringFullResponseHolder response = druidLeaderClient.go(
- druidLeaderClient.makeRequest(HttpMethod.GET,
"/druid/indexer/v1/workers")
+ druidLeaderClient.makeRequest(HttpMethod.GET,
"/druid/indexer/v1/autoScaleConfig")
.setHeader("Content-Type",
MediaType.APPLICATION_JSON)
);
-
if (!response.getStatus().equals(HttpResponseStatus.OK)) {
throw new ISE(
- "Error while getting available cluster capacity. status[%s]
content[%s]",
+ "Error while getting worker info. status[%s] content[%s]",
response.getStatus(),
response.getContent()
);
}
- final Collection<IndexingWorkerInfo> workers = jsonMapper.readValue(
+ final IndexingAutoScaleConfigInfo indexingAutoScaleConfigInfo =
jsonMapper.readValue(
response.getContent(),
- new TypeReference<Collection<IndexingWorkerInfo>>() {}
+ new TypeReference<IndexingAutoScaleConfigInfo>() {}
);
-
- return workers.stream().mapToInt(workerInfo ->
workerInfo.getWorker().getCapacity()).sum();
+ Collection<IndexingWorkerInfo> workers = getWorkers();
+ int capacityPerWorker;
+ if (workers != null && !workers.isEmpty()) {
+ capacityPerWorker =
workers.stream().findFirst().get().getWorker().getCapacity();
Review comment:
Refactored this. Moved all the logic for calculating total worker
capacity (with and without auto scaling) to a API in the overlord. For
determining the total worker capacity with auto scaling, this new API basically
delegate the calculation to the configured provisioning strategy. The
provisioning strategy already has a method to calculate the per worker
capacity, hence, we just reuse that. Note 1: that this currently only support
PendingTaskBasedWorkerProvisioningStrategy and any other provisioning strategy
will result in -1 for `maximumCapacityWithAutoScale`. Note 2:
PendingTaskBasedWorkerProvisioningStrategy assumes all nodes have the same
capacity. This is something that already existed and is how
PendingTaskBasedWorkerProvisioningStrategy determine how many new indexer/MM to
create for auto scaling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
