rmatharu commented on a change in pull request #1104: SAMZA-2266: Introduce a
backoff when there are repeated failures for host-affinity allocations
URL: https://github.com/apache/samza/pull/1104#discussion_r309816504
##########
File path:
samza-core/src/main/java/org/apache/samza/clustermanager/ResourceRequestState.java
##########
@@ -213,7 +226,7 @@ public void
updateStateAfterAssignment(SamzaResourceRequest request, String assi
public int releaseExtraResources() {
synchronized (lock) {
int numReleasedResources = 0;
- if (requestsQueue.isEmpty()) {
+ if (requestsQueue.isEmpty() && delayedRequestsQueue.isEmpty()) {
Review comment:
We don't need to check the delayedRequestsQueue here, because a
resource-request for anything in the delayedRequestsQueue will be sent out only
when the requests are no-longer delayed.
The implication of this is that if requestsQueue is empty and
delayedRequestsQueue is not, the CPM will continue to hold onto allocated
resources, in the worst case for a period of 5 mins (default).
Aggregated over num_containers, could be significant especially in
resource-crunch scenarios.
Alternatively, we could release the resource only looking at the
requestsQueue, and when a request in the delayedRequestsQueue "expires", we
will send out the request to the YARN-RM and allocation flow shall resume.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
