xintongsong commented on a change in pull request #11353: [FLINK-16438][yarn]
Make YarnResourceManager starts workers using WorkerResourceSpec requested by
SlotManager
URL: https://github.com/apache/flink/pull/11353#discussion_r404489605
##########
File path:
flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java
##########
@@ -412,30 +439,32 @@ private void
releaseFailedContainerAndRequestNewContainerIfRequired(ContainerId
final ResourceID resourceId = new
ResourceID(containerId.toString());
// release the failed container
- workerNodeMap.remove(resourceId);
+ YarnWorkerNode yarnWorkerNode =
workerNodeMap.remove(resourceId);
resourceManagerClient.releaseAssignedContainer(containerId);
// and ask for a new one
- requestYarnContainerIfRequired();
+
requestYarnContainerIfRequired(yarnWorkerNode.getContainer().getResource());
}
private void returnExcessContainer(Container excessContainer) {
log.info("Returning excess container {}.",
excessContainer.getId());
resourceManagerClient.releaseAssignedContainer(excessContainer.getId());
}
- private void removeContainerRequest(AMRMClient.ContainerRequest
pendingContainerRequest) {
- numPendingContainerRequests--;
-
- log.info("Removing container request {}. Pending container
requests {}.", pendingContainerRequest, numPendingContainerRequests);
-
+ private void removeContainerRequest(AMRMClient.ContainerRequest
pendingContainerRequest, WorkerResourceSpec workerResourceSpec) {
+ log.info("Removing container request {}.",
pendingContainerRequest);
+ pendingWorkerCounter.decreaseAndGet(workerResourceSpec);
Review comment:
True, a periodic cleanup task should work there, or we can trigger a
clean-up if the number of entries stored in
`WorkerSpecContainerResourceAdapter` reaches a certain threshold.
I think the clean-up is definitely doable. It's just not very necessary at
the time being.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
