[
https://issues.apache.org/jira/browse/FLINK-5856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894407#comment-15894407
]
ASF GitHub Bot commented on FLINK-5856:
---------------------------------------
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/3398#discussion_r104153900
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java ---
@@ -239,10 +245,26 @@ public void
onContainersCompleted(List<ContainerStatus> list) {
@Override
public void onContainersAllocated(List<Container> containers) {
for (Container container : containers) {
- numPendingContainerRequests = Math.max(0,
numPendingContainerRequests - 1);
- LOG.info("Received new container: {} - Remaining
pending container requests: {}",
- container.getId(),
numPendingContainerRequests);
+ Priority priority = container.getPriority();
+ // Yarn api say that it may give containers more than
asked, so release the redundant ones
+ if
(!numPendingContainerRequests.containsKey(priority.getPriority()) ||
+
numPendingContainerRequests.get(priority.getPriority()) <= 0) {
+ LOG.debug("Received more than asked containers,
will release the {}, priority {}",
+ container.getId(),
priority.getPriority());
+
resourceManagerClient.releaseAssignedContainer(container.getId());
+ continue;
+ }
+ int orgNum =
numPendingContainerRequests.get(priority.getPriority());
+ numPendingContainerRequests.put(priority.getPriority(),
orgNum - 1);
+ LOG.info("Received new container: {} for priority: {} -
Remaining pending container requests: {}",
+ container.getId(),
priority.getPriority(), orgNum - 1);
+
try {
+ // Yarn will not clear the request automaticly,
+ // And the resource allocated may not equal to
the resource requested
+ resourceManagerClient.removeContainerRequest(
--- End diff --
Why is that the case? How did it work before if Yarn does not clear the
container requests? Please elaborate a little bit how it worked before and why
it had to be changed.
> Need return redundant containers to yarn for yarn mode
> ------------------------------------------------------
>
> Key: FLINK-5856
> URL: https://issues.apache.org/jira/browse/FLINK-5856
> Project: Flink
> Issue Type: Bug
> Components: YARN
> Reporter: shuai.xu
> Assignee: shuai.xu
> Labels: flip-6
>
> In flip6, for flink on yarn mode, RM requests container from yarn according
> to the requirement of the JM. But the AMRMClientAsync used in yarn doesn't
> guarantee that the number of containers returned exactly equal to the number
> requested. So it need to record the number request by flink rm and return the
> redundant ones to yarn.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
