[
https://issues.apache.org/jira/browse/FLINK-10868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17251576#comment-17251576
]
Xintong Song commented on FLINK-10868:
--------------------------------------
[~ZhenqiuHuang],
I think adding counting metrics for container failures is a good idea.
Another question is how do we define a *container failure*?
* In your PR, a container failure is recorded in
{{ActiveResourceManager#requestNewWorker}} when the {{requestResourceFuture}}
completes exceptionally.
* There could be cases that the task manager process is launched and failed
immediately during initialization. In such cases, the {{requestResourceFuture}}
will complete successfully, but the {{onWorkerTerminated}} callback will be
called before the worker is registered.
I would suggest to record the failure for all containers that are being
requested but failed/terminated before registering to RM. WDYT?
Apart from this question, I think we are on the same page.
> Flink's JobCluster ResourceManager doesn't use maximum-failed-containers as
> limit of resource acquirement
> ---------------------------------------------------------------------------------------------------------
>
> Key: FLINK-10868
> URL: https://issues.apache.org/jira/browse/FLINK-10868
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Mesos, Deployment / YARN
> Affects Versions: 1.6.2, 1.7.0
> Reporter: Zhenqiu Huang
> Assignee: Zhenqiu Huang
> Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently, YarnResourceManager does use yarn.maximum-failed-containers as
> limit of resource acquirement. In worse case, when new start containers
> consistently fail, YarnResourceManager will goes into an infinite resource
> acquirement process without failing the job. Together with the
> https://issues.apache.org/jira/browse/FLINK-10848, It will quick occupy all
> resources of yarn queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
