[
https://issues.apache.org/jira/browse/SPARK-12447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066316#comment-15066316
]
Apache Spark commented on SPARK-12447:
--------------------------------------
User 'jerryshao' has created a pull request for this issue:
https://github.com/apache/spark/pull/10412
> Only update AM's internal state when executor is successfully launched by NM
> ----------------------------------------------------------------------------
>
> Key: SPARK-12447
> URL: https://issues.apache.org/jira/browse/SPARK-12447
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.6.0
> Reporter: Saisai Shao
> Assignee: Apache Spark
>
> Currently {{YarnAllocator}} will update its managed states like
> {{numExecutorsRunning}} after container is allocated but before executor are
> successfully launched.
> This happened when Spark configuration is wrong, which makes executor fail to
> launch, or NM lost when NMClient is communicated.
> In the current implementation, state will also be updated even executor is
> failed to launch, this will lead to incorrect state of AM. Also lingering
> container will only be release after timeout, this will introduce resource
> waste.
> So here we should update the states only after executor is correctly
> launched, otherwise we should release container ASAP to make it fail fast and
> retry.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
