[
https://issues.apache.org/jira/browse/SPARK-12447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin resolved SPARK-12447.
------------------------------------
Resolution: Fixed
Assignee: Saisai Shao (was: Apache Spark)
Fix Version/s: 2.0.0
> Only update AM's internal state when executor is successfully launched by NM
> ----------------------------------------------------------------------------
>
> Key: SPARK-12447
> URL: https://issues.apache.org/jira/browse/SPARK-12447
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.6.0
> Reporter: Saisai Shao
> Assignee: Saisai Shao
> Fix For: 2.0.0
>
>
> Currently {{YarnAllocator}} will update its managed states like
> {{numExecutorsRunning}} after container is allocated but before executor are
> successfully launched.
> This happened when Spark configuration is wrong (like spark_shuffle
> aux-service is not configured in NM occasionally), which makes executor fail
> to launch, or NM lost when NMClient is communicated.
> In the current implementation, state will also be updated even executor is
> failed to launch, this will lead to incorrect state of AM. Also lingering
> container will only be release after timeout, this will introduce resource
> waste.
> So here we should update the states only after executor is correctly
> launched, otherwise we should release container ASAP to make it fail fast and
> retry.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
