[
https://issues.apache.org/jira/browse/SPARK-21933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Congxian Qiu updated SPARK-21933:
---------------------------------
Description:
Spark Streaming applications run on Yarn without DynamicAllocation, Yarn enable
HA.
When Spark Streaming applications run on Yarn without DynamicAllocation, If
some nodemanager becomes lost, then the containers on the lost nodemanager will
be reported to all the applicationmasters, application master will allocate new
containers.
But after application master allocate new containers, the lost nodemanager
become available, then, resource manager restarted, after resource manager has
been restarted, the node manager will report the status of all the containers
on it before to resource manager because of Yarn's HA, then application manager
will receive a duplicated completed container message, and request more
executors, so the spark streaming application will request more resource than
it requires.
was:
When Spark Streaming application runs on Yarn without DynamicAllocation, If
some nodemanager becomes lost, then the containers on the lost nodemanager will
be reported to all the applicationmaster, application master will allocate new
containers.
But after application master allocate new containers, the lost nodemanager be
available, after this, resource manager restarted, after resource manager has
been restarted, the node manager will report all the containers running on it
before to resource manager because of Yarn's HA, then application manager will
receive a duplicated completed container message, so the spark streaming
application will request more resource than it requires.
> Spark Streaming request more executors than excepted without DynamicAllocation
> ------------------------------------------------------------------------------
>
> Key: SPARK-21933
> URL: https://issues.apache.org/jira/browse/SPARK-21933
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 2.2.0
> Reporter: Congxian Qiu
>
> Spark Streaming applications run on Yarn without DynamicAllocation, Yarn
> enable HA.
> When Spark Streaming applications run on Yarn without DynamicAllocation, If
> some nodemanager becomes lost, then the containers on the lost nodemanager
> will be reported to all the applicationmasters, application master will
> allocate new containers.
> But after application master allocate new containers, the lost nodemanager
> become available, then, resource manager restarted, after resource manager
> has been restarted, the node manager will report the status of all the
> containers on it before to resource manager because of Yarn's HA, then
> application manager will receive a duplicated completed container message,
> and request more executors, so the spark streaming application will request
> more resource than it requires.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
