Neerja Khattar created SPARK-18769:
--------------------------------------
Summary: Spark to be smarter about what the upper bound is and to
restrict number of executor when dynamic allocation is enabled
Key: SPARK-18769
URL: https://issues.apache.org/jira/browse/SPARK-18769
Project: Spark
Issue Type: New Feature
Reporter: Neerja Khattar
Currently when dynamic allocation is enabled max.executor is infinite and spark
creates so many executor and even exceed the yarn nodemanager memory limit and
vcores.
It should have a check to not exceed more that yarn resource limit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
