Mark Khaitman created SPARK-10702:
-------------------------------------
Summary: Dynamic Allocation in Standalone Breaking Parallelism
Key: SPARK-10702
URL: https://issues.apache.org/jira/browse/SPARK-10702
Project: Spark
Issue Type: Bug
Components: Scheduler
Affects Versions: 1.5.0
Environment: CentOS 7. Standalone
Reporter: Mark Khaitman
It seems that although executors are properly dropped after they've reached
their configured idle timeout setting, even if all cores in the cluster are
still available, it is not regaining the full amount back for subsequent spark
jobs within that same context.
For example:
- A stage has 40 partitions to process and completes successfully. After X
seconds, the executors are all expectedly dropped.
- Then, another stage is set to begin, and plenty of cores and memory are still
available on the nodes within the cluster, however, rather than obtaining 40
cores, only 13 got obtained and only 13 active tasks were running.
- Another concern was that it put all 13 active tasks onto a single node rather
than trying to create the usual amount of executors across the cluster
(possibly related to this??)
Not sure of the exact cause of this, though I do know dynamic allocation to the
standalone environment is still new so I kind of half-expected some scheduling
concerns to possibly come up! Wondering if anyone else has seen this behaviour.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
