[
https://issues.apache.org/jira/browse/SPARK-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-10702:
---------------------------------
Labels: bulk-closed (was: )
> Dynamic Allocation in Standalone Breaking Parallelism
> -----------------------------------------------------
>
> Key: SPARK-10702
> URL: https://issues.apache.org/jira/browse/SPARK-10702
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
> Affects Versions: 1.5.0
> Environment: CentOS 7. Standalone
> Reporter: Mark Khaitman
> Priority: Major
> Labels: bulk-closed
>
> It seems that although executors are properly dropped after they've reached
> their configured idle timeout setting, even if all cores in the cluster are
> still available, it is not regaining the full amount back for subsequent
> spark jobs within that same context.
> For example:
> - A stage has 40 partitions to process and completes successfully. After X
> seconds, the executors are all expectedly dropped.
> - Then, another stage is set to begin, and plenty of cores and memory are
> still available on the nodes within the cluster, however, rather than
> obtaining 40 cores, only 13 got obtained and only 13 active tasks were
> running.
> - Another concern was that it put all 13 active tasks onto a single node
> rather than trying to create the usual amount of executors across the cluster
> (possibly related to this??)
> Not sure of the exact cause of this, though I do know dynamic allocation to
> the standalone environment is still new so I kind of half-expected some
> scheduling concerns to possibly come up! Wondering if anyone else has seen
> this behaviour.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
