GitHub user sryza opened a pull request:
https://github.com/apache/spark/pull/4168
SPARK-4136. Under dynamic allocation, cancel outstanding executor requests
when no longer needed [WIP]
This takes advantage of the changes made in SPARK-4337 to cancel pending
requests to YARN when they are no longer needed.
Each time the timer in `ExecutorAllocationManager` strikes, we compute
`maxNumNeededExecutors`, the maximum number of executors we could fill with the
current load. This is calculated as the total number of running and pending
tasks divided by the number of cores per executor. If `maxNumNeededExecutors`
is below the total number of running and pending executors, we call
`requestTotalExecutors(maxNumNeededExecutors)` to let the cluster manager know
that it should cancel any pending requests above this amount. If not,
`maxNumNeededExecutors` is just used as a bound in alongside the configured
`maxExecutors` to limit the number of new requests.
The patch modifies the API exposed by `ExecutorAllocationClient` for
requesting additional executors by moving from `requestExecutors` to
`requestTotalExecutors`. This makes the communication between the
`ExecutorAllocationManager` and the `YarnAllocator` easier to reason about and
removes some state that needed to be kept in the
`CoarseGrainedSchedulerBackend`. I think an argument can be made that this
makes for a less attractive user-facing API in `SparkContext`, but I'm having
trouble envisioning situations where a user would want to use either of these
APIs.
This will likely break some tests, but I wanted to get feedback on the
approach before adding tests and polishing.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sryza/spark sandy-spark-4136
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4168.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4168
----
commit 3ddb88526c06be83a1daf3eb73c7bfcc67c717de
Author: Sandy Ryza <[email protected]>
Date: 2015-01-23T01:15:03Z
SPARK-4136
commit 39c4de55de1fe8df6eb70db06bd9e2052324938d
Author: Sandy Ryza <[email protected]>
Date: 2015-01-23T01:22:28Z
Make ExecutorAllocationClient request a total
commit 79763d98431e8764e29ef029992b1cd506eb8802
Author: Sandy Ryza <[email protected]>
Date: 2015-01-23T02:05:22Z
Make things right
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
