GitHub user jiangxb1987 opened a pull request:
https://github.com/apache/spark/pull/22001
[SPARK-24819][CORE] Fail fast when no enough slots to launch the barrier
stage on job submitted
## What changes were proposed in this pull request?
We shall check whether the barrier stage requires more slots (to be able to
launch all tasks in the barrier stage together) than the total number of active
slots currently, and fail fast if trying to submit a barrier stage that
requires more slots than current total number.
This PR proposes to add a new method `getNumSlots()` to try to get the
total number of currently active slots in `SchedulerBackend`, support of this
new method has been added to all the first-class scheduler backends except
`MesosFineGrainedSchedulerBackend`.
## How was this patch tested?
Added new test cases in `BarrierStageOnSubmittedSuite`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jiangxb1987/spark SPARK-24819
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22001.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22001
----
commit 52530052c896862748a86a1b77455f31534b6045
Author: Xingbo Jiang <xingbo.jiang@...>
Date: 2018-08-05T15:47:05Z
Fail fast when no enough slots to launch the barrier stage on job submitted
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
