Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22001#discussion_r207834660
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -496,6 +496,12 @@ class CoarseGrainedSchedulerBackend(scheduler:
TaskSchedulerImpl, val rpcEnv: Rp
executorDataMap.keySet.toSeq
}
+ override def getNumSlots(): Int = {
+ executorDataMap.values.foldLeft(0) { (num, executor) =>
+ num + executor.totalCores / scheduler.CPUS_PER_TASK
--- End diff --
As mentioned in the method description of `SchedulerBackend.getNumSlots()`:
```
* Note that please don't cache the value returned by this method,
because the number can change
* due to add/remove executors.
```
It shall be fine to cache that within different stages of a job, but it
requires a few more changes that will make the current PR more complicated.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
