Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/7274#discussion_r34914574
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
---
@@ -543,39 +544,72 @@ private[master] class Master(
* multiple executors from the same application may be launched on the
same worker if the worker
* has enough cores and memory. Otherwise, each executor grabs all the
cores available on the
* worker by default, in which case only one executor may be launched on
each worker.
+ *
+ * It is important to allocate coresPerExecutor on each worker at a time
(instead of 1 core
+ * at a time). Consider the following example: cluster has 4 workers
with 16 cores each.
+ * User requests 3 executors (spark.cores.max = 48, spark.executor.cores
= 16). If 1 core is
+ * allocated at a time, 12 cores from each worker would be assigned to
each executor.
+ * Since 12 < 16, no executors would launch [SPARK-8881].
*/
- private def startExecutorsOnWorkers(): Unit = {
- // Right now this is a very simple FIFO scheduler. We keep trying to
fit in the first app
- // in the queue, then the second app, etc.
+ private[master] def coresToAssign(
--- End diff --
you misunderstood me. I was asking you to rename the variable `toAssign` to
`coresToAssign`. This method name should stay as `scheduleExecutorsOnWorkers`
as before.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
