Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/19881#discussion_r166649526
--- Diff:
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -114,9 +114,13 @@ private[spark] class ExecutorAllocationManager(
// TODO: The default value of 1 for spark.executor.cores works right now
because dynamic
// allocation is only supported for YARN and the default number of cores
per executor in YARN is
// 1, but it might need to be attained differently for different cluster
managers
- private val tasksPerExecutor =
+ private val taskSlotPerExecutor =
conf.getInt("spark.executor.cores", 1) /
conf.getInt("spark.task.cpus", 1)
+ private val tasksPerExecutorSlot =
conf.getInt("spark.dynamicAllocation.tasksPerExecutorSlot", 1)
--- End diff --
I think we should change the name of this config because spark doesn't have
the concept of slots and I think it could be confusing to the users who might
expect exactly x tasks to be processed on each executor. I am thinking more
along the lines of spark.dynamicAllocation.maxExecutorsPerStageDivisor=max # of
executors based on # of tasks required for that stage divided by this number.
I'm open to other config names here though.
I think we would also need to define its interaction with
spark.dynamicAllocation.maxExecutors as well as how it works as # of running/to
be run tasks changes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
