GitHub user jcuquemelle opened a pull request:
https://github.com/apache/spark/pull/19881
[SPARK-22683][CORE] Add tasksPerExecutorSlot parameter
## What changes were proposed in this pull request?
let's say an executor has spark.executor.cores / spark.task.cpus taskSlots
The current dynamic allocation policy allocates enough executors
to have each taskSlot execute a single task, which wastes resources when
tasks are small regarding executor allocation overhead. By adding the
tasksPerExecutorSlot, it is made possible to specify how many tasks
a single slot should ideally execute to mitigate the overhead of executor
allocation.
## How was this patch tested?
Units tests and runs on various actual workloads on a Yarn Cluster
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jcuquemelle/spark AddTaskPerExecutorSlot
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19881.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19881
----
commit 895882feebc53f44a70278e0b475b2fb937d331a
Author: Julien Cuquemelle <[email protected]>
Date: 2017-11-30T16:28:06Z
[SPARK-22683][CORE] Allow tuning the number of dynamically allocated
executors
let's say an executor has spark.executor.cores / spark.task.cpus taskSlots
The current dynamic allocation policy allocates enough executors
to have each taskSlot execute a single task, which wastes resources when
tasks are small regarding executor allocation overhead. By adding the
tasksPerExecutorSlot, it is made possible to specify how many tasks
a single slot should ideally execute to mitigate the overhead of executor
allocation.
commit fce3b976d0b22c4d01ef4fdd5339835bc6d6fcb1
Author: Julien Cuquemelle <[email protected]>
Date: 2017-11-30T16:28:06Z
[SPARK-22683][DOC] document tasksPerExecutorSlot parameter
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
