[
https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16278405#comment-16278405
]
Julien Cuquemelle edited comment on SPARK-22683 at 12/5/17 1:09 PM:
--------------------------------------------------------------------
Thanks [~srowen] for your quick feedback, let me give you more context /
explanation about what I'm trying to achieve here.
- I'm not trying to mitigate the overhead of launching small tasks, I'm trying
to avoid allocating executors that will not / barely be used.
As spark tuning guideline usually aim towards having small tasks, it seems
sub-optimal to allocate a large number of executors at application start (or at
any stage that would create much more tasks than available executor slots),
that end up being discarded without having been used.
- I've scoped this parameter to dynamic allocation, as it only impacts the way
the executor count ramps up when tasks are backlogged. The default value of the
parameter falls back to the current behavior.
- I do agree that over-allocated containers will eventually be deallocated, but
this resource occupation by idle executors appears to be
non trivial on some of our workloads:
I've made several experiments on a set of similar jobs working on different
data sizes, and running on an idle or busy cluster;
The resource usage (vcore-hours allocated as seen by Yarn) of the spark
jobs is compared to the same job running in MapReduce;
The executor idle timeout is set to 30s;
When running with the current setting (1 task per executor slot), our Spark
jobs consume in average twice the amount of vcorehours than the MR job
When running with 6 tasks per executor slot, our Spark jobs consume in
average 30% less vcorehours than the MR jobs, this setting being valid for
different workload sizes.
was (Author: jcuquemelle):
Thanks [~srowen] for your quick feedback, let me give you more context /
explanation about what I'm trying to achieve here.
- I'm not trying to mitigate the overhead of launching small tasks, I'm trying
to avoid allocating executors that will not / barely be used.
As spark tuning guideline usually aim towards having small tasks, it seems
sub-optimal to allocate a large number of executors at application start
(or at any stage that would create much more tasks than available executor
slots), that end up being discarded without having been used.
- I've scoped this parameter to dynamic allocation, as it only impacts the way
the executor count ramps up when tasks are backlogged. The default value of the
parameter falls back to the current behavior.
- I do agree that over-allocated containers will eventually be deallocated, but
this resource occupation by idle executors appears to be
non trivial on some of our workloads:
I've made several experiments on a set of similar jobs working on different
data sizes, and running on an idle or busy cluster;
The resource usage (vcore-hours allocated as seen by Yarn) of the spark
jobs is compared to the same job running in MapReduce;
The executor idle timeout is set to 30s;
When running with the current setting (1 task per executor slot), our Spark
jobs consume in average twice the amount of vcorehours than the MR job
When running with 6 tasks per executor slot, our Spark jobs consume in
average 30% less vcorehours than the MR jobs, this setting being valid for
different workload sizes.
> Allow tuning the number of dynamically allocated executors wrt task number
> --------------------------------------------------------------------------
>
> Key: SPARK-22683
> URL: https://issues.apache.org/jira/browse/SPARK-22683
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.0, 2.2.0
> Reporter: Julien Cuquemelle
> Labels: pull-request-available
>
> let's say an executor has spark.executor.cores / spark.task.cpus taskSlots
> The current dynamic allocation policy allocates enough executors
> to have each taskSlot execute a single task, which minimizes latency,
> but wastes resources when tasks are small regarding executor allocation
> overhead.
> By adding the tasksPerExecutorSlot, it is made possible to specify how many
> tasks
> a single slot should ideally execute to mitigate the overhead of executor
> allocation.
> PR: https://github.com/apache/spark/pull/19881
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
