[
https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16286253#comment-16286253
]
Julien Cuquemelle edited comment on SPARK-22683 at 12/12/17 3:14 PM:
---------------------------------------------------------------------
The impression I get from our discussion is that you mainly focus on the
latency of the jobs, and that the current setting is optimized for that, which
is why you consider the current setup sufficient.
If you consider your previous example from a resource usage point of view, my
proposal would allow to have about the same resource usage in both scenarios,
but the current setup doubles the resource usage of the workload with small
tasks...
I've tried to experiment with the parameters you've proposed, but right now I
don't have a solution to optimize my type of workload (not every single job)
for resource consumption.
I don't know if the majority of Spark users run on idle clusters, but ours is
routinely full, so for us resource usage is more important than latency.
was (Author: jcuquemelle):
The impression I get from our discussion is that you mainly focus on the
latency of the jobs, and that the current setting is optimized for that, which
is why you consider the current setup sufficient.
If you consider your previous example from a resource usage point of view, my
proposal would allow to have about the same resource usage in both scenarios,
but the current setup doubles the resource usage of the workload with small
tasks...
I've tried to experiment with the parameters you've proposed, but right now I
don't have a solution to optimize my type of workload (nor every single job)
for resource consumption.
I don't know if the majority of Spark users run on idle clusters, but ours is
routinely full, so for us resource usage is more important than latency.
> Allow tuning the number of dynamically allocated executors wrt task number
> --------------------------------------------------------------------------
>
> Key: SPARK-22683
> URL: https://issues.apache.org/jira/browse/SPARK-22683
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.0, 2.2.0
> Reporter: Julien Cuquemelle
> Labels: pull-request-available
>
> let's say an executor has spark.executor.cores / spark.task.cpus taskSlots
> The current dynamic allocation policy allocates enough executors
> to have each taskSlot execute a single task, which minimizes latency,
> but wastes resources when tasks are small regarding executor allocation
> overhead.
> By adding the tasksPerExecutorSlot, it is made possible to specify how many
> tasks
> a single slot should ideally execute to mitigate the overhead of executor
> allocation.
> PR: https://github.com/apache/spark/pull/19881
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
