[
https://issues.apache.org/jira/browse/SPARK-19823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15896096#comment-15896096
]
DjvuLee edited comment on SPARK-19823 at 3/5/17 7:10 AM:
---------------------------------------------------------
When Spark distributes tasks to Executors, it uses a Round-Robin way, this
means we distribute tasks to all Executors as possible, this is a good strategy
in most case.
However, when we introduce the dynamic allocation and concurrent jobs, users
may need the gang distribution for one single job, this means we allocate tasks
for one job in few Executors, so we can release the unused resource.
In current ways, Executors may not get released since the long-running stage
will distributed at lease one task in each Executors, even we have more cores
in each Executor.
We can offer a configuration for users, and this will not introduce much
complexity.
was (Author: djvulee):
When Spark distributes tasks to Executors, it uses a Round-Robin way, this
means we distribute tasks to all Executors as possible, this is a good strategy
in most case.
However, when we introduce the dynamic allocation and concurrent jobs, users
may need the gang distribution for one single job, this means we allocate tasks
for one job in few Executors, so we can release the unused resource.
In current ways, Executors may not get released since the long-running stage
will distributed on task in each Executors, even we have more cores in each
Executor.
We can offer a configuration for users, and this will not introduce much
complexity.
> Support Gang Distribution of Task
> ----------------------------------
>
> Key: SPARK-19823
> URL: https://issues.apache.org/jira/browse/SPARK-19823
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler, Spark Core
> Affects Versions: 2.0.2
> Reporter: DjvuLee
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
