[
https://issues.apache.org/jira/browse/SPARK-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290448#comment-15290448
]
Kay Ousterhout commented on SPARK-15176:
----------------------------------------
I'm generally hesitant to add complexity to the scheduler unless there's
significant demand for a feature, since there are endless features we could add
that would benefit 1 or 2 people, but that in aggregate would make the
scheduler much more difficult to maintain. Can you elaborate a little more on
the workload for this? Why isn't it sufficient to wait for tasks in the first
job to complete? It seems like, as long as the second job has a non-zero share,
as soon as a task in the first job completes, some of the second job's tasks
should be assigned?
> Job Scheduling Within Application Suffers from Priority Inversion
> -----------------------------------------------------------------
>
> Key: SPARK-15176
> URL: https://issues.apache.org/jira/browse/SPARK-15176
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
> Affects Versions: 1.6.1
> Reporter: Nick White
>
> Say I have two pools, and N cores in my cluster:
> * I submit a job to one, which has M >> N tasks
> * N of the M tasks are scheduled
> * I submit a job to the second pool - but none of its tasks get scheduled
> until a task from the other pool finishes!
> This can lead to unbounded denial-of-service for the second pool - regardless
> of `minShare` or `weight` settings. Ideally Spark would support a pre-emption
> mechanism, or an upper bound on a pool's resource usage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
