[
https://issues.apache.org/jira/browse/SPARK-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15304589#comment-15304589
]
Mark Hamstra commented on SPARK-15176:
--------------------------------------
It's not an unreasonable use case, and is similar in many respects to the
issues that arise when trying to do multi-tenancy within a single Spark
Application -- i.e. you don't want a single query or ingestion Job to consume
all the cluster resources, blocking other Jobs from making any progress. There
currently isn't a good way to do this within a single Application since even
the fair-scheduling pools don't have a way to cap the resources that a
particular Job can grab and hold onto indefinitely as long as no other Job is
asking for those resources at the point where the greedy Job makes its resource
grab. As a consequence, the only way to address these kinds of maximum
resource usage requirements at present is to run multiple Applications while
capping the maximum resource usage for each Application. That, of course,
makes sharing RDDs much less convenient.
I'm not convinced that limiting the amount of tasks a given pool can run is the
correct approach.
There is, however, an interesting asymmetry in the current fair-scheduling.
The minShare property can set the minimum number of cores that a pool will
require to be able to schedule anything, but there is no corresponding property
to cap the maximum number of cores that a pool will use. That kind of maxShare
cap is really what this JIRA and https://github.com/apache/spark/pull/12951
are, or should be, driving at -- the discussion is currently a little muddled
with its emphasis on limiting Tasks when the focus of the discussion should
really be on limiting the cores available to a given pool.
If we want to be able to guarantee that one Job within an Application cannot
block all other Jobs from making any progress, then we need this maxShare
change; else we are forcing using users to run multiple Applications.
> Job Scheduling Within Application Suffers from Priority Inversion
> -----------------------------------------------------------------
>
> Key: SPARK-15176
> URL: https://issues.apache.org/jira/browse/SPARK-15176
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
> Affects Versions: 1.6.1
> Reporter: Nick White
>
> Say I have two pools, and N cores in my cluster:
> * I submit a job to one, which has M >> N tasks
> * N of the M tasks are scheduled
> * I submit a job to the second pool - but none of its tasks get scheduled
> until a task from the other pool finishes!
> This can lead to unbounded denial-of-service for the second pool - regardless
> of `minShare` or `weight` settings. Ideally Spark would support a pre-emption
> mechanism, or an upper bound on a pool's resource usage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
