liupengcheng created SPARK-31107:
------------------------------------
Summary: Extend FairScheduler to support pool level resource
isolation
Key: SPARK-31107
URL: https://issues.apache.org/jira/browse/SPARK-31107
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.0.0
Reporter: liupengcheng
Currently, spark only provided two types of scheduler: FIFO & FAIR, but in sql
high-concurrency scenarios, a few of drawbacks are exposed.
FIFO: it can easily causing congestion when large sql query occupies all the
resources
FAIR: the taskSets of one pool may occupies all the resource due to there are
no hard limit on the maximum usage for each pool. this case may be frequently
met under high workloads.
So we propose to add a maxShare argument for FairScheduler to control the
maximum running tasks for each pool.
One thing that needs our attention is that we should handle it well to make the
`ExecutorAllocationManager` can release resources:
e.g. Suppose we got 100 executors, if the tasks are scheduled on all executors
with max concurrency 50, there are cases that the executors may not idle, and
can not be released.
One idea is to bind those executors to each pool, then we only schedule tasks
on executors of the pool which it belongs to.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
