[
https://issues.apache.org/jira/browse/SPARK-27104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hua Zhang updated SPARK-27104:
------------------------------
Description:
Spark in standalone mode currently only supports FIFO (first-in-first-out)
scheduler across applications.
It will be great that a fair scheduler is supported. +A fair scheduler across
applications, not in a application.+
Use case (for example with the integration of zeppelin)
At certain moment, user A submits a heavy application and consumes all the
resources of the spark cluster.
At a later moment, user B submits a second application.
No matter how many work nodes you added now, all the resources go to user A due
to the FIFO. User B will never get any resource until user A release its
allocated resources.
A fair scheduler should distribute extra resources in a fair way on all running
applications, which demands resources.
was:
Spark in standalone mode currently only supports FIFO (first-in-first-out)
scheduler across applications.
It will be great that a fair scheduler is supported. +A fair scheduler across
applications, not in a application.+
Use case (for example with the integration of zeppelin)
At certain moment, user A submits an heavy application and consumes all the
resources of the spark cluster.
At a later moment, user B submits a second application.
No matter how many work nodes you added now, all the resources go to user A due
to the FIFO. User B will never get any resource until user A release its
allocated resources.
A fair scheduler should distribute extra resources in a fair way on all running
applications, which demands resources.
> Spark Fair scheduler across applications in standalone mode
> -----------------------------------------------------------
>
> Key: SPARK-27104
> URL: https://issues.apache.org/jira/browse/SPARK-27104
> Project: Spark
> Issue Type: Wish
> Components: Scheduler
> Affects Versions: 2.2.3, 2.3.3, 2.4.0
> Reporter: Hua Zhang
> Priority: Minor
>
> Spark in standalone mode currently only supports FIFO (first-in-first-out)
> scheduler across applications.
> It will be great that a fair scheduler is supported. +A fair scheduler across
> applications, not in a application.+
>
> Use case (for example with the integration of zeppelin)
> At certain moment, user A submits a heavy application and consumes all the
> resources of the spark cluster.
> At a later moment, user B submits a second application.
> No matter how many work nodes you added now, all the resources go to user A
> due to the FIFO. User B will never get any resource until user A release its
> allocated resources.
>
> A fair scheduler should distribute extra resources in a fair way on all
> running applications, which demands resources.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
