GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/8129
[SPARK-9882][Core] Priority-based scheduling for Spark applications
JIRA: https://issues.apache.org/jira/browse/SPARK-9882
This is a priority-based scheduling based on the interface proposed in
#7958. It uses a XML configuration file to specify the submission pools for
applications. This patch adds a new parameter `--pool` to SparkSubmit under
standalone mode to specify which pool the submitted application should be
assigned to. The priority of the submitted application is defined by the
assigned pool. It also defines the cores it can acquire.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/viirya/spark-1 priority_scheduler_pr
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8129.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8129
----
commit 25a3d7580bd6f3934f52b48faef1eed825bea4fb
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-04T10:53:44Z
Add interface for SchedulingAlgorithm.
commit 36db2c384e15cebee4cee95bb1fd31a97459a74b
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-05T09:34:51Z
Add class PrioritySchedulingAlgorithm.
commit 340bb1251b8ddda1ab3577752f23c220036747fc
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-06T06:31:16Z
Fix MasterSuite.
commit 79ed50138b3b287e4aed0116ceea656b403c0107
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-11T09:08:38Z
Fix MasterSuite for newly added test.
commit 6b443827a15e31159af3fde3e06a84080c0a5e4f
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-11T09:27:08Z
Add Pool and Queue.
commit 12519ebccdcb654f89b0039791174ab3426b6730
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-11T09:36:12Z
Add wrongly deleted file.
commit a0789805f0cfbe9f3ff703f9161aab3b2ea9cedd
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-11T11:02:22Z
Add test for parsing pool setting.
commit c978ddfbc170c21231f3b4a6206e9f6c3ce7a134
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-12T02:05:17Z
Add test.
commit 112573fcf4fdff588d1099acacea3cf46b12bbd7
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-12T04:49:00Z
Add basic scheduling for applications in pools.
commit cc68c84a18ec32e32467c6ab13942a653bb59636
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-12T07:53:20Z
Add --pool parameter to SparkSubmit.
commit f7ec9cce2b38f8c4f1a67f362de484dedcae7c0d
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-12T08:16:01Z
Add removeApplication into SchedulingAlgorithm.
commit 607450a9d1b126dfdeb708b34a3a611dc7dbd1d9
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-12T10:12:29Z
Fix asInstanceOf bug.
commit d09d027d94bbd38c46896a0282bed35527726e1b
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-12T11:04:57Z
Load default pool setting if no configuration file specified. Use
pool.cores - app.coresGranted instead of app.coresLeft.
commit 3a4e4cb20bee618eddeb2e40a6e7efbff3ced3bc
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-12T15:42:16Z
Use Date instead of LocalDateTime (only in Java 8).
commit 3b48cfa4781b2c386ff93da57d601eeb7abd10fe
Author: Liang-Chi Hsieh <[email protected]>
Date: 2015-08-12T15:59:35Z
Add configuration file template.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
