[
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15039617#comment-15039617
]
ASF GitHub Bot commented on STORM-898:
--------------------------------------
GitHub user jerrypeng opened a pull request:
https://github.com/apache/storm/pull/921
[STORM-898] - Add priorities and per user resource guarantees to Resource
Aware Scheduler
In a multi-tenant environment we would like to be able to give individual
users a guarantee of how much CPU/Memory/Network they will be able to use in a
cluster. We would also like to know which topologies a user feels are the most
important to keep running if there are not enough resources to run all of their
topologies.
Each user should be able to specify if their topology is production,
staging, or development. Within each of those categories a user should be able
to give a topology a priority, 0 to 30 with 30 being the highest priority (or
something like this).
If there are not enough resources on a cluster to run a topology assume
this topology is running using resources and find the user that is most over
their guaranteed resources. Shoot the lowest priority topology for that user,
and repeat until, this topology is able to run, or this topology would be the
one shot. Ideally we don't actually shoot anything until we know that we would
have made enough room.
If the cluster is over-subscribed and everyone is under their guarantee,
and this topology would not put the user over their guarantee. Shoot the lowest
priority topology in this workers resource pool until there is enough room to
run the topology or this topology is the one that would be shot. We might also
want to think about what to do if we are going to shoot a production topology
in an oversubscribed case, and perhaps we can shoot a non-production topology
instead even if the other user is not over their guarantee.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerrypeng/storm STORM-898
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/921.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #921
----
commit 6fc5b0eded64771e25c250848f284a84e1ee8a81
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-11-30T03:42:26Z
[STORM-898] - Add priorities and per user resource guarantees to Resource
Aware Scheduler
commit 4b34ccb82e863cc6ed48851856af5b11b3986cab
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-11-12T15:46:05Z
adding configs and validation
commit 7ed348df1537c3b79a9c8a3eed793b89627c573f
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-11-12T15:48:07Z
adding unit tests for STORM-898
commit a585303babcc92ed0e36afb8d6f094211184eb25
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-11-12T15:55:44Z
adding additional unit tests
commit 3d90898a5c2f3e572bdfc8b6515d8a3f1b8ae40f
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-11-16T06:45:14Z
fixing exisiting unit tests
commit 69b70ff71f26120a0b51c8776e14fea46d01fe79
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-11-18T17:09:45Z
adding checkpointing
commit 404d3bde38d08eff9ae0f51c35b9768b6bebecab
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-11-24T19:25:18Z
fixing bugs and adding more unit tests
commit e3d0de6fa65cf4b26731c7d436d971f2140f716e
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-11-24T22:36:36Z
made scheduling, eviction, and priority strategies pluggable
commit 17f3bf47efc7c925398dccf923f9b723569d7128
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-11-25T23:13:44Z
first initial implementation
commit 18595df0aa90f93bd5be07f7943b1a4c8a2b7f98
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-11-30T03:04:15Z
fixing clojure unit tests
commit 19557d170e6dc223f67b5414cd67a2996737f01d
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-11-30T04:16:22Z
making fixes after rebasing
commit 5955969c44764546bc592476fd5676e0855c128e
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-11-30T19:14:32Z
adding additional check for topology submitted without a user
commit e6ada82058f2f6182b96424ab584738dd269164a
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-11-30T19:33:00Z
fix bug related submitting a topology without a user
commit 62389030b883ac4f7c735fdccc815243f6e9cc71
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-11-30T20:15:09Z
if resource guarantees are null, set them to 0.0
commit 92eb964671ab1a9c78d2624f6dd12c681907a4d9
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-11-30T21:40:38Z
making fixes for when user does not have resource guarantees
commit 9a8a613c21951b1b47db7c722ef962489fdb17e2
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-11-30T22:34:52Z
fixing bug with setting topology status messages
commit bc7a4eec962d141bd493a10b59cd30c85c42a6d8
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-12-02T19:49:03Z
creating another unit test for fault tolerance
commit c9889996b9ba1bf64bd2f8158b6c809fa6ccbf4b
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-12-02T19:54:26Z
adding to ResourceAwareExampleTopology to demonstrate new API
commit 0c4c4fbf77f9509303c8baf584fd43ca0669d316
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-12-02T22:59:26Z
cleaning up debug comments and fixing unit test bugs
----
> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---------------------------------------------------------------------------
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
> Issue Type: New Feature
> Components: storm-core
> Reporter: Robert Joseph Evans
> Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual
> users a guarantee of how much CPU/Memory/Network they will be able to use in
> a cluster. We would also like to know which topologies a user feels are the
> most important to keep running if there are not enough resources to run all
> of their topologies.
> Each user should be able to specify if their topology is production, staging,
> or development. Within each of those categories a user should be able to give
> a topology a priority, 0 to 10 with 10 being the highest priority (or
> something like this).
> If there are not enough resources on a cluster to run a topology assume this
> topology is running using resources and find the user that is most over their
> guaranteed resources. Shoot the lowest priority topology for that user, and
> repeat until, this topology is able to run, or this topology would be the one
> shot. Ideally we don't actually shoot anything until we know that we would
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and
> this topology would not put the user over their guarantee. Shoot the lowest
> priority topology in this workers resource pool until there is enough room to
> run the topology or this topology is the one that would be shot. We might
> also want to think about what to do if we are going to shoot a production
> topology in an oversubscribed case, and perhaps we can shoot a non-production
> topology instead even if the other user is not over their guarantee.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
