GitHub user jerrypeng opened a pull request:
https://github.com/apache/storm/pull/746
[STORM-893] Resource Aware Scheduling
I have created a initial open source implementation of the Resource Aware
Scheduler as described in the paper I published:
web.engr.illinois.edu/~bpeng/files/r-storm.pdf
The paper describes the general architecture, concepts, and algorithms used.
I have written an example topology that demonstrates how to use the API I
have written to specify resource requirements in your topology. Currently the
user can only specify two types of resources: Memory and CPU. We plan on
adding support for more resources in the future.
Currently, there is no built in enforcement of resource usage in worker
processes, however, we plan to add that functionality via CGroups.
People that worked on the implementation at Yahoo with me:
Bobby Evans (Yahoo & Storm PMC)
Derek Dagit (Yahoo & Storm PMC)
Kyle Nusbaum (Yahoo & Storm PMC)
Liu Zhuo (Yahoo)
Sanket Chintapalli (Yahoo)
Reza Fravier (Yahoo & UIUC)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerrypeng/storm opensource_ras
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/746.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #746
----
commit e014804a68b9c6887c64b9d03a17929863b80909
Author: Boyang Jerry Peng <jerryp...@yahoo-inc.com>
Date: 2015-09-17T20:50:44Z
[STORM-893] - Resource Aware Scheduler implementation
[STORM-894] - Basic Resource Aware Scheduling implementation.
commit 0f3f237f158218cbe46ffc59670f300875eb7950
Author: Boyang Jerry Peng <jerryp...@yahoo-inc.com>
Date: 2015-09-17T20:02:44Z
Added functionality for users to limit the amount of memory resources
allocated to a worker (JVM) process when scheduling with resource aware
scheduler. This allows users to potentially spread executors more evenly across
workers.
Also refactored some code
commit 28ee867c5a27c220be562689e61f4bb959a1aa62
Author: Boyang Jerry Peng <jerryp...@yahoo-inc.com>
Date: 2015-09-17T20:56:23Z
regenerated thrift
commit 0256f6baff63e8c394ebfddbb68d38de5893b053
Author: Boyang Jerry Peng <jerryp...@yahoo-inc.com>
Date: 2015-09-18T18:23:46Z
adding miscellaneous things
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
