[
https://issues.apache.org/jira/browse/STORM-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876154#comment-14876154
]
ASF GitHub Bot commented on STORM-893:
--------------------------------------
GitHub user jerrypeng opened a pull request:
https://github.com/apache/storm/pull/746
[STORM-893] Resource Aware Scheduling
I have created a initial open source implementation of the Resource Aware
Scheduler as described in the paper I published:
web.engr.illinois.edu/~bpeng/files/r-storm.pdf
The paper describes the general architecture, concepts, and algorithms used.
I have written an example topology that demonstrates how to use the API I
have written to specify resource requirements in your topology. Currently the
user can only specify two types of resources: Memory and CPU. We plan on
adding support for more resources in the future.
Currently, there is no built in enforcement of resource usage in worker
processes, however, we plan to add that functionality via CGroups.
People that worked on the implementation at Yahoo with me:
Bobby Evans (Yahoo & Storm PMC)
Derek Dagit (Yahoo & Storm PMC)
Kyle Nusbaum (Yahoo & Storm PMC)
Liu Zhuo (Yahoo)
Sanket Chintapalli (Yahoo)
Reza Fravier (Yahoo & UIUC)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerrypeng/storm opensource_ras
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/746.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #746
----
commit e014804a68b9c6887c64b9d03a17929863b80909
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-09-17T20:50:44Z
[STORM-893] - Resource Aware Scheduler implementation
[STORM-894] - Basic Resource Aware Scheduling implementation.
commit 0f3f237f158218cbe46ffc59670f300875eb7950
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-09-17T20:02:44Z
Added functionality for users to limit the amount of memory resources
allocated to a worker (JVM) process when scheduling with resource aware
scheduler. This allows users to potentially spread executors more evenly across
workers.
Also refactored some code
commit 28ee867c5a27c220be562689e61f4bb959a1aa62
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-09-17T20:56:23Z
regenerated thrift
commit 0256f6baff63e8c394ebfddbb68d38de5893b053
Author: Boyang Jerry Peng <[email protected]>
Date: 2015-09-18T18:23:46Z
adding miscellaneous things
----
> Resource Aware Scheduling
> -------------------------
>
> Key: STORM-893
> URL: https://issues.apache.org/jira/browse/STORM-893
> Project: Apache Storm
> Issue Type: Umbrella
> Reporter: Robert Joseph Evans
> Assignee: Boyang Jerry Peng
>
> At Yahoo we have been working on resource aware scheduling in storm, based
> off of some work done in academia. This rollup ticket is to track the
> complete project. With several sub tasks. Some that are already done and
> need to be pushed back, and others that we have not started on yet.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
