[
https://issues.apache.org/jira/browse/STORM-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14703632#comment-14703632
]
Boyang Jerry Peng commented on STORM-893:
-----------------------------------------
Hello Everyone,
I recently published a paper in ACM Middleware on the subject of Resource Aware
Scheduling in Storm in which I demonstrated how the scheduling algorithms I
used can significantly improve performance and stability of Storm. I have
created an implementation for open source based off of a implementation I
worked on a Yahoo. The implementation is on my github. The link is provided
below:
https://github.com/jerrypeng/storm/tree/opensource_ras
The link to the paper:
http://web.engr.illinois.edu/~bpeng/files/r-storm.pdf
The paper describes the architecture, algorithms, and API of the Resource Aware
Scheduler as well as an evaluation of performance. The research and initial
implementation was done by me while I was a graduate student at University of
Illinois, Urbana-champaign. After I graduated, I joined Yahoo and we came up
with a better implementation.
The implementation should work straight out of the box. If you are running
storm in a multi-rack environment, and you want to leverage the network aware
portion of the scheduler to improve performance and decrease latency, you can
implement your own STORM_NETWORK_TOPOLOGY_PLUGIN that maps the underlying
physical network. The default plugin that comes with the implementation
assumes all machines have the same network distance from each other i.e. on the
same rack. Contact me if you need more information on this subject.
Please look at this initial implementation and provide me with some feedback,
since we intend to have a pull request up soon and merge it into master.
People that worked on the implementation at Yahoo with me:
Bobby Evans (Yahoo & Storm PMC)
Derek Dagit (Yahoo & Storm PMC)
Kyle Nusbaum (Yahoo & Storm PMC)
Liu Zhuo (Yahoo)
Sanket Chintapalli (Yahoo)
Reza Fravier (Yahoo & UIUC)
Best,
Boyang Jerry Peng
Software Engineer, Yahoo
http://web.engr.illinois.edu/~bpeng/
> Resource Aware Scheduling
> -------------------------
>
> Key: STORM-893
> URL: https://issues.apache.org/jira/browse/STORM-893
> Project: Apache Storm
> Issue Type: Umbrella
> Reporter: Robert Joseph Evans
> Assignee: Boyang Jerry Peng
>
> At Yahoo we have been working on resource aware scheduling in storm, based
> off of some work done in academia. This rollup ticket is to track the
> complete project. With several sub tasks. Some that are already done and
> need to be pushed back, and others that we have not started on yet.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
