[
https://issues.apache.org/jira/browse/STORM-132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-132:
-------------------------------
Fix Version/s: (was: 0.9.2-incubating)
0.9.3
> Default Storm scheduler should evenly use resources when managing multiple
> topologies
> -------------------------------------------------------------------------------------
>
> Key: STORM-132
> URL: https://issues.apache.org/jira/browse/STORM-132
> Project: Apache Storm
> Issue Type: Improvement
> Reporter: James Xu
> Assignee: Kang Xiao
> Fix For: 0.9.3
>
>
> https://github.com/nathanmarz/storm/issues/359
> Currently, a single topology is evenly spread across the cluster, but this is
> not the case for multiple topologies (it targets one node first, then the
> rest). The default scheduler should order the hosts for scheduling in terms
> of which has the least slots used.
> ----------
> lyogavin: To confirm, we want to firstly look at the number of used slots on
> each host. Firstly try to balance the used slot number across all the hosts.
> For the assignment inside each host, we also try to evenly balance the usage
> of each slot. Am I understanding correctly?
> When i coding, i realized it's actually a little complicated. Looks like
> there are many policies here we want to consider:
> 1. Evenness of resource usage. (Do we want to evaluate evenness according to
> number of slots used in each host or the number of executors? Maybe number of
> executors is better, but also make it very complicated)
> 2. Least rescheduling. We probabaly also want to make the assignment change
> as less as possible. Looks like this is why DefaultScheduler.bad-slots is
> coded the current way.
> 3. Number of workers.
> Then the question is how do we prioritize those policies. Sometimes they
> conflict to each other. For example, sometimes the most even distribution may
> need the most reassignment.
> Any thoughts?
> ----------
> xumingming: @lyogavin I think you might have over-thought this, as
> @nathanmarz already confirmed, you just need to update
> EvenScheduler.sort-slots to make sure the slots in the least used node appear
> first in the available-slots list.
> ----------
> lyogavin: Thanks James. I got what you mean. So i'll not worry about the
> usage of the executor usage. Only consider the balance of slots usage.
> But I think just simply change the sort-slots to sort the slots based on
> slots usage may still not work too well. For example, let's say there are 2
> hosts, with 10 slots on each. Say 1st one used 1 slots, 2nd one used 2 slots,
> and we want to assign another topology with 8 workers. If we simply sort the
> slots based on usage, we'll end up with the list [0 0 0 0 0 0 0 0 1 1], then
> the new assignment we'll get would be all from the 1st host. Not balanced.
> So in the above pull request, I implemented a solution in the way similar to
> watershed algorithm. It would firstly pick the slots from least used host,
> until that host uses the same number of slots as the second least used slots.
> Then it evenly picks slots from the 2 least used hosts until reaches the 3rd
> one. Iterating this way, we can get the best balanced assignment.
> What do you think?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
