[
https://issues.apache.org/jira/browse/STORM-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
P. Taylor Goetz updated STORM-3112:
-----------------------------------
Fix Version/s: (was: 2.0.0)
> Incremental scheduling supports
> -------------------------------
>
> Key: STORM-3112
> URL: https://issues.apache.org/jira/browse/STORM-3112
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-server
> Affects Versions: 2.0.0
> Reporter: Yuzhao Chen
> Assignee: Yuzhao Chen
> Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> As https://issues.apache.org/jira/browse/STORM-3093 described, now the
> scheduling work for a round is a complete scan and computation for all the
> topologies on cluster, which is a very heavy work when topologies increment
> to hundreds.
> So this JIRA is to refactor the scheduling logic that only care about
> topologies that need to.
> Promotions list:
> 1. Cache the id to storm base mapping which reduce the pressure to ZooKeeper.
> 2. Only schedule the topologies that need to: with dead executors or not
> enough running workers.
> 3. For some schedulers we still need a full scheduling, i.e.
> IsolationScheduler.
> 4. Cache the scheduling resource bestride multi scheduling round, i.e. nodeId
> -> used slot, nodeId -> used resource, nodeId -> totalResource.
> Cause in https://issues.apache.org/jira/browse/STORM-3093 i already cache the
> storm-id -> executors mapping, now for a scheduling round, thing we will do:
> 1. Scan all the active storm bases( cached ) and local
> storm-conf/storm-topology, then to refresh the heartbeats cache, and we will
> know which topologies need to schedule.
> 2. Compute scheduleAssignment only for need scheduling topologies.
> About robustness when nimbus restarts:
> 1. The cached storm-bases are taken care of by ILocalAssignmentsBackend.
> 2. the scheduling cache will be refresh for the first time scheduling through
> a full topologies scheduling.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
