Yuzhao Chen created STORM-2286:

             Summary: Strom Rebalance command should support arbitrary 
component parallelism
                 Key: STORM-2286
                 URL: https://issues.apache.org/jira/browse/STORM-2286
             Project: Apache Storm
          Issue Type: Bug
          Components: storm-core
    Affects Versions: 0.9.3, 0.9.6, 0.10.1, 1.0.1
            Reporter: Yuzhao Chen
             Fix For: 1.x

For legacy reasons, config TOPOLOGY-TASKS is considered first when schedule a 
topology, for a component, if user don’t specify TOPOLOGY-TASKS, storm just 
override it to be equal to component parallelism hint, and schedule based on 

This works for the most cases, but not Rebalance command. Now, when do 
Rebalance, the StormBase :component->executors attribute will be overridden in 
Zookeeper which is used to partition component tasks into executors, as we said 
above, the TOPOLOGY-TASKS is considered here as the real tasks number for 
components, something goes weird here:

If we override a bigger executor numbers for a component when do rebalance, it 
just don’t work because smaller TOPOLOGY-TASKS [ not changed since first 
submitted at all ]is partitioned into bigger number of executors which read 
from ZooKeeper overridden by Rebalance command, but for smaller task, it works 

I see that storm support a command like this now: [storm rebalance 
topology-name [-w wait-time-secs] [-n new-num-workers] [-e 
component=parallelism]*] which indicate that user can override a component 
parallelism freely, i think it’s more sensible to support this and it's 
meaningless to have a restriction like before.

This message was sent by Atlassian JIRA

Reply via email to