Yuzhao Chen created STORM-2286:
----------------------------------
Summary: Strom Rebalance command should support arbitrary
component parallelism
Key: STORM-2286
URL: https://issues.apache.org/jira/browse/STORM-2286
Project: Apache Storm
Issue Type: Bug
Components: storm-core
Affects Versions: 0.9.3, 0.9.6, 0.10.1, 1.0.1
Reporter: Yuzhao Chen
Fix For: 1.x
For legacy reasons, config TOPOLOGY-TASKS is considered first when schedule a
topology, for a component, if user don’t specify TOPOLOGY-TASKS, storm just
override it to be equal to component parallelism hint, and schedule based on
TOPOLOGY-TASKS later on.
This works for the most cases, but not Rebalance command. Now, when do
Rebalance, the StormBase :component->executors attribute will be overridden in
Zookeeper which is used to partition component tasks into executors, as we said
above, the TOPOLOGY-TASKS is considered here as the real tasks number for
components, something goes weird here:
If we override a bigger executor numbers for a component when do rebalance, it
just don’t work because smaller TOPOLOGY-TASKS [ not changed since first
submitted at all ]is partitioned into bigger number of executors which read
from ZooKeeper overridden by Rebalance command, but for smaller task, it works
fine.
I see that storm support a command like this now: [storm rebalance
topology-name [-w wait-time-secs] [-n new-num-workers] [-e
component=parallelism]*] which indicate that user can override a component
parallelism freely, i think it’s more sensible to support this and it's
meaningless to have a restriction like before.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
