GitHub user danny0405 opened a pull request:
https://github.com/apache/storm/pull/1874
Strom Rebalance command should support arbitrary component parallelism
For legacy reasons, config TOPOLOGY-TASKS is considered first when
schedule a topology, for a component, if user donât specify TOPOLOGY-TASKS,
storm just override it to be equal of component parallelism hint, and schedule
based on TOPOLOGY-TASKS later on.
This works for the most cases, but not Rebalance command, now, when do
Rebalance, the StormBase :component->executors attribute will be overridden in
Zookeeper is used to partition tasks into executors for components, as we said
above, the TOPOLOGY-TASKS is considered here as the real tasks number for
components, something goes weird here:
If we override a bigger executor numbers for a component when do
rebalance, it just donât work because smaller TOPOLOGY-TASKS [ not changed
since first submitted at all ]is partitioned to bigger number of executors
which read from ZooKeeper overridden by Rebalance command, but for smaller
task, it works fine.
I see that storm support a command like this now: [storm rebalance
topology-name [-w wait-time-secs] [-n new-num-workers] [-e
component=parallelism]*] which indicate that user can override a component
parallelism freely, i think itâs more sensible to support this and it's
meaningless to have a restriction like this.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/danny0405/storm nimbus-promotion
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1874.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1874
----
commit de4374fd1da4bb0a647f125b91f2f15644c0b2f7
Author: chenyuzhao <[email protected]>
Date: 2017-01-12T09:21:55Z
refact rebalance to allow modify component parallelism freely
commit 1384071d58020d03c018d9ce98638ee330539300
Author: chenyuzhao <[email protected]>
Date: 2017-01-13T07:22:09Z
format code
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
