[
https://issues.apache.org/jira/browse/STORM-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16424819#comment-16424819
]
Jungtaek Lim commented on STORM-2983:
-------------------------------------
If you're not concerned about retaining the optimization then let's remove it.
As explained enough (single worker should be minor one in production use
cases), I think we don't need to put it back again. Other concerns look like
the things to discuss, since we don't look like having consensus. That's why
[~ethanli] would like to file new issues regarding them, and IMHO I think it
may be better to initiate them in dev. mailing list so that we can share the
concerns and discuss them.
As you seem to think I'm missing, I may need to explain my thought below:
1. Even we want to retain the optimization, I think I proposed "the right way"
to do it in my comment. Checking topology worker count is "simplest" one for
activating/deactivating transfer thread, but "right one" is checking whether
outgoing stream exists in that worker. Multiple workers can be run
independently (not communicating with each other) depending on how scheduler
plans. Isn't it? We never utilized topology worker count in specific worker,
and I think worker still has enough information to deal with such optimization
without knowing topology worker count.
2. Regarding modifying topology configuration, I think I already put my 2 cent
on last comment. Whenever I need to check which value I set in configuration,
I'm checking the value in UI. If the value is different than what user put
before, we still need to explain why it is happening to avoid confusion. That's
not ideal. I think the topology configuration should be immutable one, though
we don't guarantee it from the code. Maybe we would need to have context object
like StormContext?
3. We already have StormTopology as well as TopologyContext which provide
necessary information. Dynamic informations like assignment are handled by
worker. The number of worker in topology is not exposed directly, but can be
calculated from Assignment. (Please refer
[https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/worker/WorkerState.java#L357]
and
[https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/generated/Assignment.java])
Based on 2, I would try to calculate it if it really needs.
4. We construct the configuration from combination of command line, code, conf.
file, etc. We assume they're not changing during runtime, which makes us safe
to read the configuration multiple times in different places. There's a chance
for the assumption to be broken if we modify the value in runtime.
5. I admit that it is too ideal for the state of current Storm, but the count
of worker in topology can be increased or decreased in runtime. If we deal with
this via relaunching all the workers then the value can be considered as
static, but if not, the value will be dynamic which topology configuration is
not doable to contain it.
> Some topologies not working properly
> -------------------------------------
>
> Key: STORM-2983
> URL: https://issues.apache.org/jira/browse/STORM-2983
> Project: Apache Storm
> Issue Type: Bug
> Reporter: Ethan Li
> Assignee: Ethan Li
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For example,
> {code:java}
> bin/storm jar storm-loadgen-*.jar
> org.apache.storm.loadgen.ThroughputVsLatency --spouts 1 --splitters 2
> --counters 1 -c topology.debug=true
> {code}
> on ResourceAwareScheduler not working properly.
> With default cluster settings, there will be only one __acker-executor and it
> will be on a separate worker. And it looks like the __acker-executor was not
> able to receive messages from spouts and bolts. And spouts and bolts
> continued to retry sending messages to acker. It then led to another problem:
> STORM-2970
> I tried to run on storm right before
> [https://github.com/apache/storm/pull/2502] and right after and confirmed
> that this bug should be related to it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
