[jira] [Commented] (STORM-2983) Some topologies not working properly
[
https://issues.apache.org/jira/browse/STORM-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421158#comment-16421158
]
Jungtaek Lim commented on STORM-2983:
-
[~roshan_naik] [~ethanli]
Looks like this is about how we provide users to specify resources which are
needed for running topology.
Other than RAS, we assume there’re homogeneous JVM processes (fixed memory, no
restriction on CPU) so overall available memory for topology is configurable
via how much memory end-users assigns to single worker and how many workers
end-users assigns to topology.
This is no longer true in RAS, and maybe container environment, too. If my
understanding is correct, the major point on RAS / container scheduler is fully
utilizing resources in cluster, so user mainly configures how much resources
each component/worker need, and physical plan and placement is left to
scheduler.
I also agree topology configuration should be considered to public API. So if
such configuration is not effective to RAS, RAS document should mention such
behavior. Even better if we explain why it ignores such configuration.
I’m cc.ing [~revans2] since I think he is the best folk to answer about RAS,
and also cc.ing [~erikdw] to see how storm-mesos works for scheduling.
> Some topologies not working properly
> -
>
> Key: STORM-2983
> URL: https://issues.apache.org/jira/browse/STORM-2983
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Ethan Li
>Assignee: Ethan Li
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For example,
> {code:java}
> bin/storm jar storm-loadgen-*.jar
> org.apache.storm.loadgen.ThroughputVsLatency --spouts 1 --splitters 2
> --counters 1 -c topology.debug=true
> {code}
> on ResourceAwareScheduler not working properly.
> With default cluster settings, there will be only one __acker-executor and it
> will be on a separate worker. And it looks like the __acker-executor was not
> able to receive messages from spouts and bolts. And spouts and bolts
> continued to retry sending messages to acker. It then led to another problem:
> STORM-2970
> I tried to run on storm right before
> [https://github.com/apache/storm/pull/2502] and right after and confirmed
> that this bug should be related to it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (STORM-2983) Some topologies not working properly
[
https://issues.apache.org/jira/browse/STORM-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421019#comment-16421019
]
Roshan Naik edited comment on STORM-2983 at 3/30/18 10:26 PM:
--
Removing / preserving the optimization will not address the core problem ...
which is .. topology.workers is not usable and therefore any code
(present/future) would be considered broken with no workaround.
There are other use cases in code for this setting plus more that I can think
of, but to limit the scope I will stay with the topic of some settings in
topoConf being unreliable. I dont think its a good idea to say any such code
should not allowed in storm.
Since topoConf is part of the API ... it is surprising to know that any
setting stored in it cannot be relied upon. The
[documentation|http://storm.apache.org/releases/1.2.1/Configuration.html]
specifically states that the system supports overriding settings. *The actual
bug here is that we are not dynamically overriding the setting to what it
should be.*
I think we need to have a good justification for making any setting unreliable
and not open the door for more of the same.
Would like to know:
- Is there good reason why this setting cannot be made reliable ?
- Are you aware of other such settings that are not reflecting the correct
values ?
was (Author: roshan_naik):
Removing / preserving the optimization will not address the core problem ...
which is .. topology.worker.count is not usable and therefore any code
(present/future) would be considered broken with no workaround.
There are other use cases in code for this setting plus more that I can think
of, but to limit the scope I will stay with the topic of some settings in
topoConf being unreliable. I dont think its a good idea to say any such code
should not allowed in storm.
Since topoConf is part of the API ... it is surprising to know that any
setting stored in it cannot be relied upon. The
[documentation|http://storm.apache.org/releases/1.2.1/Configuration.html]
specifically states that the system supports overriding settings. *The actual
bug here is that we are not dynamically overriding the setting to what it
should be.*
I think we need to have a good justification for making any setting unreliable
and not open the door for more of the same.
Would like to know:
- Is there good reason why this setting cannot be made reliable ?
- Are you aware of other such settings that are not reflecting the correct
values ?
> Some topologies not working properly
> -
>
> Key: STORM-2983
> URL: https://issues.apache.org/jira/browse/STORM-2983
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Ethan Li
>Assignee: Ethan Li
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For example,
> {code:java}
> bin/storm jar storm-loadgen-*.jar
> org.apache.storm.loadgen.ThroughputVsLatency --spouts 1 --splitters 2
> --counters 1 -c topology.debug=true
> {code}
> on ResourceAwareScheduler not working properly.
> With default cluster settings, there will be only one __acker-executor and it
> will be on a separate worker. And it looks like the __acker-executor was not
> able to receive messages from spouts and bolts. And spouts and bolts
> continued to retry sending messages to acker. It then led to another problem:
> STORM-2970
> I tried to run on storm right before
> [https://github.com/apache/storm/pull/2502] and right after and confirmed
> that this bug should be related to it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (STORM-2983) Some topologies not working properly
[
https://issues.apache.org/jira/browse/STORM-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421019#comment-16421019
]
Roshan Naik edited comment on STORM-2983 at 3/30/18 10:25 PM:
--
Removing / preserving the optimization will not address the core problem ...
which is .. topology.worker.count is not usable and therefore any code
(present/future) would be considered broken with no workaround.
There are other use cases in code for this setting plus more that I can think
of, but to limit the scope I will stay with the topic of some settings in
topoConf being unreliable. I dont think its a good idea to say any such code
should not allowed in storm.
Since topoConf is part of the API ... it is surprising to know that any
setting stored in it cannot be relied upon. The
[documentation|http://storm.apache.org/releases/1.2.1/Configuration.html]
specifically states that the system supports overriding settings. *The actual
bug here is that we are not dynamically overriding the setting to what it
should be.*
I think we need to have a good justification for making any setting unreliable
and not open the door for more of the same.
Would like to know:
- Is there good reason why this setting cannot be made reliable ?
- Are you aware of other such settings that are not reflecting the correct
values ?
was (Author: roshan_naik):
Removing / preserving the optimization will not address the core problem ...
which is .. topology.worker.count is not usable and therefore any code
(present/future) would be considered broken with no workaround. There are other
use cases in code for this setting plus more that I can think of, but to limit
the scope I will stay with the topic of some settings in topoConf being
unreliable. I dont think its a good idea to say any such code should not
allowed in storm.
Since topoConf is part of the API ... it is surprising to know that any
setting stored in it cannot be relied upon. The
[documentation|http://storm.apache.org/releases/1.2.1/Configuration.html]
specifically states that the system supports overriding settings. *The actual
bug here is that we are not dynamically overriding the setting to what it
should be.*
I think we need to have a good justification for making any setting unreliable
and not open the door for more of the same.
Would like to know:
- Is there good reason why this setting cannot be made reliable ?
- Are you aware of other such settings that are not reflecting the correct
values ?
> Some topologies not working properly
> -
>
> Key: STORM-2983
> URL: https://issues.apache.org/jira/browse/STORM-2983
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Ethan Li
>Assignee: Ethan Li
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For example,
> {code:java}
> bin/storm jar storm-loadgen-*.jar
> org.apache.storm.loadgen.ThroughputVsLatency --spouts 1 --splitters 2
> --counters 1 -c topology.debug=true
> {code}
> on ResourceAwareScheduler not working properly.
> With default cluster settings, there will be only one __acker-executor and it
> will be on a separate worker. And it looks like the __acker-executor was not
> able to receive messages from spouts and bolts. And spouts and bolts
> continued to retry sending messages to acker. It then led to another problem:
> STORM-2970
> I tried to run on storm right before
> [https://github.com/apache/storm/pull/2502] and right after and confirmed
> that this bug should be related to it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (STORM-2983) Some topologies not working properly
[
https://issues.apache.org/jira/browse/STORM-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421019#comment-16421019
]
Roshan Naik commented on STORM-2983:
Removing / preserving the optimization will not address the core problem ...
which is .. topology.worker.count is not usable and therefore any code
(present/future) would be considered broken with no workaround. There are other
use cases in code for this setting plus more that I can think of, but to limit
the scope I will stay with the topic of some settings in topoConf being
unreliable. I dont think its a good idea to say any such code should not
allowed in storm.
Since topoConf is part of the API ... it is surprising to know that any
setting stored in it cannot be relied upon. The
[documentation|http://storm.apache.org/releases/1.2.1/Configuration.html]
specifically states that the system supports overriding settings. *The actual
bug here is that we are not dynamically overriding the setting to what it
should be.*
I think we need to have a good justification for making any setting unreliable
and not open the door for more of the same.
Would like to know:
- Is there good reason why this setting cannot be made reliable ?
- Are you aware of other such settings that are not reflecting the correct
values ?
> Some topologies not working properly
> -
>
> Key: STORM-2983
> URL: https://issues.apache.org/jira/browse/STORM-2983
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Ethan Li
>Assignee: Ethan Li
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For example,
> {code:java}
> bin/storm jar storm-loadgen-*.jar
> org.apache.storm.loadgen.ThroughputVsLatency --spouts 1 --splitters 2
> --counters 1 -c topology.debug=true
> {code}
> on ResourceAwareScheduler not working properly.
> With default cluster settings, there will be only one __acker-executor and it
> will be on a separate worker. And it looks like the __acker-executor was not
> able to receive messages from spouts and bolts. And spouts and bolts
> continued to retry sending messages to acker. It then led to another problem:
> STORM-2970
> I tried to run on storm right before
> [https://github.com/apache/storm/pull/2502] and right after and confirmed
> that this bug should be related to it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (STORM-3005) [DRPC] LinearDRPCTopologyBuilder shouldn't be deprecated
[ https://issues.apache.org/jira/browse/STORM-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420971#comment-16420971 ] Lukasz Gebel commented on STORM-3005: - Hi [~revans2], Thanks, I added PRs for 1.x, 1.1.x and 1.0.x branches. Cheers, Luk. > [DRPC] LinearDRPCTopologyBuilder shouldn't be deprecated > - > > Key: STORM-3005 > URL: https://issues.apache.org/jira/browse/STORM-3005 > Project: Apache Storm > Issue Type: Task >Affects Versions: 1.1.1, 1.2.1 >Reporter: Lukasz Gebel >Assignee: Lukasz Gebel >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Apache Storm provides DRPC functionality. However LinearDRPCTopologyBuilder > is deprecated due to: "Trident subsumes the functionality provided by this > class, so it's deprecated". I think it shouldn't be so, because you may still > want to use DRPC without Trident. > LinearDRPCTopologyBuilder is also mentioned as a part of example in current > documentation: [http://storm.apache.org/releases/1.2.1/Distributed-RPC.html] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3007) Fix checkstyle warnings in storm-mqtt-examples
[ https://issues.apache.org/jira/browse/STORM-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-3007. Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~krichter], I merged this into master. keep up the good work. > Fix checkstyle warnings in storm-mqtt-examples > -- > > Key: STORM-3007 > URL: https://issues.apache.org/jira/browse/STORM-3007 > Project: Apache Storm > Issue Type: Task >Reporter: Karl Richter >Assignee: Karl Richter >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Fixing checkstyle warnings in `storm-mqtt-examples` allows to evaluate the > workload of being able to set the number of allowed checkstyle warnings > (which is an inconvenient workaround) to 0 one day. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (STORM-3007) Fix checkstyle warnings in storm-mqtt-examples
[ https://issues.apache.org/jira/browse/STORM-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans reassigned STORM-3007: -- Assignee: Karl Richter > Fix checkstyle warnings in storm-mqtt-examples > -- > > Key: STORM-3007 > URL: https://issues.apache.org/jira/browse/STORM-3007 > Project: Apache Storm > Issue Type: Task >Reporter: Karl Richter >Assignee: Karl Richter >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Fixing checkstyle warnings in `storm-mqtt-examples` allows to evaluate the > workload of being able to set the number of allowed checkstyle warnings > (which is an inconvenient workaround) to 0 one day. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2997) Add logviewer ssl module in SECURITY.md
[ https://issues.apache.org/jira/browse/STORM-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-2997. Resolution: Fixed Fix Version/s: 1.2.2 2.0.0 Thanks [~liuzhaokun], I merged this into master and 1.x > Add logviewer ssl module in SECURITY.md > > > Key: STORM-2997 > URL: https://issues.apache.org/jira/browse/STORM-2997 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Affects Versions: 1.2.1 >Reporter: liuzhaokun >Assignee: liuzhaokun >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0, 1.2.2 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Add logviewer ssl module in SECURITY.md like UI and DRPC -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2977) Update the BlowfishTupleSerializer to make it simpler to use
[ https://issues.apache.org/jira/browse/STORM-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-2977. Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~ethanli], I merged this into master. > Update the BlowfishTupleSerializer to make it simpler to use > > > Key: STORM-2977 > URL: https://issues.apache.org/jira/browse/STORM-2977 > Project: Apache Storm > Issue Type: Improvement >Reporter: Ethan Li >Assignee: Ethan Li >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > BlowfishTupleSerializer currently goes off of a secret key that you pass in > through the config, and it is a hex encoded ASCII string. We really should > make it simpler to use and if the config is there, then we use it, if not we > use the zookeeper secret if it is there, and only blow up if neither is > present. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (STORM-3006) Distributed RPC documentation needs an update
[
https://issues.apache.org/jira/browse/STORM-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans reassigned STORM-3006:
--
Assignee: Lukasz Gebel
> Distributed RPC documentation needs an update
> -
>
> Key: STORM-3006
> URL: https://issues.apache.org/jira/browse/STORM-3006
> Project: Apache Storm
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 1.2.1
>Reporter: Lukasz Gebel
>Assignee: Lukasz Gebel
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0, 1.2.2
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently in: http://storm.apache.org/releases/1.2.1/Distributed-RPC.html
> examples of DRPCClient initialization are outdated as you need to provide
> config parameter:
> Config conf = new Config();
> conf.setDebug(false);
> conf.put("storm.thrift.transport",
> "org.apache.storm.security.auth.plain.PlainSaslTransportPlugin");
> conf.put(Config.STORM_NIMBUS_RETRY_TIMES, 3);
> conf.put(Config.STORM_NIMBUS_RETRY_INTERVAL, 10);
> conf.put(Config.STORM_NIMBUS_RETRY_INTERVAL_CEILING, 20);
> this.drpcClient = new DRPCClient(conf, "10.0.9.10", 3772);
> What it more it would be useful to state that
> org.apache.storm.security.auth.plain.PlainSaslTransportPlugin can be used in
> DRPC as SimpleTransportPlugin is deprecated.
> So also storm.yaml example can be extended:
> drpc.servers:
> - "drpc1.foo.com"
> - "drpc2.foo.com"
> *storm.thrift.transport:
> "org.apache.storm.security.auth.plain.PlainSaslTransportPlugin"*
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-3006) Distributed RPC documentation needs an update
[
https://issues.apache.org/jira/browse/STORM-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans resolved STORM-3006.
Resolution: Fixed
Fix Version/s: 1.2.2
2.0.0
Thanks [~rauluka],
I merged this into master and branch-1.x
> Distributed RPC documentation needs an update
> -
>
> Key: STORM-3006
> URL: https://issues.apache.org/jira/browse/STORM-3006
> Project: Apache Storm
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 1.2.1
>Reporter: Lukasz Gebel
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0, 1.2.2
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently in: http://storm.apache.org/releases/1.2.1/Distributed-RPC.html
> examples of DRPCClient initialization are outdated as you need to provide
> config parameter:
> Config conf = new Config();
> conf.setDebug(false);
> conf.put("storm.thrift.transport",
> "org.apache.storm.security.auth.plain.PlainSaslTransportPlugin");
> conf.put(Config.STORM_NIMBUS_RETRY_TIMES, 3);
> conf.put(Config.STORM_NIMBUS_RETRY_INTERVAL, 10);
> conf.put(Config.STORM_NIMBUS_RETRY_INTERVAL_CEILING, 20);
> this.drpcClient = new DRPCClient(conf, "10.0.9.10", 3772);
> What it more it would be useful to state that
> org.apache.storm.security.auth.plain.PlainSaslTransportPlugin can be used in
> DRPC as SimpleTransportPlugin is deprecated.
> So also storm.yaml example can be extended:
> drpc.servers:
> - "drpc1.foo.com"
> - "drpc2.foo.com"
> *storm.thrift.transport:
> "org.apache.storm.security.auth.plain.PlainSaslTransportPlugin"*
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (STORM-3013) Deactivated topology restarts if data flows into Kafka
[
https://issues.apache.org/jira/browse/STORM-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420792#comment-16420792
]
Stig Rohde Døssing commented on STORM-3013:
---
Just to clarify, the only changes are in storm-kafka-client, so you only need
to build that part of Storm.
> Deactivated topology restarts if data flows into Kafka
> --
>
> Key: STORM-3013
> URL: https://issues.apache.org/jira/browse/STORM-3013
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
>Affects Versions: 2.0.0, 1.2.1
>Reporter: Ajeesh B
>Assignee: Stig Rohde Døssing
>Priority: Major
>
> Hi, I have deactivated the storm topology & then if I produce any records
> into Kafka, Storm throws an exception. Exception follows,
> {code:java}
> 2018-03-28 09:50:23.804 o.a.s.d.executor Thread-83-kafkaLogs-executor[130
> 130] [INFO] Deactivating spout kafkaLogs:(130)
> 2018-03-28 09:51:01.289 o.a.s.util Thread-17-kafkaLogs-executor[139 139]
> [ERROR] Async loop died!
> java.lang.RuntimeException: java.lang.IllegalStateException: This consumer
> has already been closed.
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:522)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:487)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:477)
> ~[storm-core-1.2.1.jar:1.2.1]
> at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:70)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$fn__4975$fn__4990$fn__5021.invoke(executor.clj:634)
> ~[storm-core-1.2.1.jar:1.2.1]
> at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484)
> [storm-core-1.2.1.jar:1.2.1]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
> Caused by: java.lang.IllegalStateException: This consumer has already been
> closed.
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.acquireAndEnsureOpen(KafkaConsumer.java:1787)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.beginningOffsets(KafkaConsumer.java:1622)
> ~[stormjar.jar:?]
> at
> org.apache.storm.kafka.spout.metrics.KafkaOffsetMetric.getValueAndReset(KafkaOffsetMetric.java:79)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$metrics_tick$fn__4899.invoke(executor.clj:345)
> ~[storm-core-1.2.1.jar:1.2.1]
> at clojure.core$map$fn__4553.invoke(core.clj:2622) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
> at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?]
> at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?]
> at clojure.core$filter$fn__4580.invoke(core.clj:2679) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
> at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.7.0.jar:?]
> at clojure.lang.RT.next(RT.java:674) ~[clojure-1.7.0.jar:?]
> at clojure.core$next__4112.invoke(core.clj:64) ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6523.invoke(protocols.clj:170)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6478$G__6473__6487.invoke(protocols.clj:19)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6506.invoke(protocols.clj:101)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13)
> ~[clojure-1.7.0.jar:?]
> at clojure.core$reduce.invoke(core.clj:6519) ~[clojure-1.7.0.jar:?]
> at clojure.core$into.invoke(core.clj:6600) ~[clojure-1.7.0.jar:?]
> at org.apache.storm.daemon.executor$metrics_tick.invoke(executor.clj:349)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$fn__4975$tuple_action_fn__4981.invoke(executor.clj:522)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__4964.invoke(executor.clj:471)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4475.onEvent(disruptor.clj:41)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:509)
> ~[storm-core-1.2.1.jar:1.2.1]
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (STORM-3013) Deactivated topology restarts if data flows into Kafka
[
https://issues.apache.org/jira/browse/STORM-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420744#comment-16420744
]
Stig Rohde Døssing commented on STORM-3013:
---
[~Ajeesh] I've fixed the metrics issue. Could you try it out and see if it
resolves the problems you're having? The code is available at
https://github.com/srdo/storm/tree/STORM-3013-1.x.
> Deactivated topology restarts if data flows into Kafka
> --
>
> Key: STORM-3013
> URL: https://issues.apache.org/jira/browse/STORM-3013
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
>Affects Versions: 2.0.0, 1.2.1
>Reporter: Ajeesh B
>Assignee: Stig Rohde Døssing
>Priority: Major
>
> Hi, I have deactivated the storm topology & then if I produce any records
> into Kafka, Storm throws an exception. Exception follows,
> {code:java}
> 2018-03-28 09:50:23.804 o.a.s.d.executor Thread-83-kafkaLogs-executor[130
> 130] [INFO] Deactivating spout kafkaLogs:(130)
> 2018-03-28 09:51:01.289 o.a.s.util Thread-17-kafkaLogs-executor[139 139]
> [ERROR] Async loop died!
> java.lang.RuntimeException: java.lang.IllegalStateException: This consumer
> has already been closed.
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:522)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:487)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:477)
> ~[storm-core-1.2.1.jar:1.2.1]
> at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:70)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$fn__4975$fn__4990$fn__5021.invoke(executor.clj:634)
> ~[storm-core-1.2.1.jar:1.2.1]
> at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484)
> [storm-core-1.2.1.jar:1.2.1]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
> Caused by: java.lang.IllegalStateException: This consumer has already been
> closed.
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.acquireAndEnsureOpen(KafkaConsumer.java:1787)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.beginningOffsets(KafkaConsumer.java:1622)
> ~[stormjar.jar:?]
> at
> org.apache.storm.kafka.spout.metrics.KafkaOffsetMetric.getValueAndReset(KafkaOffsetMetric.java:79)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$metrics_tick$fn__4899.invoke(executor.clj:345)
> ~[storm-core-1.2.1.jar:1.2.1]
> at clojure.core$map$fn__4553.invoke(core.clj:2622) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
> at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?]
> at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?]
> at clojure.core$filter$fn__4580.invoke(core.clj:2679) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
> at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.7.0.jar:?]
> at clojure.lang.RT.next(RT.java:674) ~[clojure-1.7.0.jar:?]
> at clojure.core$next__4112.invoke(core.clj:64) ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6523.invoke(protocols.clj:170)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6478$G__6473__6487.invoke(protocols.clj:19)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6506.invoke(protocols.clj:101)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13)
> ~[clojure-1.7.0.jar:?]
> at clojure.core$reduce.invoke(core.clj:6519) ~[clojure-1.7.0.jar:?]
> at clojure.core$into.invoke(core.clj:6600) ~[clojure-1.7.0.jar:?]
> at org.apache.storm.daemon.executor$metrics_tick.invoke(executor.clj:349)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$fn__4975$tuple_action_fn__4981.invoke(executor.clj:522)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__4964.invoke(executor.clj:471)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4475.onEvent(disruptor.clj:41)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:509)
> ~[storm-core-1.2.1.jar:1.2.1]
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (STORM-3005) [DRPC] LinearDRPCTopologyBuilder shouldn't be deprecated
[ https://issues.apache.org/jira/browse/STORM-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans reassigned STORM-3005: -- Assignee: Lukasz Gebel > [DRPC] LinearDRPCTopologyBuilder shouldn't be deprecated > - > > Key: STORM-3005 > URL: https://issues.apache.org/jira/browse/STORM-3005 > Project: Apache Storm > Issue Type: Task >Affects Versions: 1.1.1, 1.2.1 >Reporter: Lukasz Gebel >Assignee: Lukasz Gebel >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Apache Storm provides DRPC functionality. However LinearDRPCTopologyBuilder > is deprecated due to: "Trident subsumes the functionality provided by this > class, so it's deprecated". I think it shouldn't be so, because you may still > want to use DRPC without Trident. > LinearDRPCTopologyBuilder is also mentioned as a part of example in current > documentation: [http://storm.apache.org/releases/1.2.1/Distributed-RPC.html] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3012) Nimbus will crash if pacemaker is restarted
[
https://issues.apache.org/jira/browse/STORM-3012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans resolved STORM-3012.
Resolution: Fixed
Fix Version/s: 2.0.0
Thanks [~ethanli],
I merged this into master.
> Nimbus will crash if pacemaker is restarted
> ---
>
> Key: STORM-3012
> URL: https://issues.apache.org/jira/browse/STORM-3012
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Ethan Li
>Assignee: Ethan Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Below is the nimbus.log when I restarted pacemaker. Nimbus crashed because of
> NPE.
>
>
> {code:java}
> 2018-03-26 21:39:18.404 main o.a.s.z.LeaderElectorImp [INFO] Queued up for
> leader lock.
> 2018-03-26 21:39:18.458 main o.a.s.d.m.MetricsUtils [INFO] Using statistics
> reporter plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableRepor
> ter
> 2018-03-26 21:39:18.461 main o.a.s.d.m.r.JmxPreparableReporter [INFO]
> Preparing...
> 2018-03-26 21:39:18.527 main o.a.s.m.StormMetricsRegistry [INFO] Started
> statistics report plugin...
> 2018-03-26 21:39:18.710 main o.a.s.m.n.Login [INFO] successfully logged in.
> 2018-03-26 21:39:18.738 Refresh-TGT o.a.s.m.n.Login [INFO] TGT refresh thread
> started.
> 2018-03-26 21:39:18.739 main o.a.s.z.ClientZookeeper [INFO] Staring ZK Curator
> 2018-03-26 21:39:18.739 main o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
> 2018-03-26 21:39:18.747 Refresh-TGT o.a.s.m.n.Login [INFO] TGT valid starting
> at:Mon Mar 26 21:39:18 UTC 2018
> 2018-03-26 21:39:18.747 Refresh-TGT o.a.s.m.n.Login [INFO] TGT expires:
>Tue Mar 27 21:39:18 UTC 2018
> 2018-03-26 21:39:18.747 Refresh-TGT o.a.s.m.n.Login [INFO] TGT refresh
> sleeping until: Tue Mar 27 17:39:22 UTC 2018
> 2018-03-26 21:39:18.756 main o.a.z.ZooKeeper [INFO] Initiating client
> connection, connectString=openqe74blue-gw.blue.ygrid.yahoo.com:2181
> sessionTimeout
> =6 watcher=org.apache.curator.ConnectionState@148c7c4b
> 2018-03-26 21:39:18.807 main o.a.c.f.i.CuratorFrameworkImpl [INFO] Default
> schema
> 2018-03-26 21:39:18.814
> main-SendThread(openqe74blue-gw.blue.ygrid.yahoo.com:2181)
> o.a.z.c.ZooKeeperSaslClient [INFO] Client will use GSSAPI as SASL mec
> hanism.
> 2018-03-26 21:39:18.815
> main-SendThread(openqe74blue-gw.blue.ygrid.yahoo.com:2181) o.a.z.ClientCnxn
> [INFO] Opening socket connection to server openqe74b
> lue-gw.blue.ygrid.yahoo.com/10.215.68.156:2181. Will attempt to
> SASL-authenticate using Login Context section 'Client'
> 2018-03-26 21:39:18.816
> main-SendThread(openqe74blue-gw.blue.ygrid.yahoo.com:2181) o.a.z.ClientCnxn
> [INFO] Socket connection established to openqe74blue
> -gw.blue.ygrid.yahoo.com/10.215.68.156:2181, initiating session
> 2018-03-26 21:39:18.817
> main-SendThread(openqe74blue-gw.blue.ygrid.yahoo.com:2181) o.a.z.ClientCnxn
> [INFO] Session establishment complete on server open
> qe74blue-gw.blue.ygrid.yahoo.com/10.215.68.156:2181, sessionid =
> 0x1624f6d49dd0cdd, negotiated timeout = 4
> 2018-03-26 21:39:18.818 main-EventThread o.a.c.f.s.ConnectionStateManager
> [INFO] State change: CONNECTED
> 2018-03-26 21:39:18.839 Curator-Framework-0 o.a.c.f.i.CuratorFrameworkImpl
> [INFO] backgroundOperationsLoop exiting
> 2018-03-26 21:39:18.841 main o.a.z.ZooKeeper [INFO] Session:
> 0x1624f6d49dd0cdd closed
> 2018-03-26 21:39:18.842 main-EventThread o.a.z.ClientCnxn [INFO] EventThread
> shut down
> 2018-03-26 21:39:18.844 main o.a.s.z.ClientZookeeper [INFO] Staring ZK Curator
> 2018-03-26 21:39:18.844 main o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
> 2018-03-26 21:39:18.875 main o.a.z.ZooKeeper [INFO] Initiating client
> connection,
> connectString=openqe74blue-gw.blue.ygrid.yahoo.com:2181/storm_ystormQE
> _CI sessionTimeout=6 watcher=org.apache.curator.ConnectionState@211febf3
> 2018-03-26 21:39:18.908
> main-SendThread(openqe74blue-gw.blue.ygrid.yahoo.com:2181)
> o.a.z.c.ZooKeeperSaslClient [INFO] Client will use GSSAPI as SASL mec
> hanism.
> 2018-03-26 21:39:18.909
> main-SendThread(openqe74blue-gw.blue.ygrid.yahoo.com:2181) o.a.z.ClientCnxn
> [INFO] Opening socket connection to server openqe74b
> lue-gw.blue.ygrid.yahoo.com/10.215.68.156:2181. Will attempt to
> SASL-authenticate using Login Context section 'Client'
> 2018-03-26 21:39:18.910
> main-SendThread(openqe74blue-gw.blue.ygrid.yahoo.com:2181) o.a.z.ClientCnxn
> [INFO] Socket connection established to openqe74blue
> -gw.blue.ygrid.yahoo.com/10.215.68.156:2181, initiating session
> 2018-03-26 21:39:18.911
> main-SendThread(openqe74blue-gw.blue.ygrid.yahoo.com:2181) o.a.z.ClientCnxn
> [INFO] Session establishment complete on server open
>
[jira] [Resolved] (STORM-2993) Storm HDFS bolt throws ClosedChannelException when Time rotation policy is used
[ https://issues.apache.org/jira/browse/STORM-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-2993. Resolution: Fixed Fix Version/s: 1.2.2 2.0.0 Thanks [~arunmahadevan], I merged this into master and branch-1.x. there were merge conflicts, but appeared to be minor ones, for 1.1.x so I didn't pull it in there. If you want it pulled back please file another pull request for that. > Storm HDFS bolt throws ClosedChannelException when Time rotation policy is > used > --- > > Key: STORM-2993 > URL: https://issues.apache.org/jira/browse/STORM-2993 > Project: Apache Storm > Issue Type: Bug >Reporter: Arun Mahadevan >Assignee: Arun Mahadevan >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0, 1.2.2 > > Time Spent: 1h > Remaining Estimate: 0h > > Storm connector throws below error in the worker logs. > > 2018-03-12 18:14:58.123 o.a.s.h.c.r.MoveFileAction Timer-3 [INFO] Moving file > hdfs://ctr-e138-1518143905142-85179-01-04.hwx.site:8020/tmp/foo/my-bolt-3-0-1520878438104.txt > to /tmp/dest2/my-bolt-3-0-15 20878438104.txt 2018-03-12 18:14:58.123 > o.a.s.h.c.r.MoveFileAction Timer-0 [INFO] Moving file > hdfs://ctr-e138-1518143905142-85179-01-04.hwx.site:8020/tmp/foo/my-bolt-6-0-1520878438104.txt > to /tmp/dest2/my-bolt-6-0-15 20878438104.txt 2018-03-12 18:14:58.123 > o.a.s.h.c.r.MoveFileAction Timer-1 [INFO] Moving file > hdfs://ctr-e138-1518143905142-85179-01-04.hwx.site:8020/tmp/foo/my-bolt-5-0-1520878438104.txt > to /tmp/dest2/my-bolt-5-0-15 20878438104.txt 2018-03-12 18:14:58.124 > o.a.s.h.c.r.MoveFileAction Timer-2 [INFO] Moving file > hdfs://ctr-e138-1518143905142-85179-01-04.hwx.site:8020/tmp/foo/my-bolt-4-0-1520878438104.txt > to /tmp/dest2/my-bolt-4-0-15 20878438104.txt 2018-03-12 18:14:58.132 > o.a.s.h.b.AbstractHdfsBolt Timer-2 [INFO] File rotation took 28 ms. > 2018-03-12 18:14:58.132 o.a.s.h.b.AbstractHdfsBolt Timer-0 [INFO] File > rotation took 29 ms. 2018-03-12 18:14:58.132 o.a.s.h.b.AbstractHdfsBolt > Timer-3 [INFO] File rotation took 28 ms. 2018-03-12 18:14:58.132 > o.a.s.h.b.AbstractHdfsBolt Timer-1 [INFO] File rotation took 28 ms. > 2018-03-12 18:14:58.132 o.a.s.h.b.AbstractHdfsBolt > Thread-12-my-bolt-executor[6 6] [INFO] Tuple failed to write, forcing a flush > of existing data. 2018-03-12 18:14:58.132 o.a.s.h.b.AbstractHdfsBolt > Thread-8-my-bolt-executor[3 3] [INFO] Tuple failed to write, forcing a flush > of existing data. 2018-03-12 18:14:58.132 o.a.s.h.b.AbstractHdfsBolt > Thread-16-my-bolt-executor[5 5] [INFO] Tuple failed to write, forcing a flush > of existing data. 2018-03-12 18:14:58.132 o.a.s.d.executor > Thread-8-my-bolt-executor[3 3] [ERROR] > java.nio.channels.ClosedChannelException: null at > org.apache.hadoop.hdfs.ExceptionLastSeen.throwException4Close(ExceptionLastSeen.java:73) > ~[stormjar.jar:?] at > org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:153) > ~[stormjar.jar:?] at > org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:105) > ~[stormjar.jar:?] at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:57) > ~[stormjar.jar:?] at > java.io.DataOutputStream.write(DataOutputStream.java:107) ~[?:1.8.0_161] at > java.io.FilterOutputStream.write(FilterOutputStream.java:97) ~[?:1.8.0_161] > at org.apache.storm.hdfs.common.HDFSWriter.doWrite(HDFSWriter.java:48) > ~[stormjar.jar:?] at > org.apache.storm.hdfs.common.AbstractHDFSWriter.write(AbstractHDFSWriter.java:40) > ~[stormjar.jar:?] at > org.apache.storm.hdfs.bolt.AbstractHdfsBolt.execute(AbstractHdfsBolt.java:158) > [stormjar.jar:?] at > org.apache.storm.daemon.executor$fn__10189$tuple_action_fn__10191.invoke(executor.clj:745) > [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at > org.apache.storm.daemon.executor$mk_task_receiver$fn__10108.invoke(executor.clj:473) > [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at > org.apache.storm.disruptor$clojure_handler$reify__4115.onEvent(disruptor.clj:41) > [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:509) > [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at > org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:487) > [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at > org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:74) > [storm-core-1.2.1.3.0.0.0-1013.jar:1.2.1.3.0.0.0-1013] at > org.apache.storm.daemon.executor$fn__10189$fn__10202$fn__10257.invoke(executor.clj:868) >
[jira] [Created] (STORM-3017) Refactor pacemaker code
Ethan Li created STORM-3017: --- Summary: Refactor pacemaker code Key: STORM-3017 URL: https://issues.apache.org/jira/browse/STORM-3017 Project: Apache Storm Issue Type: Improvement Reporter: Ethan Li Assignee: Ethan Li As [~Srdo] pointed out in [https://github.com/apache/storm/pull/2587] and [~revans2] pointed out in [https://github.com/apache/storm/pull/2608] , there are some pacemaker code we need to revisit and refactor especially for exception handling and retrying. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3011) Use default bin path in flight.bash if $JAVA_HOME is undefined
[ https://issues.apache.org/jira/browse/STORM-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-3011. Resolution: Fixed Fix Version/s: 1.2.2 1.0.7 1.1.3 2.0.0 Thanks [~jnioche], I merged this into the master, 1.x, 1.1.x, and 1.0.x branches. > Use default bin path in flight.bash if $JAVA_HOME is undefined > -- > > Key: STORM-3011 > URL: https://issues.apache.org/jira/browse/STORM-3011 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 1.2.1 >Reporter: Julien Nioche >Assignee: Julien Nioche >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0, 1.1.3, 1.0.7, 1.2.2 > > Time Spent: 50m > Remaining Estimate: 0h > > I had to modify the _flight.bash_ script as $JAVA_HOME is undefined, this > yields > > _/usr/share/apache-storm-1.2.1/bin/flight.bash: line 62: /bin/jstack: No such > file or directory_ > > however, the files can be found in the default path _/usr/bin_ > > I will submit a PR so that the script checks that $JDKPATH is not empty and > exist before using it for BINPATH. I can't see any downsides in doing so. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (STORM-3011) Use default bin path in flight.bash if $JAVA_HOME is undefined
[ https://issues.apache.org/jira/browse/STORM-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans reassigned STORM-3011: -- Assignee: Julien Nioche > Use default bin path in flight.bash if $JAVA_HOME is undefined > -- > > Key: STORM-3011 > URL: https://issues.apache.org/jira/browse/STORM-3011 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 1.2.1 >Reporter: Julien Nioche >Assignee: Julien Nioche >Priority: Minor > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > I had to modify the _flight.bash_ script as $JAVA_HOME is undefined, this > yields > > _/usr/share/apache-storm-1.2.1/bin/flight.bash: line 62: /bin/jstack: No such > file or directory_ > > however, the files can be found in the default path _/usr/bin_ > > I will submit a PR so that the script checks that $JDKPATH is not empty and > exist before using it for BINPATH. I can't see any downsides in doing so. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2924) testTickTupleWorksWithSystemBolt in org.apache.storm.TickTupleTest fails intermittently
[ https://issues.apache.org/jira/browse/STORM-2924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-2924. Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~kabhwan], I merged this into master. > testTickTupleWorksWithSystemBolt in org.apache.storm.TickTupleTest fails > intermittently > --- > > Key: STORM-2924 > URL: https://issues.apache.org/jira/browse/STORM-2924 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-client >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > java.lang.AssertionError: took over 11 ms of simulated time to get a > message back... > [https://travis-ci.org/apache/storm/jobs/335075657] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2983) Some topologies not working properly
[
https://issues.apache.org/jira/browse/STORM-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420512#comment-16420512
]
Ethan Li commented on STORM-2983:
-
[~roshan_naik]
[topologyConf|https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/worker/Worker.java#L143]
in the code is read from supervisor which is the stormconf.ser from user's
submission. Some of the configs in user's submission serves as a hint.
topologyConf here won't reflect any dynamically calculated values. We might
need to check the usage of topologyConf and make sure it's not used for some
dynamically calculated values like this. As for this particular issue, I agree
with [~kabhwan] that the optimization for singe worker case is not really an
issue.
> Some topologies not working properly
> -
>
> Key: STORM-2983
> URL: https://issues.apache.org/jira/browse/STORM-2983
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Ethan Li
>Assignee: Ethan Li
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For example,
> {code:java}
> bin/storm jar storm-loadgen-*.jar
> org.apache.storm.loadgen.ThroughputVsLatency --spouts 1 --splitters 2
> --counters 1 -c topology.debug=true
> {code}
> on ResourceAwareScheduler not working properly.
> With default cluster settings, there will be only one __acker-executor and it
> will be on a separate worker. And it looks like the __acker-executor was not
> able to receive messages from spouts and bolts. And spouts and bolts
> continued to retry sending messages to acker. It then led to another problem:
> STORM-2970
> I tried to run on storm right before
> [https://github.com/apache/storm/pull/2502] and right after and confirmed
> that this bug should be related to it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (STORM-2983) Some topologies not working properly
[
https://issues.apache.org/jira/browse/STORM-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420273#comment-16420273
]
Roshan Naik edited comment on STORM-2983 at 3/30/18 8:20 AM:
-
[~kabhwan]
IMO this should not be viewed as a sidestepping an minor optimization. In which
case it does look like a minor issue. The optimization only exposed a bug. So
it is only the "messenger".
The key issue here is supporting (internal or user code) to check the state
correctly .. to do whatever it needs to do optimization or behavior
change.
By not fixing the core bug, we wlll sidestep the core issue. IMO this a
critical bug and needs to fixed. Or we need to provide a diff mechanism for
code to make such checks correctly.
was (Author: roshan_naik):
[~kabhwan]
IMO this should not be viewed as a sidestepping an optimization. In which case
it does look like a minor issue. The optimization only exposed a bug. So it is
only the "messenger".
The key issue here is supporting (internal or user code) to check the state
correctly .. to do whatever it needs to do optimization or behavior
change.
By not fixing the core bug, we wlll sidestep the core issue. IMO this a
critical bug and needs to fixed. Or we need to provide a diff mechanism for
code to make such checks correctly.
> Some topologies not working properly
> -
>
> Key: STORM-2983
> URL: https://issues.apache.org/jira/browse/STORM-2983
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Ethan Li
>Assignee: Ethan Li
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For example,
> {code:java}
> bin/storm jar storm-loadgen-*.jar
> org.apache.storm.loadgen.ThroughputVsLatency --spouts 1 --splitters 2
> --counters 1 -c topology.debug=true
> {code}
> on ResourceAwareScheduler not working properly.
> With default cluster settings, there will be only one __acker-executor and it
> will be on a separate worker. And it looks like the __acker-executor was not
> able to receive messages from spouts and bolts. And spouts and bolts
> continued to retry sending messages to acker. It then led to another problem:
> STORM-2970
> I tried to run on storm right before
> [https://github.com/apache/storm/pull/2502] and right after and confirmed
> that this bug should be related to it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (STORM-2983) Some topologies not working properly
[
https://issues.apache.org/jira/browse/STORM-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420273#comment-16420273
]
Roshan Naik commented on STORM-2983:
[~kabhwan]
IMO this should not be viewed as a sidestepping an optimization. In which case
it does look like a minor issue. The optimization only exposed a bug. So it is
only the "messenger".
The key issue here is supporting (internal or user code) to check the state
correctly .. to do whatever it needs to do optimization or behavior
change.
By not fixing the core bug, we wlll sidestep the core issue. IMO this a
critical bug and needs to fixed. Or we need to provide a diff mechanism for
code to make such checks correctly.
> Some topologies not working properly
> -
>
> Key: STORM-2983
> URL: https://issues.apache.org/jira/browse/STORM-2983
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Ethan Li
>Assignee: Ethan Li
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For example,
> {code:java}
> bin/storm jar storm-loadgen-*.jar
> org.apache.storm.loadgen.ThroughputVsLatency --spouts 1 --splitters 2
> --counters 1 -c topology.debug=true
> {code}
> on ResourceAwareScheduler not working properly.
> With default cluster settings, there will be only one __acker-executor and it
> will be on a separate worker. And it looks like the __acker-executor was not
> able to receive messages from spouts and bolts. And spouts and bolts
> continued to retry sending messages to acker. It then led to another problem:
> STORM-2970
> I tried to run on storm right before
> [https://github.com/apache/storm/pull/2502] and right after and confirmed
> that this bug should be related to it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (STORM-2983) Some topologies not working properly
[
https://issues.apache.org/jira/browse/STORM-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420251#comment-16420251
]
Jungtaek Lim commented on STORM-2983:
-
While I agree that reflecting correct worker count is better, I'm OK for that
once UI show the information correctly, with proper notice for RAS.
I don't see specific reason to optimize single worker case. What Storm
basically provides is "distributed realtime computation system", not a single
JVM process to run faster. Such optimization was effective only for single
worker case which I think is minor and looks like only for benchmarking. I
guess we concerned about inter-worker latency (as well as acker) which looks
like the thing to focus.
> Some topologies not working properly
> -
>
> Key: STORM-2983
> URL: https://issues.apache.org/jira/browse/STORM-2983
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Ethan Li
>Assignee: Ethan Li
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For example,
> {code:java}
> bin/storm jar storm-loadgen-*.jar
> org.apache.storm.loadgen.ThroughputVsLatency --spouts 1 --splitters 2
> --counters 1 -c topology.debug=true
> {code}
> on ResourceAwareScheduler not working properly.
> With default cluster settings, there will be only one __acker-executor and it
> will be on a separate worker. And it looks like the __acker-executor was not
> able to receive messages from spouts and bolts. And spouts and bolts
> continued to retry sending messages to acker. It then led to another problem:
> STORM-2970
> I tried to run on storm right before
> [https://github.com/apache/storm/pull/2502] and right after and confirmed
> that this bug should be related to it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
