[jira] [Commented] (STORM-2787) storm-kafka-client KafkaSpout should set 'initialized' flag independently of processing guarantees
[
https://issues.apache.org/jira/browse/STORM-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223830#comment-16223830
]
Jungtaek Lim commented on STORM-2787:
-
Add '2.0.0' to fix version since it was merged into master. We're waiting for
the patch against 1.x branch to resolve the issue.
> storm-kafka-client KafkaSpout should set 'initialized' flag independently of
> processing guarantees
> --
>
> Key: STORM-2787
> URL: https://issues.apache.org/jira/browse/STORM-2787
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Hugo Louro
>Assignee: Hugo Louro
>Priority: Critical
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently the method
> {code:java}
> public void onPartitionsRevoked(Collection partitions) {
> {code}
> has the following condition
> {code:java}
> if (isAtLeastOnceProcessing() && initialized) {
> initialized = false;
> ...
> }
> {code}
> initialized should be set to false independently of the processing guarantee
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (STORM-2787) storm-kafka-client KafkaSpout should set 'initialized' flag independently of processing guarantees
[
https://issues.apache.org/jira/browse/STORM-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-2787:
Fix Version/s: 2.0.0
> storm-kafka-client KafkaSpout should set 'initialized' flag independently of
> processing guarantees
> --
>
> Key: STORM-2787
> URL: https://issues.apache.org/jira/browse/STORM-2787
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Hugo Louro
>Assignee: Hugo Louro
>Priority: Critical
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently the method
> {code:java}
> public void onPartitionsRevoked(Collection partitions) {
> {code}
> has the following condition
> {code:java}
> if (isAtLeastOnceProcessing() && initialized) {
> initialized = false;
> ...
> }
> {code}
> initialized should be set to false independently of the processing guarantee
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (STORM-1927) Upgrade Jetty and Ring
[ https://issues.apache.org/jira/browse/STORM-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-1927. - Resolution: Fixed Assignee: Ethan Li (was: darion yaphet) Fix Version/s: 2.0.0 Thanks [~ethanli], I merged into master. Please raise a PR against 1.x branch if you would want to push the change to 1.x version line. > Upgrade Jetty and Ring > -- > > Key: STORM-1927 > URL: https://issues.apache.org/jira/browse/STORM-1927 > Project: Apache Storm > Issue Type: Improvement > Components: storm-ui >Affects Versions: 1.0.0, 1.0.1 >Reporter: darion yaphet >Assignee: Ethan Li > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Jetty 7 is EOL , upgrade to Jetty 9 & Ring could also support it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2783) De-couple "Spout Lag" metrics on Storm UI from Kafka Spout and StormKafkaMonitor
[ https://issues.apache.org/jira/browse/STORM-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221734#comment-16221734 ] Jungtaek Lim commented on STORM-2783: - Regarding exposing metrics in UI, this should be one of future goal of metrics V2. JStorm exposes all the metrics for the topology into UI which can be achieved cause they publish all the (aggregated) metrics to Nimbus. When necessary Nimbus publishes the metrics to external storage. IMHO that should be the thing we need to have. Regarding exposing spout information in UI, I think restricting the information to fit some format can make us easy to show the information more nicely. I'm not familiar with frontend tech./UI/programming so didn't try out yet (unfortunately I'm struggling with other works so long), but I think it should be better if we can show kafka offset information to some kinds of graph - one of example is stacked graph which partitions are displayed ordered by lag. That's possible cause we know about context of the information. Although this couldn't cover all kinds of spouts, but we sometimes need to focus on users coverage of the change. > De-couple "Spout Lag" metrics on Storm UI from Kafka Spout and > StormKafkaMonitor > > > Key: STORM-2783 > URL: https://issues.apache.org/jira/browse/STORM-2783 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Affects Versions: 2.0.0, 1.1.1 >Reporter: Stephen Powis > > As a developer of a spout, I'd love to be able to publish lag metrics to the > storm UI. After digging into the source code for how the UI interacts with > storm-kafka-monitor to get these metrics, it appears to be strongly coupled. > I believe that the concept of "Spout Lag" extends beyond the scope of just > consuming from Kafka. > I'd like to propose the idea of restructuring how these metrics are queried > by StormUI to a way that allows developers of other spouts to be able to > "plug into" the UI. The easiest way that springs to mind is to provide an > interface that allows developers to code against. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2759) Let users indicate if a worker should restart on blob download
[ https://issues.apache.org/jira/browse/STORM-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2759. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~revans2], I merged into master. Great feature. > Let users indicate if a worker should restart on blob download > -- > > Key: STORM-2759 > URL: https://issues.apache.org/jira/browse/STORM-2759 > Project: Apache Storm > Issue Type: Improvement >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Some blobs (like jar files) really should be tied to the life cycle of a > worker. If a new blob is ready the worker should be restarted. Otherwise > there is no way to pick up the contents of the newly downloaded blob. > STORM-2438 already sets the ground work for this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2706) Nimbus stuck in exception and does not fail fast
[
https://issues.apache.org/jira/browse/STORM-2706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215993#comment-16215993
]
Jungtaek Lim commented on STORM-2706:
-
[~Srdo]
We checked that only Curator version is changed in dependency tree, then I'm OK
to move forward on this.
We should do some manual tests (and automated system tests if possible) before
releasing version for every releases so I'm expecting that will be covered
later.
Maybe you would want to bring this to also 1.2.0, then please craft a pull
request on 1.x branch as well.
> Nimbus stuck in exception and does not fail fast
>
>
> Key: STORM-2706
> URL: https://issues.apache.org/jira/browse/STORM-2706
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 1.1.1
>Reporter: Bijan Fahimi Shemrani
>Assignee: Stig Rohde Døssing
> Labels: nimbus, pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> We experience a problem in nimbus which leads it to get stuck in a retry and
> fail loop. When I manually restart the nimbus it works again as expected.
> However, it would be great if nimbus would shut down so our monitoring can
> automatically restart the nimbus.
> The nimbus log.
> {noformat}
> 24.8.2017 15:39:1913:39:19.804 [pool-13-thread-51] ERROR
> org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer -
> Unexpected throwable while invoking!
> 24.8.2017
> 15:39:19org.apache.storm.shade.org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /storm/leader-lock
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1590)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:230)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:219)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:216)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:207)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:40)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.recipes.locks.LockInternals.getSortedChildren(LockInternals.java:151)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.recipes.locks.LockInternals.getParticipantNodes(LockInternals.java:133)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.recipes.leader.LeaderLatch.getLeader(LeaderLatch.java:453)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown
> Source) ~[?:?]
> 24.8.2017 15:39:19at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_131]
> 24.8.2017 15:39:19at java.lang.reflect.Method.invoke(Method.java:498)
> ~[?:1.8.0_131]
> 24.8.2017 15:39:19at
> clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
> ~[clojure-1.7.0.jar:?]
> 24.8.2017 15:39:19at
> clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:313)
> ~[clojure-1.7.0.jar:?]
> 24.8.2017 15:39:19at
> org.apache.storm.zookeeper$zk_leader_elector$reify__1043.getLeader(zookeeper.clj:296)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown
> Source) ~[?:?]
> 24.8.2017 15:39:19at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_131]
> 24.8.2017 15:39:19at java.lang.reflect.Method.invoke(Method.java:498)
> ~[?:1.8.0_131]
> 24.8.2017 15:39:19at
> clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
>
[jira] [Commented] (STORM-2786) Ackers leak tracking info on failure and lots of other cases.
[ https://issues.apache.org/jira/browse/STORM-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215158#comment-16215158 ] Jungtaek Lim commented on STORM-2786: - Great finding! There're generally two kinds of failed tuples - explicit fail and timeout - and explicitly failed tuples will be remove from the map via https://github.com/apache/storm/blob/124acb92dff04a57b530ab4d95a698abc8ff46d9/storm-client/src/jvm/org/apache/storm/daemon/Acker.java#L120-L122 but timed-out tuples still would be leaked because of what you observed. Spout handles timed-out tuples independently hence only memory leak will occur which makes users not able be aware of. > Ackers leak tracking info on failure and lots of other cases. > - > > Key: STORM-2786 > URL: https://issues.apache.org/jira/browse/STORM-2786 > Project: Apache Storm > Issue Type: Bug > Components: storm-client, storm-core >Affects Versions: 0.9.1-incubating, 0.10.0, 1.0.0, 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Critical > > Over the weekend we had an incident where ackers were running out of memory > at a really scary rate. It turns out that they were having a lot of > failures, for an unrelated reason, but each of the failures were resulting in > tuple tracking being lost because... > We don't send ticks to any system components ever... > https://github.com/apache/storm/blob/124acb92dff04a57b530ab4d95a698abc8ff46d9/storm-client/src/jvm/org/apache/storm/executor/Executor.java#L384 > and ackers are system components. > So the tracking map was never rotated and all failed tuples > https://github.com/apache/storm/blob/124acb92dff04a57b530ab4d95a698abc8ff46d9/storm-client/src/jvm/org/apache/storm/daemon/Acker.java#L97-L103 > Were never deleted from the map. > This leak eventually made the ackers crash, and when they came back up the > other components kept blasting them with messages that would never be fully > acked which also leaked because of the tick problem. > Looking back this has been in every release since 0.9.1-incubating. It > appears to have been introduced by > https://github.com/apache/storm/commit/483ce454a3b2cd31b5d1c34e9365346459b358a8 > So every apache release has this problem (which is the only reason I have not > marked this as a blocker, because apparently it is not so bad that anyone has > noticed in the past 4 years). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2783) De-couple "Spout Lag" metrics on Storm UI from Kafka Spout and StormKafkaMonitor
[ https://issues.apache.org/jira/browse/STORM-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214247#comment-16214247 ] Jungtaek Lim commented on STORM-2783: - +1 on this. If we could construct an interface on spout information and let it pushed to built-in metrics, it should be flexible and non-Kafka spouts could enjoy similar benefit. > De-couple "Spout Lag" metrics on Storm UI from Kafka Spout and > StormKafkaMonitor > > > Key: STORM-2783 > URL: https://issues.apache.org/jira/browse/STORM-2783 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Affects Versions: 2.0.0, 1.1.1 >Reporter: Stephen Powis > > As a developer of a spout, I'd love to be able to publish lag metrics to the > storm UI. After digging into the source code for how the UI interacts with > storm-kafka-monitor to get these metrics, it appears to be strongly coupled. > I believe that the concept of "Spout Lag" extends beyond the scope of just > consuming from Kafka. > I'd like to propose the idea of restructuring how these metrics are queried > by StormUI to a way that allows developers of other spouts to be able to > "plug into" the UI. The easiest way that springs to mind is to provide an > interface that allows developers to code against. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2706) Nimbus stuck in exception and does not fail fast
[
https://issues.apache.org/jira/browse/STORM-2706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210752#comment-16210752
]
Jungtaek Lim commented on STORM-2706:
-
[~Srdo]
According to zk-compatibility page, Curator 4.0.x supports both Zookeeper 3.5.x
and 3.4.x, but major support version is 3.5.x and supporting 3.4.x looks more
like a compatibility mode. We may need to do full manual/system tests with
Curator 4.0.x/Zookeeper 3.4.x before deciding to move to Curator 4.0.x.
> Nimbus stuck in exception and does not fail fast
>
>
> Key: STORM-2706
> URL: https://issues.apache.org/jira/browse/STORM-2706
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 1.1.1
>Reporter: Bijan Fahimi Shemrani
>Assignee: Stig Rohde Døssing
> Labels: nimbus, pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We experience a problem in nimbus which leads it to get stuck in a retry and
> fail loop. When I manually restart the nimbus it works again as expected.
> However, it would be great if nimbus would shut down so our monitoring can
> automatically restart the nimbus.
> The nimbus log.
> {noformat}
> 24.8.2017 15:39:1913:39:19.804 [pool-13-thread-51] ERROR
> org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer -
> Unexpected throwable while invoking!
> 24.8.2017
> 15:39:19org.apache.storm.shade.org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /storm/leader-lock
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1590)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:230)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:219)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:216)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:207)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:40)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.recipes.locks.LockInternals.getSortedChildren(LockInternals.java:151)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.recipes.locks.LockInternals.getParticipantNodes(LockInternals.java:133)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.recipes.leader.LeaderLatch.getLeader(LeaderLatch.java:453)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown
> Source) ~[?:?]
> 24.8.2017 15:39:19at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_131]
> 24.8.2017 15:39:19at java.lang.reflect.Method.invoke(Method.java:498)
> ~[?:1.8.0_131]
> 24.8.2017 15:39:19at
> clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
> ~[clojure-1.7.0.jar:?]
> 24.8.2017 15:39:19at
> clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:313)
> ~[clojure-1.7.0.jar:?]
> 24.8.2017 15:39:19at
> org.apache.storm.zookeeper$zk_leader_elector$reify__1043.getLeader(zookeeper.clj:296)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown
> Source) ~[?:?]
> 24.8.2017 15:39:19at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_131]
> 24.8.2017 15:39:19at java.lang.reflect.Method.invoke(Method.java:498)
> ~[?:1.8.0_131]
> 24.8.2017 15:39:19at
> clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
> ~[clojure-1.7.0.jar:?]
> 24.8.2017 15:39:19at
>
[jira] [Commented] (STORM-2706) Nimbus stuck in exception and does not fail fast
[
https://issues.apache.org/jira/browse/STORM-2706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208882#comment-16208882
]
Jungtaek Lim commented on STORM-2706:
-
[~randgalt]
Hi Jordan, I think it is ideal for resolving current issue if Curator could
have new 2.x release. If not we may need to go on upgrading Curator version.
AFAIK, we intended to not update Curator version because it refers Zookeeper
3.5.x which Zookeeper community has been saying it as 'alpha', and recently
saying it as 'beta'.
Does curator community feel safe for Zookeeper 3.5.x to upgrade?
> Nimbus stuck in exception and does not fail fast
>
>
> Key: STORM-2706
> URL: https://issues.apache.org/jira/browse/STORM-2706
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 1.1.1
>Reporter: Bijan Fahimi Shemrani
> Labels: nimbus
>
> We experience a problem in nimbus which leads it to get stuck in a retry and
> fail loop. When I manually restart the nimbus it works again as expected.
> However, it would be great if nimbus would shut down so our monitoring can
> automatically restart the nimbus.
> The nimbus log.
> {noformat}
> 24.8.2017 15:39:1913:39:19.804 [pool-13-thread-51] ERROR
> org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer -
> Unexpected throwable while invoking!
> 24.8.2017
> 15:39:19org.apache.storm.shade.org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /storm/leader-lock
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1590)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:230)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:219)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:216)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:207)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:40)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.recipes.locks.LockInternals.getSortedChildren(LockInternals.java:151)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.recipes.locks.LockInternals.getParticipantNodes(LockInternals.java:133)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at
> org.apache.storm.shade.org.apache.curator.framework.recipes.leader.LeaderLatch.getLeader(LeaderLatch.java:453)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown
> Source) ~[?:?]
> 24.8.2017 15:39:19at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_131]
> 24.8.2017 15:39:19at java.lang.reflect.Method.invoke(Method.java:498)
> ~[?:1.8.0_131]
> 24.8.2017 15:39:19at
> clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
> ~[clojure-1.7.0.jar:?]
> 24.8.2017 15:39:19at
> clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:313)
> ~[clojure-1.7.0.jar:?]
> 24.8.2017 15:39:19at
> org.apache.storm.zookeeper$zk_leader_elector$reify__1043.getLeader(zookeeper.clj:296)
> ~[storm-core-1.1.1.jar:1.1.1]
> 24.8.2017 15:39:19at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown
> Source) ~[?:?]
> 24.8.2017 15:39:19at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_131]
> 24.8.2017 15:39:19at java.lang.reflect.Method.invoke(Method.java:498)
> ~[?:1.8.0_131]
> 24.8.2017 15:39:19at
> clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
> ~[clojure-1.7.0.jar:?]
> 24.8.2017 15:39:19at
> clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:313)
>
[jira] [Resolved] (STORM-2779) NPE on shutting down WindowedBoltExecutor
[ https://issues.apache.org/jira/browse/STORM-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2779. - Resolution: Fixed Fix Version/s: 1.1.2 1.2.0 2.0.0 Merged into master, 1.x, 1.1.x branches. > NPE on shutting down WindowedBoltExecutor > - > > Key: STORM-2779 > URL: https://issues.apache.org/jira/browse/STORM-2779 > Project: Apache Storm > Issue Type: Bug > Components: storm-client >Affects Versions: 2.0.0, 1.2.0, 1.1.2 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim > Labels: pull-request-available > Fix For: 2.0.0, 1.2.0, 1.1.2 > > Time Spent: 2h > Remaining Estimate: 0h > > STORM-2724 introduced a bug on WindowedBoltExecutor which throws NPE when > shutting down WindowedBoltExecutor which has waterMarkEventGenerator field as > null. > https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/topology/WindowedBoltExecutor.java#L330 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (STORM-2779) NPE on shutting down WindowedBoltExecutor
Jungtaek Lim created STORM-2779: --- Summary: NPE on shutting down WindowedBoltExecutor Key: STORM-2779 URL: https://issues.apache.org/jira/browse/STORM-2779 Project: Apache Storm Issue Type: Bug Components: storm-client Affects Versions: 2.0.0, 1.2.0, 1.1.2 Reporter: Jungtaek Lim Assignee: Jungtaek Lim STORM-2724 introduced a bug on WindowedBoltExecutor which throws NPE when shutting down WindowedBoltExecutor which has waterMarkEventGenerator field as null. https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/topology/WindowedBoltExecutor.java#L330 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2666) Storm-kafka-client spout can sometimes emit messages that were already committed.
[
https://issues.apache.org/jira/browse/STORM-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-2666.
-
Resolution: Fixed
Fix Version/s: 1.1.2
1.2.0
Thanks [~Srdo], I also merged it into 1.x and 1.1.x branches.
> Storm-kafka-client spout can sometimes emit messages that were already
> committed.
> --
>
> Key: STORM-2666
> URL: https://issues.apache.org/jira/browse/STORM-2666
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
>Affects Versions: 1.0.0, 2.0.0, 1.1.0, 1.1.1, 1.2.0
>Reporter: Guang Du
>Assignee: Stig Rohde Døssing
> Labels: pull-request-available
> Fix For: 2.0.0, 1.2.0, 1.1.2
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Under a certain heavy load, for failed/timeout tuples, the retry service will
> ack tuple for failed max times. Kafka Client Spout will commit after reached
> the commit interval. However seems some 'on the way' tuples will be failed
> again, the retry service will cause Spout to emit again, and acked eventually
> to OffsetManager.
> In some cases such offsets are too many, exceeding the max-uncommit, causing
> org.apache.storm.kafka.spout.internal.OffsetManager#findNextCommitOffset
> unable to find next commit point, and Spout for this partition will not poll
> any more.
> By the way I've applied STORM-2549 PR#2156 from Stig Døssing to fix
> STORM-2625, and I'm using Python Shell Bolt as processing bolt, if this
> information helps.
> resulting logs like below. I'm not sure if the issue has already been
> raised/fixed, glad if anyone could help to point out existing JIRA. Thank you.
> 2017-07-27 22:23:48.398 o.a.s.k.s.KafkaSpout Thread-23-spout-executor[248
> 248] [INFO] Successful ack for tuple message
> [{topic-partition=kafka_bd_trigger_action-20, offset=18204, numFails=0}].
> 2017-07-27 22:23:49.203 o.a.s.k.s.i.OffsetManager
> Thread-23-spout-executor[248 248] [WARN] topic-partition
> [kafka_bd_trigger_action-18] has unexpected offset [16002]. Current committed
> Offset [16003]
> Edit:
> See
> https://issues.apache.org/jira/browse/STORM-2666?focusedCommentId=16125893=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16125893
> for the current best guess at the root cause of this issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (STORM-2716) Storm-webapp tests don't work on Windows
[ https://issues.apache.org/jira/browse/STORM-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196423#comment-16196423 ] Jungtaek Lim commented on STORM-2716: - [~Srdo] Porting back to 1.x needs another PR so please feel free to create one if you would want to, or skip one if you don't. > Storm-webapp tests don't work on Windows > > > Key: STORM-2716 > URL: https://issues.apache.org/jira/browse/STORM-2716 > Project: Apache Storm > Issue Type: Bug > Components: storm-webapp >Affects Versions: 2.0.0 > Environment: Windows 10 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Several storm-webapp tests don't work on Windows because file paths like > "/tmp" are used, and paths are sometimes constructed by String concatenation > instead of using Path.resolve. The logviewer also doesn't seem to work on > Windows, probably for the same reason. > I think there might be a few similar issues in other parts of the code, which > I'd like to also fix as part of this. > I haven't checked whether this is a problem in 1.x, but it's likely. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2716) Storm-webapp tests don't work on Windows
[ https://issues.apache.org/jira/browse/STORM-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2716. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~Srdo], I merged into master. > Storm-webapp tests don't work on Windows > > > Key: STORM-2716 > URL: https://issues.apache.org/jira/browse/STORM-2716 > Project: Apache Storm > Issue Type: Bug > Components: storm-webapp >Affects Versions: 2.0.0 > Environment: Windows 10 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Several storm-webapp tests don't work on Windows because file paths like > "/tmp" are used, and paths are sometimes constructed by String concatenation > instead of using Path.resolve. The logviewer also doesn't seem to work on > Windows, probably for the same reason. > I think there might be a few similar issues in other parts of the code, which > I'd like to also fix as part of this. > I haven't checked whether this is a problem in 1.x, but it's likely. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2756) STORM-2548 on 1.x-branch broke setting key/value deserializers with the now deprecated setKey/setValue methods
[ https://issues.apache.org/jira/browse/STORM-2756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2756. - Resolution: Fixed Fix Version/s: 1.2.0 Thanks [~Srdo], I merged into 1.x branch. > STORM-2548 on 1.x-branch broke setting key/value deserializers with the now > deprecated setKey/setValue methods > -- > > Key: STORM-2756 > URL: https://issues.apache.org/jira/browse/STORM-2756 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Affects Versions: 1.2.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing > Labels: pull-request-available > Fix For: 1.2.0 > > Time Spent: 20m > Remaining Estimate: 0h > > When STORM-2548 was backported, the setKey/setValue methods on > KafkaSpoutConfig.Builder were deprecated, and users were directed to use > setProp along with the relevant ConsumerConfig constants for setting > deserializers instead. > As part of this change, the KafkaConsumerFactoryDefault switched from using > the KafkaConsumer(props, keyDes, valDes) constructor to using the > KafkaConsumer(props) constructor. Unfortunately I forgot to update the > KafkaSpoutConfig.Builder constructor properly, so if the user configures the > deserializer via either the Builder constructor parameters or > setKey/setValue, the setting is not put in the kafkaProps map and the > deserializer is not used. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2718) Show some descriptions on LogViewer index page
[ https://issues.apache.org/jira/browse/STORM-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2718. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~ethanli], I merged into master. > Show some descriptions on LogViewer index page > -- > > Key: STORM-2718 > URL: https://issues.apache.org/jira/browse/STORM-2718 > Project: Apache Storm > Issue Type: Improvement >Reporter: Ethan Li >Assignee: Ethan Li >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > I remembered when I was pretty new to storm and I launched logviewer for the > first time. I visited x:8000 out of my natural but then got "page not > found". I thought there must be something wrong with my storm setup. It took > me a while to figure it out that logviewer doesn't work that way. I also > found some other people were also confused about this, > [example|https://stackoverflow.com/questions/25538327/storm-logviewer-page-not-found] > . We probably want to show some descriptions/guides on the index page. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2764) HDFSBlobStore leaks file system objects
[ https://issues.apache.org/jira/browse/STORM-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2764. - Resolution: Fixed Fix Version/s: 1.0.6 1.1.2 1.2.0 2.0.0 Thanks [~revans2], I merged into master, 1.x, 1.1.x, 1.0.x branches. > HDFSBlobStore leaks file system objects > --- > > Key: STORM-2764 > URL: https://issues.apache.org/jira/browse/STORM-2764 > Project: Apache Storm > Issue Type: Bug > Components: storm-hdfs >Affects Versions: 1.0.0, 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Labels: pull-request-available > Fix For: 2.0.0, 1.2.0, 1.1.2, 1.0.6 > > Time Spent: 1h > Remaining Estimate: 0h > > This impacts all of the releases. Each time we create a new HDFSBlobStore > instance we call > https://github.com/apache/storm/blob/v1.0.0/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/blobstore/HdfsBlobStore.java#L140 > loginUserFromKeytab. > This results in a new subject being created each time when ends up causing a > FileSystem object to leak each time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2357) add At-Most-Once guarantee in KafkaSpout
[ https://issues.apache.org/jira/browse/STORM-2357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185378#comment-16185378 ] Jungtaek Lim commented on STORM-2357: - The one pull request addresses both of the issues. > add At-Most-Once guarantee in KafkaSpout > > > Key: STORM-2357 > URL: https://issues.apache.org/jira/browse/STORM-2357 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Reporter: Xu Mingmin >Assignee: Stig Rohde Døssing > Fix For: 2.0.0 > > > KafkaSpout cannot guarantee exactly at-most-once semantic, as it commits > offset periodically, and KafkaSpout.close() is not guarantee to call when a > crash happens. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2357) add At-Most-Once guarantee in KafkaSpout
[ https://issues.apache.org/jira/browse/STORM-2357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2357: Fix Version/s: 2.0.0 > add At-Most-Once guarantee in KafkaSpout > > > Key: STORM-2357 > URL: https://issues.apache.org/jira/browse/STORM-2357 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Reporter: Xu Mingmin >Assignee: Stig Rohde Døssing > Fix For: 2.0.0 > > > KafkaSpout cannot guarantee exactly at-most-once semantic, as it commits > offset periodically, and KafkaSpout.close() is not guarantee to call when a > crash happens. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2648) Kafka spout can't show acks/fails and complete latency when auto commit is enabled
[ https://issues.apache.org/jira/browse/STORM-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185373#comment-16185373 ] Jungtaek Lim commented on STORM-2648: - Thanks [~Srdo] for the patch. I merged into master. Given that target version is is marked as 1.2.0, could you please craft the patch for 1.x branch? I tried to cherry pick but it doesn't looks like a clean cherry-pick. > Kafka spout can't show acks/fails and complete latency when auto commit is > enabled > -- > > Key: STORM-2648 > URL: https://issues.apache.org/jira/browse/STORM-2648 > Project: Apache Storm > Issue Type: New Feature > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.1.0, 1.2.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > The storm-kafka-client spout currently emits tuples with no message ids if > auto commit is enabled. This causes the ack/fail/complete latency counters in > Storm UI to be 0. In some cases this is desirable because the user may not > care, and doesn't want the overhead of Storm tracking tuples. > [~avermeerbergen] expressed a desire to be able to use auto commit without > these counters being disabled, presumably to monitor topology performance. > We should add a toggle that allows users to enable/disable tuple anchoring in > the auto commit case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2648) Kafka spout can't show acks/fails and complete latency when auto commit is enabled
[ https://issues.apache.org/jira/browse/STORM-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2648: Fix Version/s: 2.0.0 > Kafka spout can't show acks/fails and complete latency when auto commit is > enabled > -- > > Key: STORM-2648 > URL: https://issues.apache.org/jira/browse/STORM-2648 > Project: Apache Storm > Issue Type: New Feature > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.1.0, 1.2.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > The storm-kafka-client spout currently emits tuples with no message ids if > auto commit is enabled. This causes the ack/fail/complete latency counters in > Storm UI to be 0. In some cases this is desirable because the user may not > care, and doesn't want the overhead of Storm tracking tuples. > [~avermeerbergen] expressed a desire to be able to use auto commit without > these counters being disabled, presumably to monitor topology performance. > We should add a toggle that allows users to enable/disable tuple anchoring in > the auto commit case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2757) Links are broken when logviewer https port is used
[
https://issues.apache.org/jira/browse/STORM-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-2757.
-
Resolution: Fixed
Fix Version/s: 2.0.0
Thanks [~ethanli], I merged into master.
> Links are broken when logviewer https port is used
> --
>
> Key: STORM-2757
> URL: https://issues.apache.org/jira/browse/STORM-2757
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Ethan Li
>Assignee: Ethan Li
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Attachments: screenshot-1.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Some links are broken when logviewer.https.port is configured.
> For example, with the configuration:
> {code:java}
> logviewer.https.port: 9093
> logviewer.https.keystore.type: "JKS"
> logviewer.https.keystore.path: "/keystore-path"
> logviewer.https.keystore.password: "xx"
> logviewer.https.key.password: "xx"
> {code}
> We will get:
> [^screenshot-1.png]
> The logLink is still using http port. However, it's not reachable.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (STORM-2758) logviewer_search page not found
[ https://issues.apache.org/jira/browse/STORM-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2758. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~ethanli], I merged into master. > logviewer_search page not found > --- > > Key: STORM-2758 > URL: https://issues.apache.org/jira/browse/STORM-2758 > Project: Apache Storm > Issue Type: Bug >Reporter: Ethan Li >Assignee: Ethan Li >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0 > > Attachments: Screen Shot 2017-09-26 at 11.35.03 AM.png, Screen Shot > 2017-09-26 at 11.35.17 AM.png > > Time Spent: 20m > Remaining Estimate: 0h > > I was able to search logs using search/deepSearch. > For example, > [^Screen Shot 2017-09-26 at 11.35.17 AM.png] > But the link doesn't work. I got 404 not found when I clicked on the result > link. > [^Screen Shot 2017-09-26 at 11.35.03 AM.png] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2084) after supervisor v2 merge async localizer and localizer
[ https://issues.apache.org/jira/browse/STORM-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2084. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~revans2], I merged into master. > after supervisor v2 merge async localizer and localizer > --- > > Key: STORM-2084 > URL: https://issues.apache.org/jira/browse/STORM-2084 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Affects Versions: 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Once we mere in STORM-2018 > https://github.com/apache/storm/pull/1642 > we should look into merging the two localizers into a single class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (STORM-2762) Sort out multiple places of codes of manipulation of collections
Jungtaek Lim created STORM-2762: --- Summary: Sort out multiple places of codes of manipulation of collections Key: STORM-2762 URL: https://issues.apache.org/jira/browse/STORM-2762 Project: Apache Storm Issue Type: Bug Affects Versions: 2.0.0 Reporter: Jungtaek Lim Priority: Minor Storm has codes of manipulation of collections at multiple places. One of example would be blacklist/Sets.java which will be merged in via STORM-2083. https://github.com/apache/storm/pull/2343 It would be better to sort out and make them common utils, or if possible, replace them with specific library Storm already depends on. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2693) Topology submission or kill takes too much time when topologies grow to a few hundred
[ https://issues.apache.org/jira/browse/STORM-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180275#comment-16180275 ] Jungtaek Lim commented on STORM-2693: - [~danny0405] The patch is meant to be compatible with current. When we move to metrics V2 (also built-in) completely, we could get rid of metrics in heartbeat. Indeed you're right about current status, so let's forget about metrics V2 for now and go ahead. > Topology submission or kill takes too much time when topologies grow to a few > hundred > - > > Key: STORM-2693 > URL: https://issues.apache.org/jira/browse/STORM-2693 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Affects Versions: 0.9.6, 1.0.2, 1.1.0, 1.0.3 >Reporter: Yuzhao Chen > Labels: pull-request-available > Attachments: 2FA30CD8-AF15-4352-992D-A67BD724E7FB.png, > D4A30D40-25D5-4ACF-9A96-252EBA9E6EF6.png > > Time Spent: 5h > Remaining Estimate: 0h > > Now for a storm cluster with 40 hosts [with 32 cores/128G memory] and > hundreds of topologies, nimbus submission and killing will take about minutes > to finish. For example, for a cluster with 300 hundred of topologies，it will > take about 8 minutes to submit a topology, this affect our efficiency > seriously. > So, i check out the nimbus code and find two factor that will effect nimbus > submission/killing time for a scheduling round: > * read existing-assignments from zookeeper for every topology [will take > about 4 seconds for a 300 topologies cluster] > * read all the workers heartbeats and update the state to nimbus cache [will > take about 30 seconds for a 300 topologies cluster] > the key here is that Storm now use zookeeper to collect heartbeats [not RPC], > and also keep physical plan [assignments] using zookeeper which can be > totally local in nimbus. > So, i think we should make some changes to storm's heartbeats and assignments > management. > For assignment promotion: > 1. nimbus will put the assignments in local disk > 2. when restart or HA leader trigger nimbus will recover assignments from zk > to local disk > 3. nimbus will tell supervisor its assignment every time through RPC every > scheduling round > 4. supervisor will sync assignments at fixed time > For heartbeats promotion: > 1. workers will report executors ok or wrong to supervisor at fixed time > 2. supervisor will report workers heartbeats to nimbus at fixed time > 3. if supervisor die, it will tell nimbus through runtime hook > or let nimbus find it through aware supervisor if is survive > 4. let supervisor decide if worker is running ok or invalid , supervisor will > tell nimbus which executors of every topology are ok -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (STORM-2083) Blacklist Scheduler
[ https://issues.apache.org/jira/browse/STORM-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174987#comment-16174987 ] Jungtaek Lim edited comment on STORM-2083 at 9/21/17 3:51 PM: -- [~howardhl] Hi, I think Github mention wouldn't work for most of us, so leaving kindly reminder from JIRA. Could you spend time to continue working on PR to finalize? If you can't or you don't want by yourself please let me know so that I can take over with preserving your credit. Thanks in advance! was (Author: kabhwan): [~howardhl] Hi, I think Github mention wouldn't work for most of us, so leaving kindly reminder. Could you spend time to continue working on PR to finalize? If you can't or you don't want by yourself please let me know so that I can take over with preserving your credit. Thanks in advance! > Blacklist Scheduler > --- > > Key: STORM-2083 > URL: https://issues.apache.org/jira/browse/STORM-2083 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Howard Lee > Labels: blacklist, scheduling > Time Spent: 15h 10m > Remaining Estimate: 0h > > My company has gone through a fault in production, in which a critical switch > causes unstable network for a set of machines with package loss rate of > 30%-50%. In such fault, the supervisors and workers on the machines are not > definitely dead, which is easy to handle. Instead they are still alive but > very unstable. They lost heartbeat to the nimbus occasionally. The nimbus, in > such circumstance, will still assign jobs to these machines, but will soon > find them invalid again, result in a very slow convergence to stable status. > To deal with such unstable cases, we intend to implement a blacklist > scheduler, which will add the unstable nodes (supervisors, slots) to the > blacklist temporarily, and resume them later. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2083) Blacklist Scheduler
[ https://issues.apache.org/jira/browse/STORM-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174987#comment-16174987 ] Jungtaek Lim commented on STORM-2083: - [~howardhl] Hi, I think Github mention wouldn't work for most of us, so leaving kindly reminder. Could you spend time to continue working on PR to finalize? If you can't or you don't want by yourself please let me know so that I can take over with preserving your credit. Thanks in advance! > Blacklist Scheduler > --- > > Key: STORM-2083 > URL: https://issues.apache.org/jira/browse/STORM-2083 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Howard Lee > Labels: blacklist, scheduling > Time Spent: 15h 10m > Remaining Estimate: 0h > > My company has gone through a fault in production, in which a critical switch > causes unstable network for a set of machines with package loss rate of > 30%-50%. In such fault, the supervisors and workers on the machines are not > definitely dead, which is easy to handle. Instead they are still alive but > very unstable. They lost heartbeat to the nimbus occasionally. The nimbus, in > such circumstance, will still assign jobs to these machines, but will soon > find them invalid again, result in a very slow convergence to stable status. > To deal with such unstable cases, we intend to implement a blacklist > scheduler, which will add the unstable nodes (supervisors, slots) to the > blacklist temporarily, and resume them later. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2534) Visualization API missing stats/instances for "system" components
[
https://issues.apache.org/jira/browse/STORM-2534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-2534.
-
Resolution: Fixed
Fix Version/s: 2.0.0
PR for STORM-2533 covered this.
> Visualization API missing stats/instances for "system" components
> -
>
> Key: STORM-2534
> URL: https://issues.apache.org/jira/browse/STORM-2534
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Stephen Powis
>Assignee: Jungtaek Lim
> Fix For: 2.0.0
>
>
> The topology visualization api end point (
> /api/v1/topology/TOPOLOGY-ID/visualization ) does not return correct "stats"
> values for "system" components __system and __acker.
> See the following example *correct* response for a spout or bolt within a
> topology, shorten for brevity. Under the stats key it lists all of the
> instances of that component that is deployed.
> {code}
> {
> "spout": {
> ...
> ":stats": [{
> ":host":
> "e54bb273-2a8a-4320-b23f-7c7ace52c961-10.153.0.30",
> ":port": 6700,
> ":uptime_secs": 0,
> ":transferred": {
> ...
> }
> }],
> ...
> },
> {code}
> See the following response for the __system and __acker components. They do
> *not* correctly list any entries under the stats key.
> {code}
> {
> "__system": {
> ":type": "spout",
> ":capacity": 0,
> ":latency": null,
> ":transferred": null,
> ":stats": [],
> ":link":
> "\/component.html?id=__system_id=test-1-1495630798",
> ":inputs": []
> },
> "__acker": {
> ":type": "spout",
> ":capacity": 0,
> ":latency": null,
> ":transferred": null,
> ":stats": [],
> ":link":
> "\/component.html?id=__acker_id=test-1-1495630798",
> ":inputs": [...]
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (STORM-2712) accept arbitrary number of rows per tuple in storm-cassandra
[ https://issues.apache.org/jira/browse/STORM-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2712. - Resolution: Fixed Assignee: Yohei Kishimoto Fix Version/s: 1.1.2 1.2.0 2.0.0 Thanks [~morokosi], I merged into master, 1.x, 1.1.x branches. Keep up the good work. > accept arbitrary number of rows per tuple in storm-cassandra > > > Key: STORM-2712 > URL: https://issues.apache.org/jira/browse/STORM-2712 > Project: Apache Storm > Issue Type: Improvement > Components: storm-cassandra >Affects Versions: 2.0.0, 1.1.1 >Reporter: Yohei Kishimoto >Assignee: Yohei Kishimoto >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0, 1.2.0, 1.1.2 > > Time Spent: 1h > Remaining Estimate: 0h > > Current implementation in `TridentResultSetValuesMapper::map` restricts a > SELECT query to return one row. In `StateQueryProcessor::finishBatch`, it > checks the equality between the result size of `batchRetrieve` and input > tuple size. When the number of result rows is less than 1 or greater than 1, > it breaks the condition and an exception is thrown. > We should accept arbitrary number of rows by adjusting List dimensions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2171) blob recovery on a single host results in deadlock
[ https://issues.apache.org/jira/browse/STORM-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164339#comment-16164339 ] Jungtaek Lim commented on STORM-2171: - Just curious: Is this based on theory or are you experienced deadlock? If you experienced the thing and have stack dump it would be better to attach it. And are you planning to fix it for yourself? > blob recovery on a single host results in deadlock > -- > > Key: STORM-2171 > URL: https://issues.apache.org/jira/browse/STORM-2171 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 2.0.0 >Reporter: Robert Joseph Evans > > It might be more versions but I have only tested this on 2.x. > Essentially when trying to find replicas to copy blobs from LocalFSBlobStore > does not exclude itself. This results in a deadlock where it is holding a > lock trying to download the blob, and at the same time has done a request > back to itself trying to download the blob, but it will never finish because > it is blocked on the same lock. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2675) KafkaTridentSpoutOpaque not committing offsets to Kafka
[
https://issues.apache.org/jira/browse/STORM-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-2675.
-
Resolution: Fixed
Fix Version/s: 1.2.0
Also merged into 1.x branch.
> KafkaTridentSpoutOpaque not committing offsets to Kafka
> ---
>
> Key: STORM-2675
> URL: https://issues.apache.org/jira/browse/STORM-2675
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
>Affects Versions: 1.1.0
>Reporter: Preet Puri
>Assignee: Stig Rohde Døssing
> Labels: pull-request-available
> Fix For: 2.0.0, 1.2.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Every time I restart the topology the spout was picking the earliest message
> even though poll strategy is set UNCOMMITTED_EARLIEST. I looked at Kafka's
> __consumer_offsets topic to see if spout (consumer) is committing the offsets
> but did not find any commits. I am not even able to locate the code in the
> KafkaTridentSpoutEmitter class where we are updating the commits?
> conf.put(Config.TOPOLOGY_DEBUG, true);
> conf.put(Config.TOPOLOGY_WORKERS, 1);
> conf.put(Config.TOPOLOGY_MAX_SPOUT_PENDING, 4); //tried with1 as well
> conf.put(Config.TRANSACTIONAL_ZOOKEEPER_ROOT, "/aggregate");
> conf.put(Config.TRANSACTIONAL_ZOOKEEPER_SERVERS, Arrays.asList(new
> String[]{"localhost"}));
> conf.put(Config.TRANSACTIONAL_ZOOKEEPER_PORT, 2181);
> protected static KafkaSpoutConfig
> getPMStatKafkaSpoutConfig() {
> ByTopicRecordTranslator byTopic =
> new ByTopicRecordTranslator<>((r) -> new Values(r.topic(), r.key(),
> r.value()),
> new Fields(TOPIC, PARTITION_KEY, PAYLOAD), SENSOR_STREAM);
> return new KafkaSpoutConfig.Builder String>(Utils.getBrokerHosts(),
> StringDeserializer.class, null, Utils.getKafkaEnrichedPMSTopicName())
> .setMaxPartitionFectchBytes(10 * 1024) // 10 KB
> .setRetry(getRetryService())
> .setOffsetCommitPeriodMs(10_000)
>
> .setFirstPollOffsetStrategy(FirstPollOffsetStrategy.UNCOMMITTED_EARLIEST)
> .setMaxUncommittedOffsets(250)
> .setProp("value.deserializer",
> "io.confluent.kafka.serializers.KafkaAvroDeserializer")
> .setProp("schema.registry.url","http://localhost:8081;)
> .setProp("specific.avro.reader",true)
> .setGroupId(AGGREGATION_CONSUMER_GROUP)
> .setRecordTranslator(byTopic).build();
> }
> Stream pmStatStream =
> topology.newStream("statStream", new
> KafkaTridentSpoutOpaque<>(getPMStatKafkaSpoutConfig())).parallelismHint(1)
> storm-version - 1.1.0
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (STORM-2693) Topology submission or kill takes too much time when topologies grow to a few hundred
[ https://issues.apache.org/jira/browse/STORM-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164246#comment-16164246 ] Jungtaek Lim commented on STORM-2693: - [~danny0405] Yes I think it should work. Then I don't think we should store cache to disk, because ZK is still source of truth, and when Nimbus is restarting it must read ZK other than reading from disk. Caching to memory looks sufficient. PR for Metrics V2 is available: https://github.com/apache/storm/pull/2203 > Topology submission or kill takes too much time when topologies grow to a few > hundred > - > > Key: STORM-2693 > URL: https://issues.apache.org/jira/browse/STORM-2693 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Affects Versions: 0.9.6, 1.0.2, 1.1.0, 1.0.3 >Reporter: Yuzhao Chen > Attachments: 2FA30CD8-AF15-4352-992D-A67BD724E7FB.png, > D4A30D40-25D5-4ACF-9A96-252EBA9E6EF6.png > > Time Spent: 2.5h > Remaining Estimate: 0h > > Now for a storm cluster with 40 hosts [with 32 cores/128G memory] and > hundreds of topologies, nimbus submission and killing will take about minutes > to finish. For example, for a cluster with 300 hundred of topologies，it will > take about 8 minutes to submit a topology, this affect our efficiency > seriously. > So, i check out the nimbus code and find two factor that will effect nimbus > submission/killing time for a scheduling round: > * read existing-assignments from zookeeper for every topology [will take > about 4 seconds for a 300 topologies cluster] > * read all the workers heartbeats and update the state to nimbus cache [will > take about 30 seconds for a 300 topologies cluster] > the key here is that Storm now use zookeeper to collect heartbeats [not RPC], > and also keep physical plan [assignments] using zookeeper which can be > totally local in nimbus. > So, i think we should make some changes to storm's heartbeats and assignments > management. > For assignment promotion: > 1. nimbus will put the assignments in local disk > 2. when restart or HA leader trigger nimbus will recover assignments from zk > to local disk > 3. nimbus will tell supervisor its assignment every time through RPC every > scheduling round > 4. supervisor will sync assignments at fixed time > For heartbeats promotion: > 1. workers will report executors ok or wrong to supervisor at fixed time > 2. supervisor will report workers heartbeats to nimbus at fixed time > 3. if supervisor die, it will tell nimbus through runtime hook > or let nimbus find it through aware supervisor if is survive > 4. let supervisor decide if worker is running ok or invalid , supervisor will > tell nimbus which executors of every topology are ok -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (STORM-2693) Topology submission or kill takes too much time when topologies grow to a few hundred
[ https://issues.apache.org/jira/browse/STORM-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161132#comment-16161132 ] Jungtaek Lim edited comment on STORM-2693 at 9/11/17 12:17 PM: --- [~danny0405] Just to put some guides for contribution (sorry we didn't document it nicely): - all the patches are ideal to be against with master (currently 2.0.0) - merger or any committers can decide which version line(s) the patch should be put in, and port back if necessary - there's exceptional case: if patch addresses the bug which only resides on that version line(s) Let's back to your design. I haven't look into the detail, but I'm wondering how reading/storing from/to local disk will work with Nimbus H/A. Storm puts any critical data into zookeeper to ensure the data can be available at any time (when ZK is not available Storm will not work anyway...) This would be no longer true if we store it to only local disk. You would want to think about the cases which each Nimbus has its own assignments (which is not in sync) stored to local (only one of them is leader though) and all of them are shutting down and restarting at the same time. Which Nimbus should be a leader? How we ensure the elected Nimbus has the latest assignments? Btw, the idea brings me some thoughts: one of Storm's great point is stateless which can be achieved with stable storage (ZK for Storm), and we claims that Supervisor can work even Nimbus goes down. Now we introduced Nimbus H/A which makes Nimbus not a SPOF, and so some components like Supervisor may be able to rely on leader Nimbus instead of communicating with ZK if it is much faster and it doesn't put too much load to the leader Nimbus. (And this approach looks like same as your approach on your idea.) And since we only allow leader Nimbus to handle assignments, while we still need to write assignments to ZK, we can 'cache' it within leader Nimbus (and update cache if there's any change on assignments: note that we should also write to ZK earlier though) and avoid reading it from ZK. We still need to get heartbeat (and statistics) from ZK, but it will be handled as similar idea or another approach (like Metrics V2 in point of metrics' view) was (Author: kabhwan): [~danny0405] Just to put some guides for contribution (sorry we didn't document it nicely): - all the patches are ideal to be against with master (currently 2.0.0) - merger or any committers can decide which version line(s) the patch should be put in, and port back if necessary - there's exceptional case: if patch addresses the bug which only resides on that version line(s) Let's back to your design. I haven't look into the detail, but I'm wondering how reading/storing from/to local disk will work with Nimbus H/A. Storm puts any critical data into zookeeper to ensure the data can be available at any time (when ZK is not available Storm will not work so...) This would be no longer true if we store it to only local disk. You would want to think about the cases which each Nimbus has its own assignments (which is not in sync) stored to local (only one of them is leader though) and all of them are shutting down and restarting at the same time. Which Nimbus should be a leader? How we ensure the elected Nimbus has the latest assignments? Btw, the idea brings me some thoughts: one of Storm's great point is stateless which can be achieved with stable storage (ZK for Storm), and we claims that Supervisor can work even Nimbus goes down. Now we introduced Nimbus H/A which makes Nimbus not a SPOF, and so some components like Supervisor may be able to rely on leader Nimbus instead of communicating with ZK if it is much faster and it doesn't put too much load to the leader Nimbus. (And this approach looks like same as your approach on your idea.) And since we only allow leader Nimbus to handle assignments, while we still need to write assignments to ZK, we can 'cache' it within leader Nimbus (and update cache if there's any change on assignments: note that we should also write to ZK earlier though) and avoid reading it from ZK. We still need to get heartbeat (and statistics) from ZK, but it will be handled as similar idea or another approach (like Metrics V2 in point of metrics' view) > Topology submission or kill takes too much time when topologies grow to a few > hundred > - > > Key: STORM-2693 > URL: https://issues.apache.org/jira/browse/STORM-2693 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Affects Versions: 0.9.6, 1.0.2, 1.1.0, 1.0.3 >Reporter: Yuzhao Chen > Attachments: 2FA30CD8-AF15-4352-992D-A67BD724E7FB.png, > D4A30D40-25D5-4ACF-9A96-252EBA9E6EF6.png > > Time Spent: 0.5h
[jira] [Commented] (STORM-2693) Topology submission or kill takes too much time when topologies grow to a few hundred
[ https://issues.apache.org/jira/browse/STORM-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161132#comment-16161132 ] Jungtaek Lim commented on STORM-2693: - [~danny0405] Just to put some guides for contribution (sorry we didn't document it nicely): - all the patches are ideal to be against with master (currently 2.0.0) - merger or any committers can decide which version line(s) the patch should be put in, and port back if necessary - there's exceptional case: if patch addresses the bug which only resides on that version line(s) Let's back to your design. I haven't look into the detail, but I'm wondering how reading/storing from/to local disk will work with Nimbus H/A. Storm puts any critical data into zookeeper to ensure the data can be available at any time (when ZK is not available Storm will not work so...) This would be no longer true if we store it to only local disk. You would want to think about the cases which each Nimbus has its own assignments (which is not in sync) stored to local (only one of them is leader though) and all of them are shutting down and restarting at the same time. Which Nimbus should be a leader? How we ensure the elected Nimbus has the latest assignments? Btw, the idea brings me some thoughts: one of Storm's great point is stateless which can be achieved with stable storage (ZK for Storm), and we claims that Supervisor can work even Nimbus goes down. Now we introduced Nimbus H/A which makes Nimbus not a SPOF, and so some components like Supervisor may be able to rely on leader Nimbus instead of communicating with ZK if it is much faster and it doesn't put too much load to the leader Nimbus. (And this approach looks like same as your approach on your idea.) And since we only allow leader Nimbus to handle assignments, while we still need to write assignments to ZK, we can 'cache' it within leader Nimbus (and update cache if there's any change on assignments: note that we should also write to ZK earlier though) and avoid reading it from ZK. We still need to get heartbeat (and statistics) from ZK, but it will be handled as similar idea or another approach (like Metrics V2 in point of metrics' view) > Topology submission or kill takes too much time when topologies grow to a few > hundred > - > > Key: STORM-2693 > URL: https://issues.apache.org/jira/browse/STORM-2693 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Affects Versions: 0.9.6, 1.0.2, 1.1.0, 1.0.3 >Reporter: Yuzhao Chen > Attachments: 2FA30CD8-AF15-4352-992D-A67BD724E7FB.png, > D4A30D40-25D5-4ACF-9A96-252EBA9E6EF6.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > Now for a storm cluster with 40 hosts [with 32 cores/128G memory] and > hundreds of topologies, nimbus submission and killing will take about minutes > to finish. For example, for a cluster with 300 hundred of topologies，it will > take about 8 minutes to submit a topology, this affect our efficiency > seriously. > So, i check out the nimbus code and find two factor that will effect nimbus > submission/killing time for a scheduling round: > * read existing-assignments from zookeeper for every topology [will take > about 4 seconds for a 300 topologies cluster] > * read all the workers heartbeats and update the state to nimbus cache [will > take about 30 seconds for a 300 topologies cluster] > the key here is that Storm now use zookeeper to collect heartbeats [not RPC], > and also keep physical plan [assignments] using zookeeper which can be > totally local in nimbus. > So, i think we should make some changes to storm's heartbeats and assignments > management. > For assignment promotion: > 1. nimbus will put the assignments in local disk > 2. when restart or HA leader trigger nimbus will recover assignments from zk > to local disk > 3. nimbus will tell supervisor its assignment every time through RPC every > scheduling round > 4. supervisor will sync assignments at fixed time > For heartbeats promotion: > 1. workers will report executors ok or wrong to supervisor at fixed time > 2. supervisor will report workers heartbeats to nimbus at fixed time > 3. if supervisor die, it will tell nimbus through runtime hook > or let nimbus find it through aware supervisor if is survive > 4. let supervisor decide if worker is running ok or invalid , supervisor will > tell nimbus which executors of every topology are ok -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2693) Topology submission or kill takes too much time when topologies grow to a few hundred
[ https://issues.apache.org/jira/browse/STORM-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2693: Fix Version/s: (was: 1.1.0) > Topology submission or kill takes too much time when topologies grow to a few > hundred > - > > Key: STORM-2693 > URL: https://issues.apache.org/jira/browse/STORM-2693 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Affects Versions: 0.9.6, 1.0.2, 1.1.0, 1.0.3 >Reporter: Yuzhao Chen > Attachments: 2FA30CD8-AF15-4352-992D-A67BD724E7FB.png, > D4A30D40-25D5-4ACF-9A96-252EBA9E6EF6.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > Now for a storm cluster with 40 hosts [with 32 cores/128G memory] and > hundreds of topologies, nimbus submission and killing will take about minutes > to finish. For example, for a cluster with 300 hundred of topologies，it will > take about 8 minutes to submit a topology, this affect our efficiency > seriously. > So, i check out the nimbus code and find two factor that will effect nimbus > submission/killing time for a scheduling round: > * read existing-assignments from zookeeper for every topology [will take > about 4 seconds for a 300 topologies cluster] > * read all the workers heartbeats and update the state to nimbus cache [will > take about 30 seconds for a 300 topologies cluster] > the key here is that Storm now use zookeeper to collect heartbeats [not RPC], > and also keep physical plan [assignments] using zookeeper which can be > totally local in nimbus. > So, i think we should make some changes to storm's heartbeats and assignments > management. > For assignment promotion: > 1. nimbus will put the assignments in local disk > 2. when restart or HA leader trigger nimbus will recover assignments from zk > to local disk > 3. nimbus will tell supervisor its assignment every time through RPC every > scheduling round > 4. supervisor will sync assignments at fixed time > For heartbeats promotion: > 1. workers will report executors ok or wrong to supervisor at fixed time > 2. supervisor will report workers heartbeats to nimbus at fixed time > 3. if supervisor die, it will tell nimbus through runtime hook > or let nimbus find it through aware supervisor if is survive > 4. let supervisor decide if worker is running ok or invalid , supervisor will > tell nimbus which executors of every topology are ok -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2724) ExecutorService in WaterMarkEventGenerator never shutdown
[ https://issues.apache.org/jira/browse/STORM-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2724. - Resolution: Fixed Fix Version/s: 1.1.2 1.2.0 2.0.0 Merged into master, 1.x, 1.1.x branches. > ExecutorService in WaterMarkEventGenerator never shutdown > - > > Key: STORM-2724 > URL: https://issues.apache.org/jira/browse/STORM-2724 > Project: Apache Storm > Issue Type: Bug > Components: storm-client, storm-core >Affects Versions: 2.0.0, 1.0.4, 1.1.1 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim > Fix For: 2.0.0, 1.2.0, 1.1.2 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > I have seen a topology with event time windowing never terminated on local > mode. While looking into detail on thread dump I found only one non-daemon > thread prevents process to be not finished: executorService in > WaterMarkEventGenerator. > Btw, I dumped thread via jstack but impossible to find from jstack result > because it doesn't have thread factory hence thread name is pool-*. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2724) ExecutorService in WaterMarkEventGenerator never shutdown
[ https://issues.apache.org/jira/browse/STORM-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157015#comment-16157015 ] Jungtaek Lim commented on STORM-2724: - This issue affects all 1.x version lines. > ExecutorService in WaterMarkEventGenerator never shutdown > - > > Key: STORM-2724 > URL: https://issues.apache.org/jira/browse/STORM-2724 > Project: Apache Storm > Issue Type: Bug > Components: storm-client, storm-core >Affects Versions: 1.0.4, 1.1.1 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim > > I have seen a topology with event time windowing never terminated on local > mode. While looking into detail on thread dump I found only one non-daemon > thread prevents process to be not finished: executorService in > WaterMarkEventGenerator. > Btw, I dumped thread via jstack but impossible to find from jstack result > because it doesn't have thread factory hence thread name is pool-*. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (STORM-2724) ExecutorService in WaterMarkEventGenerator never shutdown
Jungtaek Lim created STORM-2724: --- Summary: ExecutorService in WaterMarkEventGenerator never shutdown Key: STORM-2724 URL: https://issues.apache.org/jira/browse/STORM-2724 Project: Apache Storm Issue Type: Bug Components: storm-client, storm-core Affects Versions: 1.0.4, 1.1.1 Reporter: Jungtaek Lim Assignee: Jungtaek Lim I have seen a topology with event time windowing never terminated on local mode. While looking into detail on thread dump I found only one non-daemon thread prevents process to be not finished: executorService in WaterMarkEventGenerator. Btw, I dumped thread via jstack but impossible to find from jstack result because it doesn't have thread factory hence thread name is pool-*. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (STORM-2273) Starting nimbus from arbitrary dir fails
[
https://issues.apache.org/jira/browse/STORM-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim closed STORM-2273.
---
Resolution: Duplicate
STORM-2660 covers this issue which is fixed.
> Starting nimbus from arbitrary dir fails
>
>
> Key: STORM-2273
> URL: https://issues.apache.org/jira/browse/STORM-2273
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Raghav Kumar Gautam
>
> Here is the output that I got:
> {code}
> storm@node1:/home/vagrant$ storm nimbus
> Running: java -server -Ddaemon.name=nimbus -Dstorm.options=
> -Dstorm.home=/usr/share/apache-storm-2.0.0-SNAPSHOT
> -Dstorm.log.dir=/usr/share/apache-storm-2.0.0-SNAPSHOT/logs
> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64
> -Dstorm.conf.file= -cp
> /usr/share/apache-storm-2.0.0-SNAPSHOT/lib/log4j-api-2.1.jar:/usr/share/apache-storm-2.0.0-SNAPSHOT/lib/log4j-slf4j-impl-2.1.jar:/usr/share/apache-storm-2.0.0-SNAPSHOT/lib/clojure-1.7.0.jar:/usr/share/apache-storm-2.0.0-SNAPSHOT/lib/slf4j-api-1.7.7.jar:/usr/share/apache-storm-2.0.0-SNAPSHOT/lib/asm-5.0.3.jar:/usr/share/apache-storm-2.0.0-SNAPSHOT/lib/minlog-1.3.0.jar:/usr/share/apache-storm-2.0.0-SNAPSHOT/lib/reflectasm-1.10.1.jar:/usr/share/apache-storm-2.0.0-SNAPSHOT/lib/log4j-over-slf4j-1.6.6.jar:/usr/share/apache-storm-2.0.0-SNAPSHOT/lib/log4j-core-2.1.jar:/usr/share/apache-storm-2.0.0-SNAPSHOT/lib/kryo-3.0.3.jar:/usr/share/apache-storm-2.0.0-SNAPSHOT/lib/storm-rename-hack-2.0.0-SNAPSHOT.jar:/usr/share/apache-storm-2.0.0-SNAPSHOT/lib/disruptor-3.3.2.jar:/usr/share/apache-storm-2.0.0-SNAPSHOT/lib/storm-core-2.0.0-SNAPSHOT.jar:/usr/share/apache-storm-2.0.0-SNAPSHOT/lib/objenesis-2.1.jar:/usr/share/apache-storm-2.0.0-SNAPSHOT/lib/servlet-api-2.5.jar:/usr/share/storm/conf
> -Xmx1024m -Dlogfile.name=nimbus.log
> -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector
>
> -Dlog4j.configurationFile=/usr/share/apache-storm-2.0.0-SNAPSHOT/log4j2/cluster.xml
> org.apache.storm.daemon.nimbus.Nimbus
> {code}
> Log added to nimbus.log.
> {code}
> 2017-01-04 21:44:33.089 main o.a.s.n.NimbusInfo [INFO] Nimbus figures out its
> name to node1
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (STORM-2717) Output topology Id from storm list command
[ https://issues.apache.org/jira/browse/STORM-2717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2717. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~ethanli], I merged into master. Please raise a PR against 1.x branch if you would also want to bring this to 1.x version line as well. > Output topology Id from storm list command > -- > > Key: STORM-2717 > URL: https://issues.apache.org/jira/browse/STORM-2717 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Reporter: Ethan Li >Assignee: Ethan Li >Priority: Trivial > Fix For: 2.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > I think we should output topology id from the storm list command for > debugging purpose. Currently we probably have to visit Storm UI to get the > topology Id. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2660) The Nimbus storm-local directory is relative to the working directory of the shell executing "storm nimbus"
[ https://issues.apache.org/jira/browse/STORM-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2660. - Resolution: Fixed Fix Version/s: 1.0.5 1.1.2 1.2.0 2.0.0 Merged into master, 1.x, 1.1.x, 1.0.x branches. > The Nimbus storm-local directory is relative to the working directory of the > shell executing "storm nimbus" > --- > > Key: STORM-2660 > URL: https://issues.apache.org/jira/browse/STORM-2660 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 2.0.0, 1.1.0, 1.0.4 > Environment: Windows 10, Ubuntu 17.04 >Reporter: Stig Rohde Døssing >Assignee: Jungtaek Lim > Fix For: 2.0.0, 1.2.0, 1.1.2, 1.0.5 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > When the storm.local.dir property is set to a relative directory, it should > be interpreted as relative to STORM_HOME. This is how it works for "storm > supervisor". For "storm nimbus" it is instead relative to the working > directory of the shell, so running "storm nimbus" from STORM_HOME/bin will > put a storm-local directory in STORM_HOME/bin/storm-local. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (STORM-2533) Visualization API returns "spout" for system components
[
https://issues.apache.org/jira/browse/STORM-2533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim reassigned STORM-2533:
---
Assignee: Jungtaek Lim
> Visualization API returns "spout" for system components
> ---
>
> Key: STORM-2533
> URL: https://issues.apache.org/jira/browse/STORM-2533
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Stephen Powis
>Assignee: Jungtaek Lim
>
> Hitting the visualization api end point (
> /api/v1/topology/TOPOLOGY-ID/visualization ) returns system components
> labeled as spouts. These should not be labeled as spouts, but instead
> 'system' or some other more appropriate label.
> See the following example response:
> {code}
> {
> ...
> "__system": {
> ":type": "spout",
> ...
> },
> "__acker": {
> ":type": "spout",
>...
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (STORM-2534) Visualization API missing stats/instances for "system" components
[
https://issues.apache.org/jira/browse/STORM-2534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim reassigned STORM-2534:
---
Assignee: Jungtaek Lim
> Visualization API missing stats/instances for "system" components
> -
>
> Key: STORM-2534
> URL: https://issues.apache.org/jira/browse/STORM-2534
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Stephen Powis
>Assignee: Jungtaek Lim
>
> The topology visualization api end point (
> /api/v1/topology/TOPOLOGY-ID/visualization ) does not return correct "stats"
> values for "system" components __system and __acker.
> See the following example *correct* response for a spout or bolt within a
> topology, shorten for brevity. Under the stats key it lists all of the
> instances of that component that is deployed.
> {code}
> {
> "spout": {
> ...
> ":stats": [{
> ":host":
> "e54bb273-2a8a-4320-b23f-7c7ace52c961-10.153.0.30",
> ":port": 6700,
> ":uptime_secs": 0,
> ":transferred": {
> ...
> }
> }],
> ...
> },
> {code}
> See the following response for the __system and __acker components. They do
> *not* correctly list any entries under the stats key.
> {code}
> {
> "__system": {
> ":type": "spout",
> ":capacity": 0,
> ":latency": null,
> ":transferred": null,
> ":stats": [],
> ":link":
> "\/component.html?id=__system_id=test-1-1495630798",
> ":inputs": []
> },
> "__acker": {
> ":type": "spout",
> ":capacity": 0,
> ":latency": null,
> ":transferred": null,
> ":stats": [],
> ":link":
> "\/component.html?id=__acker_id=test-1-1495630798",
> ":inputs": [...]
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (STORM-2445) Topology log search refers supervisor ID as host of worker which contains UUID
[ https://issues.apache.org/jira/browse/STORM-2445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned STORM-2445: --- Assignee: Jungtaek Lim > Topology log search refers supervisor ID as host of worker which contains UUID > -- > > Key: STORM-2445 > URL: https://issues.apache.org/jira/browse/STORM-2445 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim > Attachments: Screen Shot 2017-03-31 at 5.38.07 PM.png > > Time Spent: 10m > Remaining Estimate: 0h > > It seems to take supervisor ID as worker's host in topology log search, so > API request to logviewer fails. > Please check attachment. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (STORM-2660) The Nimbus storm-local directory is relative to the working directory of the shell executing "storm nimbus"
[ https://issues.apache.org/jira/browse/STORM-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned STORM-2660: --- Assignee: Jungtaek Lim > The Nimbus storm-local directory is relative to the working directory of the > shell executing "storm nimbus" > --- > > Key: STORM-2660 > URL: https://issues.apache.org/jira/browse/STORM-2660 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 2.0.0, 1.1.0, 1.0.4 > Environment: Windows 10, Ubuntu 17.04 >Reporter: Stig Rohde Døssing >Assignee: Jungtaek Lim > Time Spent: 10m > Remaining Estimate: 0h > > When the storm.local.dir property is set to a relative directory, it should > be interpreted as relative to STORM_HOME. This is how it works for "storm > supervisor". For "storm nimbus" it is instead relative to the working > directory of the shell, so running "storm nimbus" from STORM_HOME/bin will > put a storm-local directory in STORM_HOME/bin/storm-local. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2443) Nimbus throws error when changing log level on UI topology page
[
https://issues.apache.org/jira/browse/STORM-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-2443:
Fix Version/s: 2.0.0
> Nimbus throws error when changing log level on UI topology page
> ---
>
> Key: STORM-2443
> URL: https://issues.apache.org/jira/browse/STORM-2443
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
> Fix For: 2.0.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Here's stacktrace from Nimbus log:
> {code}
> 2017-03-30 16:53:26.954 o.a.s.d.n.Nimbus pool-14-thread-56 [WARN] set log
> config topology exception. (topology id='rolling-1-1490860365')
> java.lang.NullPointerException: null
> at
> org.apache.storm.daemon.nimbus.Nimbus.setLogConfig(Nimbus.java:2688)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.generated.Nimbus$Processor$setLogConfig.getResult(Nimbus.java:3295)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.generated.Nimbus$Processor$setLogConfig.getResult(Nimbus.java:3280)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:160)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_66]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_66]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_66]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (STORM-2660) The Nimbus storm-local directory is relative to the working directory of the shell executing "storm nimbus"
[ https://issues.apache.org/jira/browse/STORM-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148477#comment-16148477 ] Jungtaek Lim commented on STORM-2660: - [~Srdo] Thanks for clarifying. I guess we have related issue before as well (though the summary is less descriptive) so would like to see two's relevance. https://issues.apache.org/jira/browse/STORM-2273 > The Nimbus storm-local directory is relative to the working directory of the > shell executing "storm nimbus" > --- > > Key: STORM-2660 > URL: https://issues.apache.org/jira/browse/STORM-2660 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 2.0.0, 1.1.0, 1.0.4 > Environment: Windows 10, Ubuntu 17.04 >Reporter: Stig Rohde Døssing > > When the storm.local.dir property is set to a relative directory, it should > be interpreted as relative to STORM_HOME. This is how it works for "storm > supervisor". For "storm nimbus" it is instead relative to the working > directory of the shell, so running "storm nimbus" from STORM_HOME/bin will > put a storm-local directory in STORM_HOME/bin/storm-local. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (STORM-2443) Nimbus throws error when changing log level on UI topology page
[
https://issues.apache.org/jira/browse/STORM-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim reassigned STORM-2443:
---
Assignee: Jungtaek Lim
> Nimbus throws error when changing log level on UI topology page
> ---
>
> Key: STORM-2443
> URL: https://issues.apache.org/jira/browse/STORM-2443
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> Here's stacktrace from Nimbus log:
> {code}
> 2017-03-30 16:53:26.954 o.a.s.d.n.Nimbus pool-14-thread-56 [WARN] set log
> config topology exception. (topology id='rolling-1-1490860365')
> java.lang.NullPointerException: null
> at
> org.apache.storm.daemon.nimbus.Nimbus.setLogConfig(Nimbus.java:2688)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.generated.Nimbus$Processor$setLogConfig.getResult(Nimbus.java:3295)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.generated.Nimbus$Processor$setLogConfig.getResult(Nimbus.java:3280)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:160)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_66]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_66]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_66]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (STORM-2253) Storm PMML Bolt - Unit Tests
[
https://issues.apache.org/jira/browse/STORM-2253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-2253:
Issue Type: Test (was: Bug)
> Storm PMML Bolt - Unit Tests
>
>
> Key: STORM-2253
> URL: https://issues.apache.org/jira/browse/STORM-2253
> Project: Apache Storm
> Issue Type: Test
> Components: storm-pmml
>Affects Versions: 2.0.0
>Reporter: Hugo Louro
>Assignee: Hugo Louro
>
> Currently the patch has integration tests through the form of a test topology
> that loads a PMML Model and raw input data from a CSV file. The {@link
> RawInputFromCSVSpout}
> creates a stream of tuples with raw inputs, and the {@link PMMLPredictorBolt}
> computes the predicted scores.
> The main focus of the initial patch was to design the classes in such a way
> that they can accommodate arbitrary runtime environments. The default
> implementation provided uses one such runtime execution library, which is
> more suited to be tested using integration tests.
> Will add some unit tests around to assert for edge and some common cases
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (STORM-2714) Release Apache Storm 2.0.0
Jungtaek Lim created STORM-2714: --- Summary: Release Apache Storm 2.0.0 Key: STORM-2714 URL: https://issues.apache.org/jira/browse/STORM-2714 Project: Apache Storm Issue Type: Epic Reporter: Jungtaek Lim This is to track remaining issues on releasing Storm 2.0.0. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-1555) Required field 'topology_id' is unset! seen failing integration-test in Travis CI Test
[
https://issues.apache.org/jira/browse/STORM-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-1555.
-
Resolution: Cannot Reproduce
I don't see the issue from recent PR builds against master. Closing for now. We
could reopen if we find one.
> Required field 'topology_id' is unset! seen failing integration-test in
> Travis CI Test
> --
>
> Key: STORM-1555
> URL: https://issues.apache.org/jira/browse/STORM-1555
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Derek Dagit
>
> A recent travis ci
> [error|https://travis-ci.org/apache/storm/jobs/109239992#L1570] was seen.
> Here is the stack trace:
> {noformat}
> 107323 [main] ERROR i.o.a.s.t.integration-test - Error in cluster
> java.lang.RuntimeException: org.apache.thrift.protocol.TProtocolException:
> Required field 'topology_id' is unset! Struct:LSTopoHistory(topology_id:null,
> time_stamp:1455491275, users:[], groups:[])
> at org.apache.storm.utils.LocalState.serialize(LocalState.java:186)
> ~[classes/:?]
> at org.apache.storm.utils.LocalState.put(LocalState.java:142)
> ~[classes/:?]
> at org.apache.storm.utils.LocalState.put(LocalState.java:136)
> ~[classes/:?]
> at
> org.apache.storm.local_state$ls_topo_hist_BANG_.invoke(local_state.clj:48)
> ~[classes/:?]
> at
> org.apache.storm.daemon.nimbus$add_topology_to_history_log.invoke(nimbus.clj:1279)
> ~[classes/:?]
> at
> org.apache.storm.daemon.nimbus$fn__4836$exec_fn__1827__auto__$reify__4865.killTopologyWithOpts(nimbus.clj:1587)
> ~[classes/:?]
> at sun.reflect.GeneratedMethodAccessor298.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.7.0_76]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_76]
> at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
> ~[clojure-1.7.0.jar:?]
> at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
> ~[clojure-1.7.0.jar:?]
> at
> org.apache.storm.trident.testing$with_topology_STAR_.invoke(testing.clj:62)
> ~[classes/:?]
> at
> integration.org.apache.storm.trident.integration_test$fn__5251.invoke(integration_test.clj:149)
> [?:?]
> at clojure.test$test_var$fn__7670.invoke(test.clj:704)
> [clojure-1.7.0.jar:?]
> at clojure.test$test_var.invoke(test.clj:704) [clojure-1.7.0.jar:?]
> at clojure.test$test_vars$fn__7692$fn__7697.invoke(test.clj:722)
> [clojure-1.7.0.jar:?]
> at clojure.test$default_fixture.invoke(test.clj:674)
> [clojure-1.7.0.jar:?]
> at clojure.test$test_vars$fn__7692.invoke(test.clj:722)
> [clojure-1.7.0.jar:?]
> at clojure.test$default_fixture.invoke(test.clj:674)
> [clojure-1.7.0.jar:?]
> at clojure.test$test_vars.invoke(test.clj:718) [clojure-1.7.0.jar:?]
> at clojure.test$test_all_vars.invoke(test.clj:728) [clojure-1.7.0.jar:?]
> at clojure.test$test_ns.invoke(test.clj:747) [clojure-1.7.0.jar:?]
> at clojure.core$map$fn__4553.invoke(core.clj:2624) [clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.sval(LazySeq.java:40) [clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.seq(LazySeq.java:49) [clojure-1.7.0.jar:?]
> at clojure.lang.Cons.next(Cons.java:39) [clojure-1.7.0.jar:?]
> at clojure.lang.RT.boundedLength(RT.java:1735) [clojure-1.7.0.jar:?]
> at clojure.lang.RestFn.applyTo(RestFn.java:130) [clojure-1.7.0.jar:?]
> at clojure.core$apply.invoke(core.clj:632) [clojure-1.7.0.jar:?]
> at clojure.test$run_tests.doInvoke(test.clj:762) [clojure-1.7.0.jar:?]
> at clojure.lang.RestFn.invoke(RestFn.java:408) [clojure-1.7.0.jar:?]
> at
> org.apache.storm.testrunner$eval10993$iter__10994__10998$fn__10999$fn__11000$fn__11001.invoke(test_runner.clj:107)
> [?:?]
> at
> org.apache.storm.testrunner$eval10993$iter__10994__10998$fn__10999$fn__11000.invoke(test_runner.clj:53)
> [?:?]
> at
> org.apache.storm.testrunner$eval10993$iter__10994__10998$fn__10999.invoke(test_runner.clj:52)
> [?:?]
> at clojure.lang.LazySeq.sval(LazySeq.java:40) [clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.seq(LazySeq.java:49) [clojure-1.7.0.jar:?]
> at clojure.lang.RT.seq(RT.java:507) [clojure-1.7.0.jar:?]
> at clojure.core$seq__4128.invoke(core.clj:137) [clojure-1.7.0.jar:?]
> at clojure.core$dorun.invoke(core.clj:3009) [clojure-1.7.0.jar:?]
> at org.apache.storm.testrunner$eval10993.invoke(test_runner.clj:52)
> [?:?]
> at clojure.lang.Compiler.eval(Compiler.java:6782) [clojure-1.7.0.jar:?]
> at clojure.lang.Compiler.load(Compiler.java:7227) [clojure-1.7.0.jar:?]
> at
[jira] [Resolved] (STORM-2071) nimbus-test test-leadership failing with Exception
[
https://issues.apache.org/jira/browse/STORM-2071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-2071.
-
Resolution: Fixed
Fix Version/s: 1.0.3
1.1.0
2.0.0
Covered by STORM-2018.
> nimbus-test test-leadership failing with Exception
> --
>
> Key: STORM-2071
> URL: https://issues.apache.org/jira/browse/STORM-2071
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.0.0, 1.0.1
> Environment: Mac os X
>Reporter: Paul Poulosky
>Assignee: Robert Joseph Evans
> Fix For: 2.0.0, 1.1.0, 1.0.3
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> When running unit tests on my Mac, I get repeated failures in test-leadership.
>
> 73752 [main] INFO o.a.s.l.ThriftAccessLogger - Request ID: 1 access from:
> null principal: null operation: deactivate
> ]]>
> Uncaught
> exception, not in assertion.
> expected: nil
> actual: java.lang.RuntimeException: No transition for event: :inactivate,
> status: {:type :rebalancing} storm-id: t1-1-1472598899
> at org.apache.storm.daemon.nimbus$transition_BANG_$get_event__4879.invoke
> (nimbus.clj:365)
> org.apache.storm.daemon.nimbus$transition_BANG_.invoke (nimbus.clj:373)
> clojure.lang.AFn.applyToHelper (AFn.java:165)
> clojure.lang.AFn.applyTo (AFn.java:144)
> clojure.core$apply.invoke (core.clj:636)
> org.apache.storm.daemon.nimbus$transition_name_BANG_.doInvoke
> (nimbus.clj:391)
> clojure.lang.RestFn.invoke (RestFn.java:467)
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__5850.deactivate
> (nimbus.clj:1773)
> sun.reflect.NativeMethodAccessorImpl.invoke0
> (NativeMethodAccessorImpl.java:-2)
> sun.reflect.NativeMethodAccessorImpl.invoke
> (NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke (Method.java:497)
> clojure.lang.Reflector.invokeMatchingMethod (Reflector.java:93)
> clojure.lang.Reflector.invokeInstanceMethod (Reflector.java:28)
> org.apache.storm.nimbus_test$fn__1203$fn__1209.invoke
> (nimbus_test.clj:1222)
> org.apache.storm.nimbus_test/fn (nimbus_test.clj:1210)
> clojure.test$test_var$fn__7670.invoke (test.clj:704)
> clojure.test$test_var.invoke (test.clj:704)
> clojure.test$test_vars$fn__7692$fn__7697.invoke (test.clj:722)
> clojure.test$default_fixture.invoke (test.clj:674)
> clojure.test$test_vars$fn__7692.invoke (test.clj:722)
> clojure.test$default_fixture.invoke (test.clj:674)
> clojure.test$test_vars.invoke (test.clj:718)
> clojure.test$test_all_vars.invoke (test.clj:728)
> (test.clj:747)
> clojure.core$map$fn__4553.invoke (core.clj:2624)
> clojure.lang.LazySeq.sval (LazySeq.java:40)
> clojure.lang.LazySeq.seq (LazySeq.java:49)
> clojure.lang.Cons.next (Cons.java:39)
> clojure.lang.RT.boundedLength (RT.java:1735)
> clojure.lang.RestFn.applyTo (RestFn.java:130)
> clojure.core$apply.invoke (core.clj:632)
> clojure.test$run_tests.doInvoke (test.clj:762)
> clojure.lang.RestFn.invoke (RestFn.java:408)
>
> org.apache.storm.testrunner$eval8358$iter__8359__8363$fn__8364$fn__8365$fn__8366.invoke
> (test_runner.clj:107)
>
> org.apache.storm.testrunner$eval8358$iter__8359__8363$fn__8364$fn__8365.invoke
> (test_runner.clj:53)
> org.apache.storm.testrunner$eval8358$iter__8359__8363$fn__8364.invoke
> (test_runner.clj:52)
> clojure.lang.LazySeq.sval (LazySeq.java:40)
> clojure.lang.LazySeq.seq (LazySeq.java:49)
> clojure.lang.RT.seq (RT.java:507)
> clojure.core/seq (core.clj:137)
> clojure.core$dorun.invoke (core.clj:3009)
> org.apache.storm.testrunner$eval8358.invoke (test_runner.clj:52)
> clojure.lang.Compiler.eval (Compiler.java:6782)
> clojure.lang.Compiler.load (Compiler.java:7227)
> clojure.lang.Compiler.loadFile (Compiler.java:7165)
> clojure.main$load_script.invoke (main.clj:275)
> clojure.main$script_opt.invoke (main.clj:337)
> clojure.main$main.doInvoke (main.clj:421)
> clojure.lang.RestFn.invoke (RestFn.java:421)
> clojure.lang.Var.invoke (Var.java:383)
> clojure.lang.AFn.applyToHelper (AFn.java:156)
> clojure.lang.Var.applyTo (Var.java:700)
> clojure.main.main (main.java:37)
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (STORM-2265) Incorrectly Serialized JSON in TransactionalState causes Worker to Die
[ https://issues.apache.org/jira/browse/STORM-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim closed STORM-2265. --- Resolution: Won't Fix Closing as we don't have any better approaches for now. > Incorrectly Serialized JSON in TransactionalState causes Worker to Die > -- > > Key: STORM-2265 > URL: https://issues.apache.org/jira/browse/STORM-2265 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 2.0.0, 1.1.0, 1.0.3 >Reporter: Hugo Louro >Assignee: Hugo Louro >Priority: Critical > Attachments: Screen Shot 2016-12-28 at 12.10.59 AM.png > > Time Spent: 50m > Remaining Estimate: 0h > > TransactionalState uses JSONValue to serialize / deserialize objects. > However, the object GlobalPartitionInformation is incorrectly serialized by > default, causing the exception bellow. To get around this problem, > GlobalPartitionInformation must implement JSONAware. > 2016-12-23 14:37:26.980 o.a.s.e.e.ReportError > Thread-21-$spoutcoord-spout-spout1-executor[2, 2] [ERROR] Error > java.lang.RuntimeException: java.lang.RuntimeException: Unexpected character > (G) at position 1. > at org.apache.storm.utils.Utils$6.run(Utils.java:2190) > ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] > Caused by: java.lang.RuntimeException: Unexpected character (G) at position 1. > at > org.apache.storm.trident.topology.state.TransactionalState.getData(TransactionalState.java:174) > ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at > org.apache.storm.trident.topology.state.RotatingTransactionalState.sync(RotatingTransactionalState.java:165) > ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at > org.apache.storm.trident.topology.state.RotatingTransactionalState.(RotatingTransactionalState.java:46) > ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at > org.apache.storm.trident.spout.TridentSpoutCoordinator.prepare(TridentSpoutCoordinator.java:57) > ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at > org.apache.storm.topology.BasicBoltExecutor.prepare(BasicBoltExecutor.java:43) > ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at > org.apache.storm.executor.bolt.BoltExecutor.init(BoltExecutor.java:84) > ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at > org.apache.storm.executor.bolt.BoltExecutor.call(BoltExecutor.java:93) > ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at > org.apache.storm.executor.bolt.BoltExecutor.call(BoltExecutor.java:45) > ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at org.apache.storm.utils.Utils$6.run(Utils.java:2179) > ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > ... 1 more > Caused by: org.apache.storm.shade.org.json.simple.parser.ParseException > at org.apache.storm.shade.org.json.simple.parser.Yylex.yylex(Unknown > Source) ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at > org.apache.storm.shade.org.json.simple.parser.JSONParser.nextToken(Unknown > Source) ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at > org.apache.storm.shade.org.json.simple.parser.JSONParser.parse(Unknown > Source) ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at > org.apache.storm.shade.org.json.simple.parser.JSONParser.parse(Unknown > Source) ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at > org.apache.storm.shade.org.json.simple.parser.JSONParser.parse(Unknown > Source) ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at > org.apache.storm.shade.org.json.simple.JSONValue.parseWithException(Unknown > Source) ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at > org.apache.storm.trident.topology.state.TransactionalState.getData(TransactionalState.java:167) > ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at > org.apache.storm.trident.topology.state.RotatingTransactionalState.sync(RotatingTransactionalState.java:165) > ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at > org.apache.storm.trident.topology.state.RotatingTransactionalState.(RotatingTransactionalState.java:46) > ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at > org.apache.storm.trident.spout.TridentSpoutCoordinator.prepare(TridentSpoutCoordinator.java:57) > ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at > org.apache.storm.topology.BasicBoltExecutor.prepare(BasicBoltExecutor.java:43) > ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at > org.apache.storm.executor.bolt.BoltExecutor.init(BoltExecutor.java:84) > ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] > at >
[jira] [Updated] (STORM-2288) Nimbus client can timeout in log running tests
[ https://issues.apache.org/jira/browse/STORM-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2288: Fix Version/s: (was: 2.0.0) > Nimbus client can timeout in log running tests > -- > > Key: STORM-2288 > URL: https://issues.apache.org/jira/browse/STORM-2288 > Project: Apache Storm > Issue Type: Bug > Components: integration-test >Affects Versions: 2.0.0 >Reporter: Raghav Kumar Gautam >Assignee: Raghav Kumar Gautam > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2329) Topology halts when getting HDFS writer in a secure environment
[ https://issues.apache.org/jira/browse/STORM-2329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2329: Fix Version/s: (was: 2.0.0) > Topology halts when getting HDFS writer in a secure environment > --- > > Key: STORM-2329 > URL: https://issues.apache.org/jira/browse/STORM-2329 > Project: Apache Storm > Issue Type: Bug > Components: storm-hdfs >Affects Versions: 0.10.1, 2.0.0, 1.x > Environment: Kerberos >Reporter: Kristopher Kane > Labels: patch > Attachments: STORM-2329.patch > > > Simple topologies writing to Kerberized HDFS will sometimes stop while > getting a new writer (storm-hdfs) in a Kerberized environment: > java.io.IOException: Failed on local exception: java.io.IOException: Couldn't > setup connection for principal@realm to nn1/nn1IP; Host Details : local host > is: "hostname/ip"; destination host is: "nn hostname":8020; at > org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772) at > org.apache.hadoop.ipc.Client.call(Client.java:1473) at > org.apache.hadoop.ipc.Client.call(Client.java:1400) at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) > at com.sun.proxy.$Proxy26.create(Unknown Source) at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296) > at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy27.create(Unknown Source) at > org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1726) > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1668) at > org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1593) at > org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:397) > at > org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:393) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:393) > at > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:337) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908) at > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889) at > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786) at > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:775) at > org.apache.storm.hdfs.bolt.AvroGenericRecordBolt.makeNewWriter(AvroGenericRecordBolt.java:115) > at > org.apache.storm.hdfs.bolt.AbstractHdfsBolt.getOrCreateWriter(AbstractHdfsBolt.java:222) > at > org.apache.storm.hdfs.bolt.AbstractHdfsBolt.execute(AbstractHdfsBolt.java:154) > at > backtype.storm.daemon.executor$fn_3697$tuple_action_fn3699.invoke(executor.clj:670) > at > backtype.storm.daemon.executor$mk_task_receiver$fn3620.invoke(executor.clj:426) > at > backtype.storm.disruptor$clojure_handler$reify3196.onEvent(disruptor.clj:58) > at > backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) > at > backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) > at > backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) > at > backtype.storm.daemon.executor$fn3697$fn3710$fn3761.invoke(executor.clj:808) > at backtype.storm.util$async_loop$fn_544.invoke(util.clj:475) at > clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: Couldn't setup connection for principal@realm > to nn1/nn1IP:8020 at > org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:673) at > java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at > org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:644) > at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:731) > at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:369) at > org.apache.hadoop.ipc.Client.getConnection(Client.java:1522) at > org.apache.hadoop.ipc.Client.call(Client.java:1439) ... 35 more Caused by: > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos
[jira] [Closed] (STORM-1711) Kerberos principals gets mixed up while using storm-hive
[ https://issues.apache.org/jira/browse/STORM-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim closed STORM-1711. --- Resolution: Won't Fix The patch couldn't fix the issue and we have no following patch. Closing for now. > Kerberos principals gets mixed up while using storm-hive > > > Key: STORM-1711 > URL: https://issues.apache.org/jira/browse/STORM-1711 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 1.0.0, 2.0.0 >Reporter: Arun Mahadevan >Assignee: Arun Mahadevan > Time Spent: 20m > Remaining Estimate: 0h > > Storm-hive uses UserGroupInformation.loginUserFromKeytab which updates the > static variable that stores current UGI. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-1767) metrics log entries are being appended to root log
[
https://issues.apache.org/jira/browse/STORM-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-1767:
Fix Version/s: (was: 2.0.0)
(was: 1.0.0)
> metrics log entries are being appended to root log
> --
>
> Key: STORM-1767
> URL: https://issues.apache.org/jira/browse/STORM-1767
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Daniel Drozdzewski
> Labels: easyfix
>
> Current setup of metrics logger ( {{storm/log4j2/worker.xml}}) uses fully
> qualified name of the class where the logging is happening from i.e
> `org.apache.storm.metric.LoggingMetricsConsumer`, which is problematic and
> does not achieve the original intent as stated by the METRICS appender
> defined in {{storm/log4j2/worker.xml}}.
> Currently the metrics logger created explicitly by using the name above:
> {{LoggerFactory.getLogger("org.apache.storm.metric.LoggingMetricsConsumer")}}
> or implicitly from within the {{LoggingMetricsConsumer}} by calling
> {{LoggerFactory.getLogger(LoggingMetricsConsumer.class)}} will be logging to
> **root** logger.
> This happens because logger names use Java namespaces and as such create
> hierarchies.
> The solution is to name metrics logger outside of {{org.apache.storm.*}}
> namespace which is what is happening for all other non-root loggers defined
> within the {{storm/log4j2/worker.xml}} file.
> This will also mean a code change to {{LoggingMetricsConsumer}} class itself
> for it to use the logger with an explicit name matching the name defined in
> the {{worker.xml}} file.
> The fix is easy.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (STORM-1501) launch worker process exception will cause supervisor process exited
[ https://issues.apache.org/jira/browse/STORM-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim closed STORM-1501. --- Resolution: Won't Fix Closing due to inactivity. Looks like that patch would not be expected to pass the vote. > launch worker process exception will cause supervisor process exited > - > > Key: STORM-1501 > URL: https://issues.apache.org/jira/browse/STORM-1501 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 2.0.0 >Reporter: caofangkun >Assignee: caofangkun >Priority: Trivial > > [util.clj/async-loop | > https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/util.clj#L474] > default kill-fn will kill current process > when supervisor use [util.clj/launch-process | > https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/util.clj#L546] > to launch worker process , if exeception occurs , supervisor process will > exit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2693) Topology submission or kill takes too much time when topologies grow to a few hundred
[ https://issues.apache.org/jira/browse/STORM-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146533#comment-16146533 ] Jungtaek Lim commented on STORM-2693: - [~danny0405] Yes totally agreed. I think some of us are trying to make huge change to metrics (named as Metrics V2) but looks like it requires more time to come into realize. > Topology submission or kill takes too much time when topologies grow to a few > hundred > - > > Key: STORM-2693 > URL: https://issues.apache.org/jira/browse/STORM-2693 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Affects Versions: 0.9.6, 1.0.2, 1.1.0, 1.0.3 >Reporter: Yuzhao Chen > Fix For: 1.1.0 > > Attachments: 2FA30CD8-AF15-4352-992D-A67BD724E7FB.png, > D4A30D40-25D5-4ACF-9A96-252EBA9E6EF6.png > > > Now for a storm cluster with 40 hosts [with 32 cores/128G memory] and > hundreds of topologies, nimbus submission and killing will take about minutes > to finish. For example, for a cluster with 300 hundred of topologies，it will > take about 8 minutes to submit a topology, this affect our efficiency > seriously. > So, i check out the nimbus code and find two factor that will effect nimbus > submission/killing time for a scheduling round: > * read existing-assignments from zookeeper for every topology [will take > about 4 seconds for a 300 topologies cluster] > * read all the workers heartbeats and update the state to nimbus cache [will > take about 30 seconds for a 300 topologies cluster] > the key here is that Storm now use zookeeper to collect heartbeats [not RPC], > and also keep physical plan [assignments] using zookeeper which can be > totally local in nimbus. > So, i think we should make some changes to storm's heartbeats and assignments > management. > For assignment promotion: > 1. nimbus will put the assignments in memory also cache a copy to > local disk > 2. when restart or HA leader trigger nimbus will recover > assignments from disk > 3. nimbus will tell supervisor its assignment every time scheduling > 4. supervisor will sync assignments at fixed time > For heartbeats promotion: > 1. workers will report executors ok or wrong to supervisor at fixed time > 2. supervisor will report workers heartbeats to nimbus at fixed time > 3. if supervisor die, it will tell nimbus through runtime hook > or let nimbus find it through aware supervisor if survive > 4. let supervisor decide if worker is running ok or invalid , supervisor will > tell nimbus which executors of every topology are ok -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2521) "storm sql" fails since '--jars' can't handle wildcard
[ https://issues.apache.org/jira/browse/STORM-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2521. - Resolution: Fixed Fix Version/s: 1.2.0 2.0.0 I fixed the issue in STORM-2598: marking as resolved. https://github.com/apache/storm/commit/c2477520c18ee56401d2517b09e372b6d58df559 > "storm sql" fails since '--jars' can't handle wildcard > -- > > Key: STORM-2521 > URL: https://issues.apache.org/jira/browse/STORM-2521 > Project: Apache Storm > Issue Type: Bug > Components: storm-sql >Affects Versions: 2.0.0, 1.2.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Critical > Fix For: 2.0.0, 1.2.0 > > > STORM-2191 changes the approach to include jars for classpath: before the > patch, storm script lists jars in directory and add them to classpath one by > one. After the patch, we just use wildcard which is great to shorten the > classpath. > storm-sql-runtime jars use '--jars' to be uploaded to blobstore and '--jars' > doesn't support wildcard. So unfortunately, STORM-2191 breaks "storm sql". > We should choose whether making --jars supporting wildcard, or have an > exceptional case, lists jars in directory only for above case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2521) "storm sql" fails since '--jars' can't handle wildcard
[ https://issues.apache.org/jira/browse/STORM-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2521: Affects Version/s: (was: 1.1.1) 1.2.0 > "storm sql" fails since '--jars' can't handle wildcard > -- > > Key: STORM-2521 > URL: https://issues.apache.org/jira/browse/STORM-2521 > Project: Apache Storm > Issue Type: Bug > Components: storm-sql >Affects Versions: 2.0.0, 1.2.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Critical > > STORM-2191 changes the approach to include jars for classpath: before the > patch, storm script lists jars in directory and add them to classpath one by > one. After the patch, we just use wildcard which is great to shorten the > classpath. > storm-sql-runtime jars use '--jars' to be uploaded to blobstore and '--jars' > doesn't support wildcard. So unfortunately, STORM-2191 breaks "storm sql". > We should choose whether making --jars supporting wildcard, or have an > exceptional case, lists jars in directory only for above case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2522) examples in package do not build with checkstyle issues
[
https://issues.apache.org/jira/browse/STORM-2522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-2522.
-
Resolution: Fixed
Assignee: Jungtaek Lim
Fix Version/s: 2.0.0
Fixed via https://github.com/apache/storm/pull/2274
> examples in package do not build with checkstyle issues
> ---
>
> Key: STORM-2522
> URL: https://issues.apache.org/jira/browse/STORM-2522
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Robert Joseph Evans
>Assignee: Jungtaek Lim
> Fix For: 2.0.0
>
>
> {code}
> $ mvn clean package
> [INFO] Scanning for projects...
> [INFO]
> [INFO]
>
> [INFO] Building storm-starter 2.0.0-SNAPSHOT
> [INFO]
>
> [INFO]
> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ storm-starter ---
> [INFO]
> [INFO] --- maven-clean-plugin:2.5:clean (cleanup) @ storm-starter ---
> [INFO]
> [INFO] --- maven-checkstyle-plugin:2.17:check (validate) @ storm-starter ---
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 1.468 s
> [INFO] Finished at: 2017-05-18T15:58:20-05:00
> [INFO] Final Memory: 26M/437M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-checkstyle-plugin:2.17:check (validate) on
> project storm-starter: Failed during checkstyle execution: Unable to find
> configuration file at location: storm-buildtools/storm_checkstyle.xml: Could
> not find resource 'storm-buildtools/storm_checkstyle.xml'. -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (STORM-1744) Missing javadoc in Trident code
[ https://issues.apache.org/jira/browse/STORM-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-1744: Issue Type: Documentation (was: Bug) > Missing javadoc in Trident code > --- > > Key: STORM-1744 > URL: https://issues.apache.org/jira/browse/STORM-1744 > Project: Apache Storm > Issue Type: Documentation > Components: documentation, trident >Affects Versions: 1.0.0, 0.9.6, 0.10.1, 2.0.0 >Reporter: Samuel Nelson > > Some or most of the core Trident classes don't have javadoc. It makes it > really difficult to use. > http://storm.apache.org/releases/2.0.0-SNAPSHOT/javadocs/index.html > Examples: > TridentTopologyBuilder > IBackingMap -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2201) Dynamic scheduler configuration loader
[ https://issues.apache.org/jira/browse/STORM-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2201. - Resolution: Fixed Assignee: Ethan Li (was: Paul Poulosky) Fix Version/s: 2.0.0 Thanks [~ethanli], I merged into master. Also thanks also [~ppoulosk] to provide earlier patches before Ethan starts working on this. > Dynamic scheduler configuration loader > -- > > Key: STORM-2201 > URL: https://issues.apache.org/jira/browse/STORM-2201 > Project: Apache Storm > Issue Type: Improvement >Reporter: Paul Poulosky >Assignee: Ethan Li > Fix For: 2.0.0 > > Original Estimate: 168h > Time Spent: 13.5h > Remaining Estimate: 154.5h > > It would be useful if scheduler configuration for multitenant or resource > aware scheduler could be loaded and updated dynamically through a local file > change or through an update to a configuration archive in an Artifactory > repository. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (STORM-2709) Release Apache Storm 1.1.2
Jungtaek Lim created STORM-2709: --- Summary: Release Apache Storm 1.1.2 Key: STORM-2709 URL: https://issues.apache.org/jira/browse/STORM-2709 Project: Apache Storm Issue Type: Epic Reporter: Jungtaek Lim This is to track remaining issues on releasing Storm 1.1.2. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2231) NULL in DisruptorQueue while multi-threaded ack
[ https://issues.apache.org/jira/browse/STORM-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2231. - Resolution: Fixed Fix Version/s: 1.0.5 1.1.2 1.2.0 2.0.0 Merged to master, 1.x, 1.1.x, 1.0.x branches. > NULL in DisruptorQueue while multi-threaded ack > --- > > Key: STORM-2231 > URL: https://issues.apache.org/jira/browse/STORM-2231 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.1, 1.1.0 >Reporter: Alexander Kharitonov >Assignee: Jungtaek Lim >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.1.2, 1.0.5 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > I use simple topology with one spout (9 workers) and one bolt (9 workers). > I have topology.backpressure.enable: false in storm.yaml. > Spouts send about 10 000 000 tuples in 10 minutes. Pending for spout is 80 > 000. > Bolts buffer theirs tuples for 60 seconds and flush to database and ack > tuples in parallel (10 threads). > I read that OutputCollector can be used in many threads safely, so i use it. > I don't have any bottleneck in bolts(flushing to database) or spouts(kafka > spout), but about 2% of tuples fail due to tuple processing timeout (fails > are recordered in spout stats only). > I am sure that bolts ack all tuples. But some of acks don't come to spouts. > While multi-threaded acking i see many errors in worker logs like that: > 2016-12-01 13:21:10.741 o.a.s.u.DisruptorQueue [ERROR] NULL found in > disruptor-executor[3 3]-send-queue:853877 > I tried to use synchronized wrapper around OutputCollector to fix the error. > But it didn't help. > I found the workaround that helps me: i do all processing in bolt in multiple > threads but call OutputCollector.ack methods in a one single separate thread. > I think Storm has an error in the multi-threaded use of OutputCollector. > If my topology has much less load, like 500 000 tuples per 10 minutes, then > i don't lose any acks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2231) NULL in DisruptorQueue while multi-threaded ack
[ https://issues.apache.org/jira/browse/STORM-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142789#comment-16142789 ] Jungtaek Lim commented on STORM-2231: - [~kevinconaway] I just merged this into all 1.x version lines, so the patch will be shipped to 1.1.2. I would be appreciated if you can test this patch before releasing. Btw, I guess only the case that "wrapping output collector with synchronized keyword doesn't help" is N tasks : 1 executor. If it is not your case and wrapping output collector doesn't work in your case, please let me know details on your topology. > NULL in DisruptorQueue while multi-threaded ack > --- > > Key: STORM-2231 > URL: https://issues.apache.org/jira/browse/STORM-2231 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.1, 1.1.0 >Reporter: Alexander Kharitonov >Assignee: Jungtaek Lim >Priority: Critical > Time Spent: 1h 10m > Remaining Estimate: 0h > > I use simple topology with one spout (9 workers) and one bolt (9 workers). > I have topology.backpressure.enable: false in storm.yaml. > Spouts send about 10 000 000 tuples in 10 minutes. Pending for spout is 80 > 000. > Bolts buffer theirs tuples for 60 seconds and flush to database and ack > tuples in parallel (10 threads). > I read that OutputCollector can be used in many threads safely, so i use it. > I don't have any bottleneck in bolts(flushing to database) or spouts(kafka > spout), but about 2% of tuples fail due to tuple processing timeout (fails > are recordered in spout stats only). > I am sure that bolts ack all tuples. But some of acks don't come to spouts. > While multi-threaded acking i see many errors in worker logs like that: > 2016-12-01 13:21:10.741 o.a.s.u.DisruptorQueue [ERROR] NULL found in > disruptor-executor[3 3]-send-queue:853877 > I tried to use synchronized wrapper around OutputCollector to fix the error. > But it didn't help. > I found the workaround that helps me: i do all processing in bolt in multiple > threads but call OutputCollector.ack methods in a one single separate thread. > I think Storm has an error in the multi-threaded use of OutputCollector. > If my topology has much less load, like 500 000 tuples per 10 minutes, then > i don't lose any acks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2701) Supervisor crashed if blob has bad permission
[
https://issues.apache.org/jira/browse/STORM-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-2701.
-
Resolution: Fixed
Fix Version/s: 2.0.0
> Supervisor crashed if blob has bad permission
> -
>
> Key: STORM-2701
> URL: https://issues.apache.org/jira/browse/STORM-2701
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Ethan Li
>Assignee: Ethan Li
> Fix For: 2.0.0
>
>
> When
>
> {code:java}
> storm.blobstore.acl.validation.enabled: true
> {code}
> is set and we submit a topology with topology.blobstore.map. If the topology
> has no access to any one blob in the map, the supervisor will crash.
> Error:
> {code:java}
> 2017-08-21 14:21:56.597 o.a.s.d.s.Slot SLOT_6701 [ERROR] Error when
> processing event
> java.util.concurrent.ExecutionException: AuthorizationException(msg:[ethan]
> does not have access to key1)
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> ~[?:1.8.0_131]
> at java.util.concurrent.FutureTask.get(FutureTask.java:206)
> ~[?:1.8.0_131]
> at
> org.apache.storm.localizer.LocalDownloadedResource$NoCancelFuture.get(LocalDownloadedResource.java:63)
> ~[storm-server-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.daemon.supervisor.Slot.handleWaitingForBlobLocalization(Slot.java:410)
> ~[storm-server-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.daemon.supervisor.Slot.stateMachineStep(Slot.java:305)
> ~[storm-server-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:789)
> ~[storm-server-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> Caused by: org.apache.storm.generated.AuthorizationException
> at
> org.apache.storm.generated.Nimbus$getBlobMeta_result$getBlobMeta_resultStandardScheme.read(Nimbus.java:24484)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.generated.Nimbus$getBlobMeta_result$getBlobMeta_resultStandardScheme.read(Nimbus.java:24461)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.generated.Nimbus$getBlobMeta_result.read(Nimbus.java:24392)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.storm.generated.Nimbus$Client.recv_getBlobMeta(Nimbus.java:785)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.generated.Nimbus$Client.getBlobMeta(Nimbus.java:772)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.blobstore.NimbusBlobStore.getBlobMeta(NimbusBlobStore.java:306)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.utils.ServerUtils.nimbusVersionOfBlob(ServerUtils.java:375)
> ~[storm-server-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.localizer.Localizer.downloadBlob(Localizer.java:508)
> ~[storm-server-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.localizer.Localizer.access$000(Localizer.java:68)
> ~[storm-server-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.localizer.Localizer$DownloadBlob.call(Localizer.java:497)
> ~[storm-server-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.localizer.Localizer$DownloadBlob.call(Localizer.java:473)
> ~[storm-server-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_131]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_131]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
> 2017-08-21 14:21:56.597 o.a.s.u.Utils SLOT_6700 [ERROR] Halting process:
> Error when processing an event
> java.lang.RuntimeException: Halting process: Error when processing an event
> at org.apache.storm.utils.Utils.exitProcess(Utils.java:437)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:823)
> ~[storm-server-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> 2017-08-21 14:21:56.597 o.a.s.u.Utils SLOT_6701 [ERROR] Halting process:
> Error when processing an event
> java.lang.RuntimeException: Halting process: Error when processing an event
> at org.apache.storm.utils.Utils.exitProcess(Utils.java:437)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:823)
> ~[storm-server-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> 2017-08-21 14:21:56.610 o.a.s.d.s.Supervisor Thread-6 [INFO] Shutting down
>
[jira] [Resolved] (STORM-2704) Check blob permission before submitting the topology
[ https://issues.apache.org/jira/browse/STORM-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2704. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~ethanli], I merged into master. Same applies from STORM-2703: Please craft a new pull request if you would like to address this to 1.x version line as well, > Check blob permission before submitting the topology > > > Key: STORM-2704 > URL: https://issues.apache.org/jira/browse/STORM-2704 > Project: Apache Storm > Issue Type: Sub-task >Reporter: Ethan Li >Assignee: Ethan Li > Fix For: 2.0.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > We should check blob permission before submitting the topology. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2705) DRPCSpout sleeps twice when idle
[
https://issues.apache.org/jira/browse/STORM-2705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-2705.
-
Resolution: Fixed
Assignee: Daniel Klessing
Fix Version/s: 1.0.5
1.1.2
1.2.0
2.0.0
Thanks [~sakanaou], I merged into master, 1.x, 1.1.x, 1.0.x branches.
> DRPCSpout sleeps twice when idle
>
>
> Key: STORM-2705
> URL: https://issues.apache.org/jira/browse/STORM-2705
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Reporter: Daniel Klessing
>Assignee: Daniel Klessing
> Fix For: 2.0.0, 1.2.0, 1.1.2, 1.0.5
>
> Attachments: drpc_idle_overhead.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Profiling our topology I noticed that the {{DRPCSpout}} actually sleeps twice
> if it has nothing to do. This is due to a {{Utils.sleep(1)}} call at the end
> of {{nextTuple()}} while the {{SleepSpoutWaitStrategy}} performs the default
> sleep if a spout did not emit any tuple.
> This causes - in my understanding - unnecessary context switches, especially
> with a high number of parallelism.
> Attached the corresponding YourKit screenshot.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (STORM-2231) NULL in DisruptorQueue while multi-threaded ack
[ https://issues.apache.org/jira/browse/STORM-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned STORM-2231: --- Assignee: Jungtaek Lim > NULL in DisruptorQueue while multi-threaded ack > --- > > Key: STORM-2231 > URL: https://issues.apache.org/jira/browse/STORM-2231 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.1, 1.1.0 >Reporter: Alexander Kharitonov >Assignee: Jungtaek Lim >Priority: Critical > Time Spent: 10m > Remaining Estimate: 0h > > I use simple topology with one spout (9 workers) and one bolt (9 workers). > I have topology.backpressure.enable: false in storm.yaml. > Spouts send about 10 000 000 tuples in 10 minutes. Pending for spout is 80 > 000. > Bolts buffer theirs tuples for 60 seconds and flush to database and ack > tuples in parallel (10 threads). > I read that OutputCollector can be used in many threads safely, so i use it. > I don't have any bottleneck in bolts(flushing to database) or spouts(kafka > spout), but about 2% of tuples fail due to tuple processing timeout (fails > are recordered in spout stats only). > I am sure that bolts ack all tuples. But some of acks don't come to spouts. > While multi-threaded acking i see many errors in worker logs like that: > 2016-12-01 13:21:10.741 o.a.s.u.DisruptorQueue [ERROR] NULL found in > disruptor-executor[3 3]-send-queue:853877 > I tried to use synchronized wrapper around OutputCollector to fix the error. > But it didn't help. > I found the workaround that helps me: i do all processing in bolt in multiple > threads but call OutputCollector.ack methods in a one single separate thread. > I think Storm has an error in the multi-threaded use of OutputCollector. > If my topology has much less load, like 500 000 tuples per 10 minutes, then > i don't lose any acks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2693) Topology submission or kill takes too much time when topologies grow to a few hundred
[ https://issues.apache.org/jira/browse/STORM-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139904#comment-16139904 ] Jungtaek Lim commented on STORM-2693: - [~danny0405] Just curious, could you try out pacemaker to see it helps on your case? > Topology submission or kill takes too much time when topologies grow to a few > hundred > - > > Key: STORM-2693 > URL: https://issues.apache.org/jira/browse/STORM-2693 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Affects Versions: 0.9.6, 1.0.2, 1.1.0, 1.0.3 >Reporter: Yuzhao Chen > Fix For: 1.1.0 > > Attachments: 2FA30CD8-AF15-4352-992D-A67BD724E7FB.png, > D4A30D40-25D5-4ACF-9A96-252EBA9E6EF6.png > > > Now for a storm cluster with 40 hosts [with 32 cores/128G memory] and > hundreds of topologies, nimbus submission and killing will take about minutes > to finish. For example, for a cluster with 300 hundred of topologies，it will > take about 8 minutes to submit a topology, this affect our efficiency > seriously. > So, i check out the nimbus code and find two factor that will effect nimbus > submission/killing time for a scheduling round: > * read existing-assignments from zookeeper for every topology [will take > about 4 seconds for a 300 topologies cluster] > * read all the workers heartbeats and update the state to nimbus cache [will > take about 30 seconds for a 300 topologies cluster] > the key here is that Storm now use zookeeper to collect heartbeats [not RPC], > and also keep physical plan [assignments] using zookeeper which can be > totally local in nimbus. > So, i think we should make some changes to storm's heartbeats and assignments > management. > For assignment promotion: > 1. nimbus will put the assignments in memory also cache a copy to > local disk > 2. when restart or HA leader trigger nimbus will recover > assignments from disk > 3. nimbus will tell supervisor its assignment every time scheduling > 4. supervisor will sync assignments at fixed time > For heartbeats promotion: > 1. workers will report executors ok or wrong to supervisor at fixed time > 2. supervisor will report workers heartbeats to nimbus at fixed time > 3. if supervisor die, it will tell nimbus through runtime hook > or let nimbus find it through aware supervisor if survive > 4. let supervisor decide if worker is running ok or invalid , supervisor will > tell nimbus which executors of every topology are ok -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2231) NULL in DisruptorQueue while multi-threaded ack
[ https://issues.apache.org/jira/browse/STORM-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139831#comment-16139831 ] Jungtaek Lim commented on STORM-2231: - [~chemist] [~kevinconaway] I guess I found suspicious spot, but possible fixes may affect performance so would like to spend some time to do performance tests on fixes. I didn't reproduce the issue and don't know it is easy to reproduce, so if one of you can help testing and verifying the patch it should be really helpful. > NULL in DisruptorQueue while multi-threaded ack > --- > > Key: STORM-2231 > URL: https://issues.apache.org/jira/browse/STORM-2231 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.1, 1.1.0 >Reporter: Alexander Kharitonov >Priority: Critical > > I use simple topology with one spout (9 workers) and one bolt (9 workers). > I have topology.backpressure.enable: false in storm.yaml. > Spouts send about 10 000 000 tuples in 10 minutes. Pending for spout is 80 > 000. > Bolts buffer theirs tuples for 60 seconds and flush to database and ack > tuples in parallel (10 threads). > I read that OutputCollector can be used in many threads safely, so i use it. > I don't have any bottleneck in bolts(flushing to database) or spouts(kafka > spout), but about 2% of tuples fail due to tuple processing timeout (fails > are recordered in spout stats only). > I am sure that bolts ack all tuples. But some of acks don't come to spouts. > While multi-threaded acking i see many errors in worker logs like that: > 2016-12-01 13:21:10.741 o.a.s.u.DisruptorQueue [ERROR] NULL found in > disruptor-executor[3 3]-send-queue:853877 > I tried to use synchronized wrapper around OutputCollector to fix the error. > But it didn't help. > I found the workaround that helps me: i do all processing in bolt in multiple > threads but call OutputCollector.ack methods in a one single separate thread. > I think Storm has an error in the multi-threaded use of OutputCollector. > If my topology has much less load, like 500 000 tuples per 10 minutes, then > i don't lose any acks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2231) NULL in DisruptorQueue while multi-threaded ack
[ https://issues.apache.org/jira/browse/STORM-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139736#comment-16139736 ] Jungtaek Lim commented on STORM-2231: - [~chemist] [~kevinconaway] Could you give us some information around this? Also could you share why you're launching threads in bolt and ack from there instead of increasing tasks or additional bolts? For me it is possible use cases but didn't see the case actually. > NULL in DisruptorQueue while multi-threaded ack > --- > > Key: STORM-2231 > URL: https://issues.apache.org/jira/browse/STORM-2231 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.1, 1.1.0 >Reporter: Alexander Kharitonov >Priority: Critical > > I use simple topology with one spout (9 workers) and one bolt (9 workers). > I have topology.backpressure.enable: false in storm.yaml. > Spouts send about 10 000 000 tuples in 10 minutes. Pending for spout is 80 > 000. > Bolts buffer theirs tuples for 60 seconds and flush to database and ack > tuples in parallel (10 threads). > I read that OutputCollector can be used in many threads safely, so i use it. > I don't have any bottleneck in bolts(flushing to database) or spouts(kafka > spout), but about 2% of tuples fail due to tuple processing timeout (fails > are recordered in spout stats only). > I am sure that bolts ack all tuples. But some of acks don't come to spouts. > While multi-threaded acking i see many errors in worker logs like that: > 2016-12-01 13:21:10.741 o.a.s.u.DisruptorQueue [ERROR] NULL found in > disruptor-executor[3 3]-send-queue:853877 > I tried to use synchronized wrapper around OutputCollector to fix the error. > But it didn't help. > I found the workaround that helps me: i do all processing in bolt in multiple > threads but call OutputCollector.ack methods in a one single separate thread. > I think Storm has an error in the multi-threaded use of OutputCollector. > If my topology has much less load, like 500 000 tuples per 10 minutes, then > i don't lose any acks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-1977) Leader Nimbus crashes with getClusterInfo when it doesn't have one or more replicated topology codes
[
https://issues.apache.org/jira/browse/STORM-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-1977:
Fix Version/s: (was: 1.1.1)
1.1.0
> Leader Nimbus crashes with getClusterInfo when it doesn't have one or more
> replicated topology codes
>
>
> Key: STORM-1977
> URL: https://issues.apache.org/jira/browse/STORM-1977
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.0, 1.0.1
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Critical
> Fix For: 2.0.0, 1.0.2, 1.1.0
>
>
> While investigating STORM-1976, I found that there're cases for nimbus to not
> having topology codes.
> Before BlobStore, only nimbuses which is having all topology codes can gain
> leadership, otherwise they give up leadership immediately. While introducing
> BlobStore, this logic is removed.
> I don't know it's intended or not, but it incurs one of nimbus to gain
> leadership which doesn't have replicated topology code, and the nimbus will
> be crashed when getClusterInfo is requested.
> Easiest way to reproduce is:
> 1. comment cleanup-corrupt-topologies! from nimbus.clj (It's a quick
> workaround for resolving STORM-1976), and patch Storm cluster
> 2. Launch Nimbus 1 (leader)
> 3. Run topology
> 4. Kill Nimbus 1
> 5. Launch Nimbus 2 from different node
> 6. Nimbus 2 gains leadership
> 7. getClusterInfo is requested to Nimbus 2, and Nimbus 2 gets crashed
> Log:
> {code}
> 2016-07-17 08:47:48.378 o.a.s.b.FileBlobStoreImpl [INFO] Creating new blob
> store based in /grid/0/hadoop/storm/blobs
> ...
> 2016-07-17 08:47:48.619 o.a.s.zookeeper [INFO] Queued up for leader lock.
> 2016-07-17 08:47:48.651 o.a.s.zookeeper [INFO] gained leadership
> ...
> 2016-07-17 08:47:48.833 o.a.s.d.nimbus [INFO] Starting nimbus server for
> storm version '1.1.1-SNAPSHOT'
> 2016-07-17 08:47:49.295 o.a.s.t.ProcessFunction [ERROR] Internal error
> processing getClusterInfo
> KeyNotFoundException(msg:production-topology-2-1468745167-stormcode.ser)
> at
> org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:149)
> at
> org.apache.storm.blobstore.LocalFsBlobStore.getBlobReplication(LocalFsBlobStore.java:268)
> ...
> at
> org.apache.storm.daemon.nimbus$get_blob_replication_count.invoke(nimbus.clj:498)
> at
> org.apache.storm.daemon.nimbus$get_cluster_info$iter__9520__9524$fn__9525.invoke(nimbus.clj:1427)
> ...
> at
> org.apache.storm.daemon.nimbus$get_cluster_info.invoke(nimbus.clj:1401)
> at
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__9612.getClusterInfo(nimbus.clj:1838)
> at
> org.apache.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:3724)
> at
> org.apache.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:3708)
> at
> org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ...
> 2016-07-17 08:47:49.397 o.a.s.b.BlobStoreUtils [ERROR] Could not download
> blob with keyproduction-topology-2-1468745167-stormconf.ser
> 2016-07-17 08:47:49.400 o.a.s.b.BlobStoreUtils [ERROR] Could not update the
> blob with keyproduction-topology-2-1468745167-stormconf.ser
> 2016-07-17 08:47:49.402 o.a.s.d.nimbus [ERROR] Error when processing event
> KeyNotFoundException(msg:production-topology-2-1468745167-stormconf.ser)
> at
> org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:149)
> at
> org.apache.storm.blobstore.LocalFsBlobStore.getBlob(LocalFsBlobStore.java:239)
> at org.apache.storm.blobstore.BlobStore.readBlobTo(BlobStore.java:271)
> at org.apache.storm.blobstore.BlobStore.readBlob(BlobStore.java:300)
> ...
>at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
> at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
> at
> org.apache.storm.daemon.nimbus$read_storm_conf_as_nimbus.invoke(nimbus.clj:548)
> at
> org.apache.storm.daemon.nimbus$read_topology_details.invoke(nimbus.clj:555)
> at
> org.apache.storm.daemon.nimbus$mk_assignments$iter__9205__9209$fn__9210.invoke(nimbus.clj:912)
> ...
> at
> org.apache.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:911)
> at clojure.lang.RestFn.invoke(RestFn.java:410)
> at
> org.apache.storm.daemon.nimbus$fn__9769$exec_fn__1363__auto9770$fn__9781$fn__9782.invoke(nimbus.clj:2216)
> at
> org.apache.storm.daemon.nimbus$fn__9769$exec_fn__1363__auto9770$fn__9781.invoke(nimbus.clj:2215)
> at
> org.apache.storm.timer$schedule_recurring$this__1732.invoke(timer.clj:105)
> at
>
[jira] [Updated] (STORM-1977) Leader Nimbus crashes with getClusterInfo when it doesn't have one or more replicated topology codes
[
https://issues.apache.org/jira/browse/STORM-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-1977:
Fix Version/s: (was: 1.1.0)
1.1.1
> Leader Nimbus crashes with getClusterInfo when it doesn't have one or more
> replicated topology codes
>
>
> Key: STORM-1977
> URL: https://issues.apache.org/jira/browse/STORM-1977
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.0, 1.0.1
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Critical
> Fix For: 2.0.0, 1.0.2, 1.1.1
>
>
> While investigating STORM-1976, I found that there're cases for nimbus to not
> having topology codes.
> Before BlobStore, only nimbuses which is having all topology codes can gain
> leadership, otherwise they give up leadership immediately. While introducing
> BlobStore, this logic is removed.
> I don't know it's intended or not, but it incurs one of nimbus to gain
> leadership which doesn't have replicated topology code, and the nimbus will
> be crashed when getClusterInfo is requested.
> Easiest way to reproduce is:
> 1. comment cleanup-corrupt-topologies! from nimbus.clj (It's a quick
> workaround for resolving STORM-1976), and patch Storm cluster
> 2. Launch Nimbus 1 (leader)
> 3. Run topology
> 4. Kill Nimbus 1
> 5. Launch Nimbus 2 from different node
> 6. Nimbus 2 gains leadership
> 7. getClusterInfo is requested to Nimbus 2, and Nimbus 2 gets crashed
> Log:
> {code}
> 2016-07-17 08:47:48.378 o.a.s.b.FileBlobStoreImpl [INFO] Creating new blob
> store based in /grid/0/hadoop/storm/blobs
> ...
> 2016-07-17 08:47:48.619 o.a.s.zookeeper [INFO] Queued up for leader lock.
> 2016-07-17 08:47:48.651 o.a.s.zookeeper [INFO] gained leadership
> ...
> 2016-07-17 08:47:48.833 o.a.s.d.nimbus [INFO] Starting nimbus server for
> storm version '1.1.1-SNAPSHOT'
> 2016-07-17 08:47:49.295 o.a.s.t.ProcessFunction [ERROR] Internal error
> processing getClusterInfo
> KeyNotFoundException(msg:production-topology-2-1468745167-stormcode.ser)
> at
> org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:149)
> at
> org.apache.storm.blobstore.LocalFsBlobStore.getBlobReplication(LocalFsBlobStore.java:268)
> ...
> at
> org.apache.storm.daemon.nimbus$get_blob_replication_count.invoke(nimbus.clj:498)
> at
> org.apache.storm.daemon.nimbus$get_cluster_info$iter__9520__9524$fn__9525.invoke(nimbus.clj:1427)
> ...
> at
> org.apache.storm.daemon.nimbus$get_cluster_info.invoke(nimbus.clj:1401)
> at
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__9612.getClusterInfo(nimbus.clj:1838)
> at
> org.apache.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:3724)
> at
> org.apache.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:3708)
> at
> org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ...
> 2016-07-17 08:47:49.397 o.a.s.b.BlobStoreUtils [ERROR] Could not download
> blob with keyproduction-topology-2-1468745167-stormconf.ser
> 2016-07-17 08:47:49.400 o.a.s.b.BlobStoreUtils [ERROR] Could not update the
> blob with keyproduction-topology-2-1468745167-stormconf.ser
> 2016-07-17 08:47:49.402 o.a.s.d.nimbus [ERROR] Error when processing event
> KeyNotFoundException(msg:production-topology-2-1468745167-stormconf.ser)
> at
> org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:149)
> at
> org.apache.storm.blobstore.LocalFsBlobStore.getBlob(LocalFsBlobStore.java:239)
> at org.apache.storm.blobstore.BlobStore.readBlobTo(BlobStore.java:271)
> at org.apache.storm.blobstore.BlobStore.readBlob(BlobStore.java:300)
> ...
>at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
> at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
> at
> org.apache.storm.daemon.nimbus$read_storm_conf_as_nimbus.invoke(nimbus.clj:548)
> at
> org.apache.storm.daemon.nimbus$read_topology_details.invoke(nimbus.clj:555)
> at
> org.apache.storm.daemon.nimbus$mk_assignments$iter__9205__9209$fn__9210.invoke(nimbus.clj:912)
> ...
> at
> org.apache.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:911)
> at clojure.lang.RestFn.invoke(RestFn.java:410)
> at
> org.apache.storm.daemon.nimbus$fn__9769$exec_fn__1363__auto9770$fn__9781$fn__9782.invoke(nimbus.clj:2216)
> at
> org.apache.storm.daemon.nimbus$fn__9769$exec_fn__1363__auto9770$fn__9781.invoke(nimbus.clj:2215)
> at
> org.apache.storm.timer$schedule_recurring$this__1732.invoke(timer.clj:105)
> at
>
[jira] [Updated] (STORM-1167) Add sliding & tumbling window support for core storm
[ https://issues.apache.org/jira/browse/STORM-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-1167: Fix Version/s: 1.0.0 > Add sliding & tumbling window support for core storm > > > Key: STORM-1167 > URL: https://issues.apache.org/jira/browse/STORM-1167 > Project: Apache Storm > Issue Type: Improvement >Reporter: Arun Mahadevan >Assignee: Arun Mahadevan > Fix For: 1.0.0 > > > Currently, topologies that needs windowing support requires writing custom > logic inside bolts making it tedious to handle the windowing and acking logic > with custom logic. > We can add framework level support to core storm bolts to process tuples in a > time or a count based window. Sliding and tumbling windows can be supported. > Later this can be extended to trident apis as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-1176) Checkpoint window evaluated state and use this to prune duplicate evaluations
[ https://issues.apache.org/jira/browse/STORM-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-1176: Fix Version/s: 1.0.0 > Checkpoint window evaluated state and use this to prune duplicate evaluations > - > > Key: STORM-1176 > URL: https://issues.apache.org/jira/browse/STORM-1176 > Project: Apache Storm > Issue Type: Sub-task >Reporter: Arun Mahadevan >Assignee: Arun Mahadevan > Fix For: 1.0.0 > > > Evaluated state of sliding/tumbling windows should be checkpointed > periodically and on event replay during restart this info should be used to > prune duplicate evaluations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-1179) Create Maven Profiles for Integration Tests
[ https://issues.apache.org/jira/browse/STORM-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-1179: Fix Version/s: 1.0.0 > Create Maven Profiles for Integration Tests > --- > > Key: STORM-1179 > URL: https://issues.apache.org/jira/browse/STORM-1179 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0 >Reporter: Hugo Louro >Assignee: Hugo Louro > Fix For: 1.0.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-1211) Add trident state and query support for cassandra connector
[ https://issues.apache.org/jira/browse/STORM-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-1211: Fix Version/s: 1.0.0 > Add trident state and query support for cassandra connector > --- > > Key: STORM-1211 > URL: https://issues.apache.org/jira/browse/STORM-1211 > Project: Apache Storm > Issue Type: Improvement >Reporter: Satish Duggana >Assignee: Satish Duggana > Fix For: 1.0.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Reopened] (STORM-1336) Evalute/Port JStorm cgroup support
[ https://issues.apache.org/jira/browse/STORM-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reopened STORM-1336: - > Evalute/Port JStorm cgroup support > -- > > Key: STORM-1336 > URL: https://issues.apache.org/jira/browse/STORM-1336 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Boyang Jerry Peng > Labels: jstorm-merger > Fix For: 2.0.0 > > > Supports controlling the upper limit of CPU core usage for a worker using > cgroups > Sounds like a good start, will be nice to integrate it with RAS requests too. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-1402) Test missing for topology history during merge
[ https://issues.apache.org/jira/browse/STORM-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-1402: Fix Version/s: 1.0.0 > Test missing for topology history during merge > -- > > Key: STORM-1402 > URL: https://issues.apache.org/jira/browse/STORM-1402 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0 >Reporter: Zhuo Liu >Assignee: Zhuo Liu >Priority: Minor > Fix For: 1.0.0 > > > a test for topology history is missing in nimbus test when we merge that into > community. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-1336) Evalute/Port JStorm cgroup support
[ https://issues.apache.org/jira/browse/STORM-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-1336. - Resolution: Fixed Fix Version/s: 2.0.0 > Evalute/Port JStorm cgroup support > -- > > Key: STORM-1336 > URL: https://issues.apache.org/jira/browse/STORM-1336 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Boyang Jerry Peng > Labels: jstorm-merger > Fix For: 2.0.0 > > > Supports controlling the upper limit of CPU core usage for a worker using > cgroups > Sounds like a good start, will be nice to integrate it with RAS requests too. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-1637) MongoDB pom.xml fails on JDK8 Travis
[
https://issues.apache.org/jira/browse/STORM-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-1637:
Fix Version/s: 1.0.0
> MongoDB pom.xml fails on JDK8 Travis
>
>
> Key: STORM-1637
> URL: https://issues.apache.org/jira/browse/STORM-1637
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-mongodb
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>Priority: Critical
> Fix For: 1.0.0
>
>
> https://travis-ci.org/apache/storm/builds/116573238
> JDK7 builds pass JDK8 builds fail with.
> {code}
> [ERROR] The build could not read 1 project -> [Help 1]
> [ERROR]
> [ERROR] The project org.apache.storm:storm-mongodb:[unknown-version]
> (/home/travis/build/apache/storm/external/storm-mongodb/pom.xml) has 1 error
> [ERROR] Non-resolvable parent POM: Could not transfer artifact
> org.apache.storm:storm:pom:2.0.0-SNAPSHOT from/to codehaus-snapshots
> (https://nexus.codehaus.org/snapshots/): nexus.codehaus.org and
> 'parent.relativePath' points at wrong local POM @ line 21, column 13: Unknown
> host nexus.codehaus.org -> [Help 2]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
> [ERROR] [Help 2]
> http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (STORM-1645) nimbus.thrift.port command line argument leads to java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Integer
[
https://issues.apache.org/jira/browse/STORM-1645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-1645:
Fix Version/s: 0.10.1
> nimbus.thrift.port command line argument leads to
> java.lang.ClassCastException: java.lang.Long cannot be cast to
> java.lang.Integer
> --
>
> Key: STORM-1645
> URL: https://issues.apache.org/jira/browse/STORM-1645
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 0.10.0
>Reporter: Waatze Goldewijk
>Assignee: Abhishek Agarwal
> Fix For: 0.10.1
>
>
> When you supply the commandline parameter for a custom Nimbus thrift port,
> the number value is interpreted as a Long, but internally used as an Integer.
> This leads to a ClassCastException.
> This is executed (redacted):
> /opt/storm/apache-storm-0.10.0/bin/storm kill -w 10 -c
> nimbus.thrift.port=6627 -c nimbus.host=vm0009 #{topology}"
> This is the output:
> ** [out :: vm0009] 2627 [main] INFO b.s.u.Utils - Using defaults.yaml from
> resources
> ** [out :: vm0009] 2795 [main] INFO b.s.u.Utils - Using storm.yaml from
> resources
> ** [out :: vm0009] 4262 [main] INFO b.s.u.Utils - Using defaults.yaml from
> resources
> ** [out :: vm0009] 4287 [main] INFO b.s.u.Utils - Using storm.yaml from
> resources
> ** [out :: vm0009] 4328 [main] INFO b.s.thrift - Connecting to Nimbus at
> vm0009:6627 as user:
> ** [out :: vm0009] 4328 [main] INFO b.s.u.Utils - Using defaults.yaml from
> resources
> ** [out :: vm0009] 4348 [main] INFO b.s.u.Utils - Using storm.yaml from
> resources
> ** [out :: vm0009] Exception in thread "main"
> ** [out :: vm0009] java.lang.ClassCastException: java.lang.Long cannot be
> cast to java.lang.Integer
> ** [out :: vm0009]
> ** [out :: vm0009] at
> backtype.storm.thrift$nimbus_client_and_conn.invoke(thrift.clj:75)
> ** [out :: vm0009]
> ** [out :: vm0009] at
> backtype.storm.thrift$nimbus_client_and_conn.invoke(thrift.clj:72)
> ** [out :: vm0009]
> ** [out :: vm0009]
> ** [out :: vm0009] at
> backtype.storm.command.kill_topology$_main.doInvoke(kill_topology.clj:26)
> ** [out :: vm0009]
> ** [out :: vm0009] at clojure.lang.RestFn.applyTo(RestFn.java:137)
> ** [out :: vm0009]
> ** [out :: vm0009] at backtype.storm.command.kill_topology.main(Unknown
> Source)
> ** [out :: vm0009]
> I have seen other related issues:
> https://issues.apache.org/jira/browse/STORM-1578
> I believe this is the same issue (internally using an Integer, but converting
> the input to a Long) in a different area.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (STORM-1695) Create trident spout that uses the new kafka consumer API
[ https://issues.apache.org/jira/browse/STORM-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-1695: Fix Version/s: 1.1.0 2.0.0 > Create trident spout that uses the new kafka consumer API > - > > Key: STORM-1695 > URL: https://issues.apache.org/jira/browse/STORM-1695 > Project: Apache Storm > Issue Type: Improvement > Components: storm-kafka >Reporter: Thomas Graves >Assignee: Hugo Louro > Fix For: 2.0.0, 1.1.0 > > > In storm-822 we added a new kafka spout > (org.apache.storm.kafka.spout.KafkaSpout) that uses the new consumer Api in > Kafka 0.9. We decided in that one to handle the Trident support separately. > So this jira is to add Trident support for it similar to what > OpaqueTridentKafkaSpout does for the kafka old consumer api. We need to > support the new consumer api to allow access to a secure Kafka cluster which > was added in Kafka 0.9. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-1769) Add a test for submitting topologies to local nimbus with STORM_TOPOLOGY_SUBMISSION_NOTIFIER_PLUGIN.
[ https://issues.apache.org/jira/browse/STORM-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-1769. - Resolution: Fixed Fix Version/s: 2.0.0 > Add a test for submitting topologies to local nimbus with > STORM_TOPOLOGY_SUBMISSION_NOTIFIER_PLUGIN. > > > Key: STORM-1769 > URL: https://issues.apache.org/jira/browse/STORM-1769 > Project: Apache Storm > Issue Type: Task >Reporter: Satish Duggana >Assignee: Satish Duggana > Fix For: 2.0.0 > > > Add a test with > - local cluster with nimbus > - register Config.STORM_TOPOLOGY_SUBMISSION_NOTIFIER_PLUGIN > check for > - local cluster comes up > - submit a topology on that > - registered notifier plugin should receive a callback for submitted topology -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Reopened] (STORM-1769) Add a test for submitting topologies to local nimbus with STORM_TOPOLOGY_SUBMISSION_NOTIFIER_PLUGIN.
[ https://issues.apache.org/jira/browse/STORM-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reopened STORM-1769: - > Add a test for submitting topologies to local nimbus with > STORM_TOPOLOGY_SUBMISSION_NOTIFIER_PLUGIN. > > > Key: STORM-1769 > URL: https://issues.apache.org/jira/browse/STORM-1769 > Project: Apache Storm > Issue Type: Task >Reporter: Satish Duggana >Assignee: Satish Duggana > Fix For: 2.0.0 > > > Add a test with > - local cluster with nimbus > - register Config.STORM_TOPOLOGY_SUBMISSION_NOTIFIER_PLUGIN > check for > - local cluster comes up > - submit a topology on that > - registered notifier plugin should receive a callback for submitted topology -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Reopened] (STORM-2055) Exception when running topology from Maven exec with Flux
[
https://issues.apache.org/jira/browse/STORM-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim reopened STORM-2055:
-
> Exception when running topology from Maven exec with Flux
> -
>
> Key: STORM-2055
> URL: https://issues.apache.org/jira/browse/STORM-2055
> Project: Apache Storm
> Issue Type: Bug
> Components: Flux
>Affects Versions: 1.0.2
>Reporter: Julien Nioche
>Assignee: Julien Nioche
>
> When running a topology from Maven with Flux as a dependency, we get
> {code}
> 11335 [Thread-8] ERROR o.a.s.event - Error when processing event
> java.io.FileNotFoundException: Source
> 'file:/home/julien/.m2/repository/org/apache/storm/flux-core/1.0.1/flux-core-1.0.1.jar!/resources'
> does not exist
> at
> org.apache.storm.shade.org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1368)
> ~[storm-core-1.0.1.jar:1.0.1]
> at
> org.apache.storm.shade.org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1261)
> ~[storm-core-1.0.1.jar:1.0.1]
> at
> org.apache.storm.shade.org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1230)
> ~[storm-core-1.0.1.jar:1.0.1]
> at
> org.apache.storm.daemon.supervisor$fn__9359.invoke(supervisor.clj:1194)
> ~[storm-core-1.0.1.jar:1.0.1]
> at clojure.lang.MultiFn.invoke(MultiFn.java:243) ~[clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.supervisor$mk_synchronize_supervisor$this__9078$fn__9096.invoke(supervisor.clj:582)
> ~[storm-core-1.0.1.jar:1.0.1]
> at
> org.apache.storm.daemon.supervisor$mk_synchronize_supervisor$this__9078.invoke(supervisor.clj:581)
> ~[storm-core-1.0.1.jar:1.0.1]
> at org.apache.storm.event$event_manager$fn__8630.invoke(event.clj:40)
> [storm-core-1.0.1.jar:1.0.1]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
> {code}
> The same topology runs fine when executed with Eclipse or via the storm
> command.
> See [https://github.com/DigitalPebble/storm-crawler/issues/324]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (STORM-2207) Kafka Spout NullPointerException during ack
[
https://issues.apache.org/jira/browse/STORM-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-2207.
-
Resolution: Duplicate
> Kafka Spout NullPointerException during ack
> ---
>
> Key: STORM-2207
> URL: https://issues.apache.org/jira/browse/STORM-2207
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
>Affects Versions: 1.0.2
>Reporter: Nick Cuneo
>Assignee: Stig Rohde Døssing
>Priority: Critical
>
> This occurs on startup of the topology. There should be some null check
> safeguards, but i'm not sure what's causing it to occur in the first
> place...my guess is the topic partition is not found in the ack map.
> 2016-11-17 23:11:05.366 o.a.s.util [ERROR] Async loop died!
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420)
> ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628)
> ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484)
> [storm-core-1.0.2.jar:1.0.2]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
> Caused by: java.lang.NullPointerException
> at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> ~[storm-core-1.0.2.jar:1.0.2]
> ... 7 more
> 2016-11-17 23:11:05.379 o.a.s.d.executor [ERROR]
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420)
> ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628)
> ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484)
> [storm-core-1.0.2.jar:1.0.2]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
> Caused by: java.lang.NullPointerException
> at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> ~[storm-core-1.0.2.jar:1.0.2]
> ... 7 more
> 2016-11-17 23:11:05.473 o.a.s.util [ERROR] Halting process: ("Worker died")
> java.lang.RuntimeException: ("Worker died")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> [storm-core-1.0.2.jar:1.0.2]
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.worker$fn__8663$fn__8664.invoke(worker.clj:765)
> [storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.daemon.executor$mk_executor_data$fn__7875$fn__7876.invoke(executor.clj:274)
> [storm-core-1.0.2.jar:1.0.2]
> at
[jira] [Reopened] (STORM-2207) Kafka Spout NullPointerException during ack
[
https://issues.apache.org/jira/browse/STORM-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim reopened STORM-2207:
-
> Kafka Spout NullPointerException during ack
> ---
>
> Key: STORM-2207
> URL: https://issues.apache.org/jira/browse/STORM-2207
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
>Affects Versions: 1.0.2
>Reporter: Nick Cuneo
>Assignee: Stig Rohde Døssing
>Priority: Critical
>
> This occurs on startup of the topology. There should be some null check
> safeguards, but i'm not sure what's causing it to occur in the first
> place...my guess is the topic partition is not found in the ack map.
> 2016-11-17 23:11:05.366 o.a.s.util [ERROR] Async loop died!
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420)
> ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628)
> ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484)
> [storm-core-1.0.2.jar:1.0.2]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
> Caused by: java.lang.NullPointerException
> at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> ~[storm-core-1.0.2.jar:1.0.2]
> ... 7 more
> 2016-11-17 23:11:05.379 o.a.s.d.executor [ERROR]
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420)
> ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628)
> ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484)
> [storm-core-1.0.2.jar:1.0.2]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
> Caused by: java.lang.NullPointerException
> at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40)
> ~[storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> ~[storm-core-1.0.2.jar:1.0.2]
> ... 7 more
> 2016-11-17 23:11:05.473 o.a.s.util [ERROR] Halting process: ("Worker died")
> java.lang.RuntimeException: ("Worker died")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> [storm-core-1.0.2.jar:1.0.2]
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.worker$fn__8663$fn__8664.invoke(worker.clj:765)
> [storm-core-1.0.2.jar:1.0.2]
> at
> org.apache.storm.daemon.executor$mk_executor_data$fn__7875$fn__7876.invoke(executor.clj:274)
> [storm-core-1.0.2.jar:1.0.2]
> at
[jira] [Updated] (STORM-2371) Replace existing AMQP eventhub client with the lastest one from Microsoft eventhubs
[ https://issues.apache.org/jira/browse/STORM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2371: Fix Version/s: 2.0.0 > Replace existing AMQP eventhub client with the lastest one from Microsoft > eventhubs > --- > > Key: STORM-2371 > URL: https://issues.apache.org/jira/browse/STORM-2371 > Project: Apache Storm > Issue Type: Bug > Components: storm-eventhubs >Reporter: Ranjan Banerjee > Fix For: 2.0.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > The existing AMQP client of eventhub is causing random issues like NPE on the > client side after a period of inactivity. The proposal is to use the latest > eventhub client library provided by Microsoft. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-1156) Link Developer.md doc to the contributing page
[ https://issues.apache.org/jira/browse/STORM-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-1156: Issue Type: Documentation (was: Improvement) > Link Developer.md doc to the contributing page > -- > > Key: STORM-1156 > URL: https://issues.apache.org/jira/browse/STORM-1156 > Project: Apache Storm > Issue Type: Documentation > Components: documentation >Reporter: Thomas Graves >Assignee: Thomas Graves > > It would be nice to have something on the contributing page: > http://storm.apache.org/contribute/Contributing-to-Storm.html about how to > build storm, etc. There is a Developer.md file checked in so perhaps we can > just point to that. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-1262) port backtype.storm.command.dev-zookeeper to java
[ https://issues.apache.org/jira/browse/STORM-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-1262. - Resolution: Fixed Fix Version/s: 2.0.0 > port backtype.storm.command.dev-zookeeper to java > - > > Key: STORM-1262 > URL: https://issues.apache.org/jira/browse/STORM-1262 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: John Fang > Labels: java-migration, jstorm-merger > Fix For: 2.0.0 > > > Launch a development zookeeper instance -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-1246) port backtype.storm.local-state to java
[ https://issues.apache.org/jira/browse/STORM-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-1246: Fix Version/s: 2.0.0 > port backtype.storm.local-state to java > --- > > Key: STORM-1246 > URL: https://issues.apache.org/jira/browse/STORM-1246 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Abhishek Agarwal > Labels: java-migration, jstorm-merger > Fix For: 2.0.0 > > > Wrapper around LocalState, with some helper functions for converting between > storm and thrift. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Reopened] (STORM-1262) port backtype.storm.command.dev-zookeeper to java
[ https://issues.apache.org/jira/browse/STORM-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reopened STORM-1262: - > port backtype.storm.command.dev-zookeeper to java > - > > Key: STORM-1262 > URL: https://issues.apache.org/jira/browse/STORM-1262 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: John Fang > Labels: java-migration, jstorm-merger > Fix For: 2.0.0 > > > Launch a development zookeeper instance -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-1227) port backtype.storm.config to java
[ https://issues.apache.org/jira/browse/STORM-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-1227: Fix Version/s: 2.0.0 > port backtype.storm.config to java > -- > > Key: STORM-1227 > URL: https://issues.apache.org/jira/browse/STORM-1227 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Zhuo Liu > Labels: java-migration, jstorm-merger > Fix For: 2.0.0 > > > port backtype.storm.config to java. There are some parts of this that for > convince sake may need to stay in clojure, or have clojure equivalents. > The code that turns Config.* into clojure constants needs to stay, and > reading the config may need to stay in clojure until we can cleanup all of > the places it is used to not need the migration. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-1312) port backtype.storm.worker-test to java
[ https://issues.apache.org/jira/browse/STORM-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-1312: Fix Version/s: 2.0.0 > port backtype.storm.worker-test to java > --- > > Key: STORM-1312 > URL: https://issues.apache.org/jira/browse/STORM-1312 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Labels: java-migration, jstorm-merger > Fix For: 2.0.0 > > > Test Worker -- This message was sent by Atlassian JIRA (v6.4.14#64029)
