[jira] [Commented] (STORM-3468) https://github.com/apache/storm/blob/asf-site/README.md incorrect pointer
[ https://issues.apache.org/jira/browse/STORM-3468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16888346#comment-16888346 ] Jungtaek Lim commented on STORM-3468: - Existence of "asf-site" branch just gives confusion. Proper documentation (we should no longer have "asf-site" branch pointed out in other doc) would just work. Do we have specific reason to keep the branch? > https://github.com/apache/storm/blob/asf-site/README.md incorrect pointer > - > > Key: STORM-3468 > URL: https://issues.apache.org/jira/browse/STORM-3468 > Project: Apache Storm > Issue Type: Bug >Reporter: Sebb >Priority: Major > > https://github.com/apache/storm/blob/asf-site/README.md says: > bq. All of this moved back to subversion. Please see > https://svn.apache.org/viewvc/storm/site/ > However it is at: > https://github.com/apache/storm-site -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (STORM-3468) https://github.com/apache/storm/blob/asf-site/README.md incorrect pointer
[ https://issues.apache.org/jira/browse/STORM-3468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16888330#comment-16888330 ] Jungtaek Lim commented on STORM-3468: - I guess we just need to remove "asf-site" branch in apache/storm repo. > https://github.com/apache/storm/blob/asf-site/README.md incorrect pointer > - > > Key: STORM-3468 > URL: https://issues.apache.org/jira/browse/STORM-3468 > Project: Apache Storm > Issue Type: Bug >Reporter: Sebb >Priority: Major > > https://github.com/apache/storm/blob/asf-site/README.md says: > bq. All of this moved back to subversion. Please see > https://svn.apache.org/viewvc/storm/site/ > However it is at: > https://github.com/apache/storm-site -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (STORM-3392) Topology page should show components even if workers haven't started
[ https://issues.apache.org/jira/browse/STORM-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3392. - Resolution: Fixed Fix Version/s: 2.0.1 Thanks [~dandsager], I merged into master. > Topology page should show components even if workers haven't started > > > Key: STORM-3392 > URL: https://issues.apache.org/jira/browse/STORM-3392 > Project: Apache Storm > Issue Type: Improvement > Components: storm-client >Affects Versions: 2.0.1 >Reporter: David Andsager >Assignee: David Andsager >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.1 > > Time Spent: 1.5h > Remaining Estimate: 0h > > After starting a topology, the user can view the topology UI, but the > components are not shown until aggregate stats are available which causes a > delay. The components are known even before the stats are populated. > Improvement is to populate placeholder components aggregate stats until > actual aggregate stats become available, which immediately shows the > components. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2720) Add timestamp based FirstPollOffsetStrategy in KafkaTridentSpoutOpaque
[
https://issues.apache.org/jira/browse/STORM-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-2720.
-
Resolution: Fixed
Fix Version/s: 2.0.1
Thanks [~janithkv], I merged into master.
> Add timestamp based FirstPollOffsetStrategy in KafkaTridentSpoutOpaque
> --
>
> Key: STORM-2720
> URL: https://issues.apache.org/jira/browse/STORM-2720
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-kafka-client, trident
>Affects Versions: 1.1.1
>Reporter: Janith Kaiprath Valiyalappil
>Assignee: Janith Kaiprath Valiyalappil
>Priority: Minor
> Labels: pull-request-available
> Fix For: 2.0.1
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> Offsets for a given partition at a particular timestamp can now be found
> using offsetsForTimes API.
> https://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#offsetsForTimes(java.util.Map).
> One way to make use of this api would be to :
> Add a new option for FirstPollOffsetStrategy called TIMESTAMP
> Add a new startTimeStamp option to KafkaSpoutConfig, which would be used only
> when FirstPollOffsetStrategy is set to TIMESTAMP.
> Later in the KafkaTridentSpoutEmitter, when we do the first seek, we can do
> something like :
> {code}
> if(firstPollOffsetStrategy.equals(TIMESTAMP)) {
> try {
> startTimeStampOffset =
>
> kafkaConsumer.offsetsForTimes(Collections.singletonMap(tp,
> startTimeStamp)).get(tp).offset();
> } catch (IllegalArgumentException e) {
> LOG.error("Illegal timestamp {} provided for tp {}
> ",startTimeStamp,tp.toString());
> } catch (UnsupportedVersionException e) {
> LOG.error("Kafka Server do not support offsetsForTimes(),
> probably < 0.10.1",e);
> }
> if(startTimeStampOffset!=null) {
> LOG.info("Kafka consumer offset reset for TopicPartition
> {}, TimeStamp {}, Offset {}",tp,startTimeStamp,startTimeStampOffset);
> kafkaConsumer.seek(tp, startTimeStampOffset);
> } else {
> LOG.info("Kafka consumer offset reset by timestamp failed
> for TopicPartition {}, TimeStamp {}, Offset {}. Restart with a different
> Strategy ",tp,startTimeStamp,startTimeStampOffset);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (STORM-3202) Include offset information to spout metrics and remove storm-kafka-monitor
[ https://issues.apache.org/jira/browse/STORM-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807094#comment-16807094 ] Jungtaek Lim commented on STORM-3202: - [~Srdo] Please feel free to take this over. I thought I can make it soon but it wasn't and I couldn't even start. > Include offset information to spout metrics and remove storm-kafka-monitor > -- > > Key: STORM-3202 > URL: https://issues.apache.org/jira/browse/STORM-3202 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Priority: Major > > To provide offset information on Kafka spout (old and new), we have > storm-kafka-monitor module which is being run by UI (shell). This approach > requires UI doing too many things - basically UI process does most of things > via interacting with Nimbus - and also running external shell process in UI > process per opening topology page doesn't look right. > We could just let Spout include offset information into spout metric, and let > UI leverage the information. I have been thinking about this approach but > forgot about addressing it while thinking about generalizing the format. Now > I think we don't have to put too much effort to generalize format, because > Kafka spout is used mainly. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (STORM-3202) Include offset information to spout metrics and remove storm-kafka-monitor
[ https://issues.apache.org/jira/browse/STORM-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned STORM-3202: --- Assignee: (was: Jungtaek Lim) > Include offset information to spout metrics and remove storm-kafka-monitor > -- > > Key: STORM-3202 > URL: https://issues.apache.org/jira/browse/STORM-3202 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Priority: Major > > To provide offset information on Kafka spout (old and new), we have > storm-kafka-monitor module which is being run by UI (shell). This approach > requires UI doing too many things - basically UI process does most of things > via interacting with Nimbus - and also running external shell process in UI > process per opening topology page doesn't look right. > We could just let Spout include offset information into spout metric, and let > UI leverage the information. I have been thinking about this approach but > forgot about addressing it while thinking about generalizing the format. Now > I think we don't have to put too much effort to generalize format, because > Kafka spout is used mainly. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3274) Ensure Python 3 and 2.7 compatibility for all scripts (disallow 2.6)
[ https://issues.apache.org/jira/browse/STORM-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3274. - Resolution: Fixed Fix Version/s: 2.0.0 Looks like PR already merged. > Ensure Python 3 and 2.7 compatibility for all scripts (disallow 2.6) > > > Key: STORM-3274 > URL: https://issues.apache.org/jira/browse/STORM-3274 > Project: Apache Storm > Issue Type: Improvement > Components: storm-client, storm-client-misc >Affects Versions: 2.0.1 >Reporter: Govind Menon >Assignee: Govind Menon >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 5h 40m > Remaining Estimate: 0h > > * Also test with Travis -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3362) EventHubSpout uses a blocking receiver in nextTuple()
[ https://issues.apache.org/jira/browse/STORM-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3362. - Resolution: Fixed Assignee: Jungtaek Lim Fix Version/s: 2.0.0 Thanks [~ysyyork], I merged into master. If you would like to get this in for 1.x version line, please raise a PR for 1.x-branch as well. Thanks! > EventHubSpout uses a blocking receiver in nextTuple() > - > > Key: STORM-3362 > URL: https://issues.apache.org/jira/browse/STORM-3362 > Project: Apache Storm > Issue Type: Bug >Reporter: York Yang >Assignee: Jungtaek Lim >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > As title indicates, the line below: > eventDatawrap = partitionManager.receive(); > when I dig into it, it's a blocking call which affects ack and fail messages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3354) LeaderElector is not shut down properly
[
https://issues.apache.org/jira/browse/STORM-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-3354.
-
Resolution: Fixed
Fix Version/s: 2.0.0
Thanks [~Srdo], I merged into master.
> LeaderElector is not shut down properly
> ---
>
> Key: STORM-3354
> URL: https://issues.apache.org/jira/browse/STORM-3354
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Nimbus' LeaderElector is not shut down when Nimbus shuts down. This can cause
> test flakiness because the elector callback may be called after the Zookeeper
> client is closed.
> Additionally the LeaderListenerCallback may in some cases quit the leadership
> election if Nimbus isn't ready to become master, but I don't see any code to
> re-enter the election.
>
> {quote}java.lang.IllegalStateException: Client is not started
> at
> org.apache.storm.shade.org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
> ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.shade.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:139)
> ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:602)
> ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.shade.org.apache.curator.framework.imps.CreateBuilderImpl$17.call(CreateBuilderImpl.java:1191)
> ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.shade.org.apache.curator.framework.imps.CreateBuilderImpl$17.call(CreateBuilderImpl.java:1158)
> ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.shade.org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:64)
> ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:100)
> ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.shade.org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:1155)
> ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.shade.org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:605)
> ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:595)
> ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.shade.org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:360)
> ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.shade.org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:308)
> ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.zookeeper.ClientZookeeper.createNode(ClientZookeeper.java:98)
> ~[storm-client-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.nimbus.LeaderListenerCallback.setUpNimbusInfo(LeaderListenerCallback.java:154)
> ~[storm-server-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.nimbus.LeaderListenerCallback.leaderCallBack(LeaderListenerCallback.java:96)
> ~[storm-server-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.zookeeper.Zookeeper$1.isLeader(Zookeeper.java:123)
> ~[storm-server-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.shade.org.apache.curator.framework.recipes.leader.LeaderLatch$9.apply(LeaderLatch.java:665)
> ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.shade.org.apache.curator.framework.recipes.leader.LeaderLatch$9.apply(LeaderLatch.java:661)
> ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.shade.org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93)
> [shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.shade.org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435)
> [shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
> at
> org.apache.storm.shade.org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85)
>
[jira] [Created] (STORM-3363) Migrate Aether to maven-resolver as Aether was brought to ASF as a subproject of Apache Maven
Jungtaek Lim created STORM-3363: --- Summary: Migrate Aether to maven-resolver as Aether was brought to ASF as a subproject of Apache Maven Key: STORM-3363 URL: https://issues.apache.org/jira/browse/STORM-3363 Project: Apache Storm Issue Type: Improvement Components: storm-submit-tools Affects Versions: 2.0.0 Reporter: Jungtaek Lim Assignee: Jungtaek Lim Looks like Aether is donated to ASF (in couple of years) and renamed as "maven-resolver" as a subproject of Apache Maven. [https://github.com/apache/maven-resolver] https://issues.apache.org/jira/browse/MNG-6007 [http://incubator.apache.org/ip-clearance/maven-aether.html] As it is a kind of complete fork of Aether and further versions have been published here (the latest version of Aether is 1.1.0, whereas the latest version maven-resolver is 1.3.3), it should be easy to migrate to maven-resolver and reduce using EPLv1 license in Apache Storm. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3357) Bump Clojure to 1.10
[ https://issues.apache.org/jira/browse/STORM-3357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3357. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~Srdo], I merged into master. > Bump Clojure to 1.10 > > > Key: STORM-3357 > URL: https://issues.apache.org/jira/browse/STORM-3357 > Project: Apache Storm > Issue Type: Dependency upgrade >Affects Versions: 2.0.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Clojure 1.10 contains a few fixes for Java 9+, notably > https://dev.clojure.org/jira/browse/CLJ-2284, and we're currently using > 1.7.0. We should upgrade. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3356) Storm-hive should not pull in a compile-scope sfl4j binding
[ https://issues.apache.org/jira/browse/STORM-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3356. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~Srdo], I merged into master. > Storm-hive should not pull in a compile-scope sfl4j binding > --- > > Key: STORM-3356 > URL: https://issues.apache.org/jira/browse/STORM-3356 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Our Hive dependencies are leaking org.apache.logging.log4j:log4j-slf4j-impl > into our compile-scope dependencies. This causes a multiple bindings warning > when starting e.g. storm-starter, since the jar gets bundled into the > topology jar, and there is also a log4j-slf4j-impl present in the Storm > cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-3259) NUMA support for Storm
[ https://issues.apache.org/jira/browse/STORM-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-3259: Fix Version/s: (was: 2.0.1) > NUMA support for Storm > -- > > Key: STORM-3259 > URL: https://issues.apache.org/jira/browse/STORM-3259 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core, storm-server >Reporter: Govind Menon >Assignee: Govind Menon >Priority: Major > Labels: pull-request-available > Time Spent: 6.5h > Remaining Estimate: 0h > > * Supervisors with a config will present to Nimbus as multiple supervisors > (one per NUMA zone and one left over if ports/resources are left) > * Workers scheduled on a particular NUMA supervisor will be launched by the > actual supervisor pinned to that NUMA zone -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-3310) JCQueueTest is flaky
[
https://issues.apache.org/jira/browse/STORM-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-3310:
Fix Version/s: (was: 2.0.1)
2.0.0
> JCQueueTest is flaky
>
>
> Key: STORM-3310
> URL: https://issues.apache.org/jira/browse/STORM-3310
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The JCQueueTest is flaky
> {quote}
> [ERROR] JCQueueTest.testFirstMessageFirst:61 We expect to receive first
> published message first, but received null expected: but was:
> {quote}
> The issue is that the test has a race condition. There is no check that the
> consumer thread has read all (or any) of the produced messages before the
> test terminates.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (STORM-3309) TickTupleTest is still flaky
[
https://issues.apache.org/jira/browse/STORM-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-3309:
Fix Version/s: (was: 2.0.1)
2.0.0
> TickTupleTest is still flaky
>
>
> Key: STORM-3309
> URL: https://issues.apache.org/jira/browse/STORM-3309
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-server
>Affects Versions: 2.0.0
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {quote}
> testTickTupleWorksWithSystemBolt Time elapsed: 6.802 s <<< FAILURE!
> java.lang.AssertionError: Iteration 1 expected:<52000> but was:<51000>
> {quote}
> The test runs a topology in a local cluster with time simulation. One of the
> bolts has tick tuples enabled, and the test tries to check that the ticks
> arrive with 1 second intervals.
> As far as I can tell, the problem is that time simulation doesn't cover the
> bolt and spout executors. When the test increases simulated time by 1 second
> and waits for the cluster to idle, the test expects that to mean that the
> bolt will at that point have consumed the tick. In some cases this doesn't
> happen, and multiple tick tuples may end up queued before the bolt consumes
> them. Since the bolt is responsible for generating the timestamp, the test
> will fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (STORM-3319) Slot can fail assertions in some cases
[
https://issues.apache.org/jira/browse/STORM-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-3319:
Fix Version/s: (was: 2.0.1)
2.0.0
> Slot can fail assertions in some cases
> --
>
> Key: STORM-3319
> URL: https://issues.apache.org/jira/browse/STORM-3319
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {quote}
> 2019-01-19 22:47:03.045 [SLOT_1024] ERROR
> org.apache.storm.daemon.supervisor.Slot - Error when processing event
> java.lang.AssertionError: null
> at org.apache.storm.daemon.supervisor.Slot.handleEmpty(Slot.java:781)
> ~[classes/:?]
> at
> org.apache.storm.daemon.supervisor.Slot.stateMachineStep(Slot.java:217)
> ~[classes/:?]
> at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:900)
> [classes/:?]
> 2019-01-19 22:47:03.045 [SLOT_1025] ERROR
> org.apache.storm.daemon.supervisor.Slot - Error when processing event
> java.lang.AssertionError: null
> at org.apache.storm.daemon.supervisor.Slot.handleEmpty(Slot.java:781)
> ~[classes/:?]
> at
> org.apache.storm.daemon.supervisor.Slot.stateMachineStep(Slot.java:217)
> ~[classes/:?]
> at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:900)
> [classes/:?]
> {quote}
> The issue is that Slot tries to go from WAITING_FOR_LOCALIZATION to EMPTY
> when there's an exception downloading a blob. It then fails one of the
> assertions in EMPTY because it doesn't clear its
> pendingChangingBlobsAssignment field.
> There's no reason to go back to EMPTY. The Slot still wants to download some
> blobs, so it should just restart the downloads and go back to
> WAITING_FOR_LOCALIZATION.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (STORM-3282) Fix ServerUtils Estimating Worker Count for RAS Topologies
[ https://issues.apache.org/jira/browse/STORM-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-3282: Fix Version/s: (was: 2.0.1) 2.0.0 > Fix ServerUtils Estimating Worker Count for RAS Topologies > -- > > Key: STORM-3282 > URL: https://issues.apache.org/jira/browse/STORM-3282 > Project: Apache Storm > Issue Type: Bug > Components: storm-server >Reporter: Kishor Patil >Assignee: Kishor Patil >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The RAS worker count estimation does not take into consideration topology > level configuration that allows changing heap size - > _topology.worker.max.heap.size.mb_ if configured. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-3344) blacklist scheduler causing nimbus restart
[
https://issues.apache.org/jira/browse/STORM-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-3344:
Fix Version/s: (was: 2.0.1)
2.0.0
> blacklist scheduler causing nimbus restart
> --
>
> Key: STORM-3344
> URL: https://issues.apache.org/jira/browse/STORM-3344
> Project: Apache Storm
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Aaron Gresch
>Assignee: Aaron Gresch
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> {code:java}
> 2019-02-22 10:48:41.460 o.a.s.d.n.Nimbus timer [ERROR] Error while processing
> event
> java.lang.RuntimeException: java.lang.UnsupportedOperationException
> at
> org.apache.storm.daemon.nimbus.Nimbus.lambda$launchServer$27(Nimbus.java:2872)
> ~[storm-server-2.0.1.y.jar:2.0.1.y]
> at org.apache.storm.StormTimer$1.run(StormTimer.java:110)
> ~[storm-client-2.0.1.y.jar:2.0.1.y]
> at
> org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:226)
> [storm-client-2.0.1.y.jar:2.0.1.y]
> Caused by: java.lang.UnsupportedOperationException
> at
> org.apache.storm.shade.com.google.common.collect.UnmodifiableIterator.remove(UnmodifiableIterator.java:43)
> ~[shaded-deps-2.0.1.y.jar:2.0.1.y]
> at java.util.AbstractCollection.remove(AbstractCollection.java:293)
> ~[?:1.8.0_102]
> at
> org.apache.storm.scheduler.blacklist.BlacklistScheduler.removeLongTimeDisappearFromCache(BlacklistScheduler.java:216)
> ~[storm-server-2.0.1.y.jar:2.0.1.y]
> at
> org.apache.storm.scheduler.blacklist.BlacklistScheduler.schedule(BlacklistScheduler.java:110)
> ~[storm-server-2.0.1.y.jar:2.0.1.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.computeNewSchedulerAssignments(Nimbus.java:2070)
> ~[storm-server-2.0.1.y.jar:2.0.1.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.lockingMkAssignments(Nimbus.java:2234)
> ~[storm-server-2.0.1.y.jar:2.0.1.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.mkAssignments(Nimbus.java:2220)
> ~[storm-server-2.0.1.y.jar:2.0.1.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.mkAssignments(Nimbus.java:2165)
> ~[storm-server-2.0.1.y.jar:2.0.1.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.lambda$launchServer$27(Nimbus.java:2868)
> ~[storm-server-2.0.1.y.jar:2.0.1.y]
> ... 2 more
> 2019-02-22 10:48:41.461 o.a.s.u.Utils timer [ERROR] Halting process: Error
> while processing event
> java.lang.RuntimeException: Halting process: Error while processing event
> at org.apache.storm.utils.Utils.exitProcess(Utils.java:520)
> ~[storm-client-2.0.1.y.jar:2.0.1.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.lambda$new$9(Nimbus.java:564)
> ~[storm-server-2.0.1.y.jar:2.0.1.y]
> at
> org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:253)
> [storm-client-2.0.1.y.jar:2.0.1.y]
> 2019-02-22 10:48:41.462 o.a.s.u.Utils Thread-19 [INFO] Halting after 10
> seconds
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (STORM-3318) Complete information in Class NewKafkaSpoutOffsetQuery
[ https://issues.apache.org/jira/browse/STORM-3318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-3318: Fix Version/s: (was: 2.0.1) 2.0.0 > Complete information in Class NewKafkaSpoutOffsetQuery > -- > > Key: STORM-3318 > URL: https://issues.apache.org/jira/browse/STORM-3318 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: MichealShin >Assignee: MichealShin >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Just complete information in three methods(toString , equals, hashCode) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-3317) upload credentials fails when using different java.security.auth.login.config file
[ https://issues.apache.org/jira/browse/STORM-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-3317: Fix Version/s: (was: 2.0.1) 2.0.0 > upload credentials fails when using different java.security.auth.login.config > file > -- > > Key: STORM-3317 > URL: https://issues.apache.org/jira/browse/STORM-3317 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron Gresch >Assignee: Aaron Gresch >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Our launcher box has a differing version of java.security.auth.login.config > from the system property. Having this property set differently causes > upload-credentials to fail with the current code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3282) Fix ServerUtils Estimating Worker Count for RAS Topologies
[ https://issues.apache.org/jira/browse/STORM-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3282. - Resolution: Fixed > Fix ServerUtils Estimating Worker Count for RAS Topologies > -- > > Key: STORM-3282 > URL: https://issues.apache.org/jira/browse/STORM-3282 > Project: Apache Storm > Issue Type: Bug > Components: storm-server >Reporter: Kishor Patil >Assignee: Kishor Patil >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The RAS worker count estimation does not take into consideration topology > level configuration that allows changing heap size - > _topology.worker.max.heap.size.mb_ if configured. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3352) Use Netty BOM to lock all Netty artifacts to same version
[
https://issues.apache.org/jira/browse/STORM-3352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-3352.
-
Resolution: Fixed
Fix Version/s: 2.0.0
Thanks [~Srdo] I merged into master.
> Use Netty BOM to lock all Netty artifacts to same version
> -
>
> Key: STORM-3352
> URL: https://issues.apache.org/jira/browse/STORM-3352
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> We are not properly ensuring that Netty artifacts are the same version. I got
> a test failure in storm-cassandra, because netty-all is version 4.1.30, but
> cassandra pulls in Netty in version 4.0.37.
> {quote}
> java.lang.NoSuchMethodError:
> io.netty.util.internal.PlatformDependent.normalizedArch()Ljava/lang/String;
> at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:180)
> ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
> at io.netty.channel.epoll.Native.(Native.java:61)
> ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
> at io.netty.channel.epoll.Epoll.(Epoll.java:38)
> ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
> at org.apache.cassandra.transport.Server.run(Server.java:147)
> ~[cassandra-all-2.1.7.jar:2.1.7]
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-3344) blacklist scheduler causing nimbus restart
[
https://issues.apache.org/jira/browse/STORM-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-3344.
-
Resolution: Fixed
Fix Version/s: 2.0.1
Thanks [~agresch], I merged into master.
> blacklist scheduler causing nimbus restart
> --
>
> Key: STORM-3344
> URL: https://issues.apache.org/jira/browse/STORM-3344
> Project: Apache Storm
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Aaron Gresch
>Assignee: Aaron Gresch
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.1
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> {code:java}
> 2019-02-22 10:48:41.460 o.a.s.d.n.Nimbus timer [ERROR] Error while processing
> event
> java.lang.RuntimeException: java.lang.UnsupportedOperationException
> at
> org.apache.storm.daemon.nimbus.Nimbus.lambda$launchServer$27(Nimbus.java:2872)
> ~[storm-server-2.0.1.y.jar:2.0.1.y]
> at org.apache.storm.StormTimer$1.run(StormTimer.java:110)
> ~[storm-client-2.0.1.y.jar:2.0.1.y]
> at
> org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:226)
> [storm-client-2.0.1.y.jar:2.0.1.y]
> Caused by: java.lang.UnsupportedOperationException
> at
> org.apache.storm.shade.com.google.common.collect.UnmodifiableIterator.remove(UnmodifiableIterator.java:43)
> ~[shaded-deps-2.0.1.y.jar:2.0.1.y]
> at java.util.AbstractCollection.remove(AbstractCollection.java:293)
> ~[?:1.8.0_102]
> at
> org.apache.storm.scheduler.blacklist.BlacklistScheduler.removeLongTimeDisappearFromCache(BlacklistScheduler.java:216)
> ~[storm-server-2.0.1.y.jar:2.0.1.y]
> at
> org.apache.storm.scheduler.blacklist.BlacklistScheduler.schedule(BlacklistScheduler.java:110)
> ~[storm-server-2.0.1.y.jar:2.0.1.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.computeNewSchedulerAssignments(Nimbus.java:2070)
> ~[storm-server-2.0.1.y.jar:2.0.1.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.lockingMkAssignments(Nimbus.java:2234)
> ~[storm-server-2.0.1.y.jar:2.0.1.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.mkAssignments(Nimbus.java:2220)
> ~[storm-server-2.0.1.y.jar:2.0.1.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.mkAssignments(Nimbus.java:2165)
> ~[storm-server-2.0.1.y.jar:2.0.1.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.lambda$launchServer$27(Nimbus.java:2868)
> ~[storm-server-2.0.1.y.jar:2.0.1.y]
> ... 2 more
> 2019-02-22 10:48:41.461 o.a.s.u.Utils timer [ERROR] Halting process: Error
> while processing event
> java.lang.RuntimeException: Halting process: Error while processing event
> at org.apache.storm.utils.Utils.exitProcess(Utils.java:520)
> ~[storm-client-2.0.1.y.jar:2.0.1.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.lambda$new$9(Nimbus.java:564)
> ~[storm-server-2.0.1.y.jar:2.0.1.y]
> at
> org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:253)
> [storm-client-2.0.1.y.jar:2.0.1.y]
> 2019-02-22 10:48:41.462 o.a.s.u.Utils Thread-19 [INFO] Halting after 10
> seconds
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-3319) Slot can fail assertions in some cases
[
https://issues.apache.org/jira/browse/STORM-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-3319.
-
Resolution: Fixed
Fix Version/s: 2.0.1
Thanks [~Srdo] to the great effort on fixing instability of tests, and sorry to
let you wait too long. I merged into master.
> Slot can fail assertions in some cases
> --
>
> Key: STORM-3319
> URL: https://issues.apache.org/jira/browse/STORM-3319
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {quote}
> 2019-01-19 22:47:03.045 [SLOT_1024] ERROR
> org.apache.storm.daemon.supervisor.Slot - Error when processing event
> java.lang.AssertionError: null
> at org.apache.storm.daemon.supervisor.Slot.handleEmpty(Slot.java:781)
> ~[classes/:?]
> at
> org.apache.storm.daemon.supervisor.Slot.stateMachineStep(Slot.java:217)
> ~[classes/:?]
> at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:900)
> [classes/:?]
> 2019-01-19 22:47:03.045 [SLOT_1025] ERROR
> org.apache.storm.daemon.supervisor.Slot - Error when processing event
> java.lang.AssertionError: null
> at org.apache.storm.daemon.supervisor.Slot.handleEmpty(Slot.java:781)
> ~[classes/:?]
> at
> org.apache.storm.daemon.supervisor.Slot.stateMachineStep(Slot.java:217)
> ~[classes/:?]
> at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:900)
> [classes/:?]
> {quote}
> The issue is that Slot tries to go from WAITING_FOR_LOCALIZATION to EMPTY
> when there's an exception downloading a blob. It then fails one of the
> assertions in EMPTY because it doesn't clear its
> pendingChangingBlobsAssignment field.
> There's no reason to go back to EMPTY. The Slot still wants to download some
> blobs, so it should just restart the downloads and go back to
> WAITING_FOR_LOCALIZATION.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-3343) JCQueueTest can still be flaky
[
https://issues.apache.org/jira/browse/STORM-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-3343.
-
Resolution: Fixed
Fix Version/s: 2.0.0
Thanks [~Srdo], I merged into master.
> JCQueueTest can still be flaky
> --
>
> Key: STORM-3343
> URL: https://issues.apache.org/jira/browse/STORM-3343
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Made a mistake in the fix for STORM-3310. The consumer in one of the tests
> check for interrupt in a place it shouldn't.
> {code}
> [ERROR] Failures:
> [ERROR] JCQueueTest.lambda$testFirstMessageFirst$0:63 We expect to receive
> first published message first, but received null expected: but
> was:
> Exception in thread "Thread-125" java.lang.RuntimeException:
> java.lang.InterruptedException: ConsumerThd interrupted
> at org.apache.storm.utils.JCQueueTest$1.accept(JCQueueTest.java:48)
> at org.apache.storm.utils.JCQueue.consumeImpl(JCQueue.java:133)
> at org.apache.storm.utils.JCQueue.consume(JCQueue.java:110)
> at org.apache.storm.utils.JCQueue.consume(JCQueue.java:101)
> at
> org.apache.storm.utils.JCQueueTest$ConsumerThd.run(JCQueueTest.java:207)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.InterruptedException: ConsumerThd interrupted
> ... 6 more
> {code}
> The consumer accept method shouldn't check for interrupt, as that is handled
> by the ConsumerThd.run method. When the accept check for interrupt is hit,
> the consumer exits without draining the JCQueue, and the test may fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-3320) Executors should start when all worker connections are ready
[ https://issues.apache.org/jira/browse/STORM-3320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3320. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~Srdo], I merged into master. > Executors should start when all worker connections are ready > > > Key: STORM-3320 > URL: https://issues.apache.org/jira/browse/STORM-3320 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > We conflate "being activated" with "all workers are ready" in WorkerState, by > making isWorkerActivated a part of isTopologyActivated. > The issue with this is that isTopologyActivated is used to communicate > activation/deactivation to the executors, and is updated on a timer (default > only every 10 seconds). isWorkerActivated is really meant to be a one-way > switch, which lets us delay executor initialization until all other workers > in the topology are also started. > Since we mix the two up, if a worker is started in the topology and all other > connections aren't ready immediately (e.g. as happens every time you deploy a > topology, some workers will boot faster than others), the worker may have to > wait up to 10 seconds to start. > We should make sure the wait for isWorkerActivated happens via CountDownLatch > instead, so the executor will start as soon as the connections are ready. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3342) Add plugin to generate list of dependency licenses to build
[ https://issues.apache.org/jira/browse/STORM-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3342. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~Srdo], I merged into master. > Add plugin to generate list of dependency licenses to build > --- > > Key: STORM-3342 > URL: https://issues.apache.org/jira/browse/STORM-3342 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > I think it would be helpful if we could easily generate a list of the > licenses used by our dependencies. When we do a release, we need to make sure > we don't let dependencies with e.g. GPL license slip through, and I think it > will be easier if we can list dependencies with their licenses. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3292) Trident HiveState must flush writers when the batch commits
[ https://issues.apache.org/jira/browse/STORM-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3292. - Resolution: Fixed Assignee: Arun Mahadevan Fix Version/s: 1.2.3 2.0.0 Thanks [~arunmahadevan], I merged into master and 1.x-branch. > Trident HiveState must flush writers when the batch commits > --- > > Key: STORM-3292 > URL: https://issues.apache.org/jira/browse/STORM-3292 > Project: Apache Storm > Issue Type: Improvement >Reporter: Arun Mahadevan >Assignee: Arun Mahadevan >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0, 1.2.3 > > Time Spent: 0.5h > Remaining Estimate: 0h > > For trident the hive writer is flushed only after it hits the batch size. > see - > https://github.com/apache/storm/blob/master/external/storm-hive/src/main/java/org/apache/storm/hive/trident/HiveState.java#L108 > Trident HiveState does not flush during the batch commit and it appears to be > an oversight. Without this trident state cannot guarantee at-least once. > (E.g. if the transaction is open but trident moves to the next txid and later > fails the data in the open transaction is lost). > So I think for at-least once, the HiveState must flush all the writers > irrespective of the batch sizes when trident invokes the "commit(txid)" . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-3123) Storm Kafka Monitor does not work with Kafka over two-way SSL
[
https://issues.apache.org/jira/browse/STORM-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691237#comment-16691237
]
Jungtaek Lim commented on STORM-3123:
-
Just filed an issue in INFRA: https://issues.apache.org/jira/browse/INFRA-17266
> Storm Kafka Monitor does not work with Kafka over two-way SSL
> -
>
> Key: STORM-3123
> URL: https://issues.apache.org/jira/browse/STORM-3123
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-monitor
>Affects Versions: 1.2.2
>Reporter: Vipin Rathor
>Assignee: Vipin Rathor
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0, 1.2.3
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Storm Kafka Monitor has no option to read / parse SSL truststore/keystore
> properties which are required to connect to Kafka running over two-way SSL.
> As a fix, it needs to understand the following additional Kafka properties:
> {code:java}
> ssl.truststore.location=
> ssl.truststore.password=
> ssl.keystore.location=
> ssl.keystore.password=
> ssl.key.password=
> {code}
> Since, JVM has a fallback mechanism for loading SSL truststore, Storm Kafka
> Monitor would always endup using some truststore and would eventually work
> with one-way SSL (which is also a default for Kafka setup).
> Since there is no such fallback for SSL keystore, Storm Kafka Monitor would
> start without a keystore and would eventually throw this error (in SSL debug
> mode):
> {code:java}
> Warning: no suitable certificate found - continuing without client
> authentication
> *** Certificate chain
>
> ***
> {code}
> At this time, Kafka broker would complain about above like this:
> {code:java}
> kafka-network-thread-1002-SSL-7, READ: TLSv1.2 Handshake, length = 141
> *** Certificate chain
>
> ***
> kafka-network-thread-1002-SSL-7, fatal error: 42: null cert chain
> javax.net.ssl.SSLHandshakeException: null cert chain
> {code}
> Therefore, in the absence of this fix, the only available workaround is to
> stick to one-way SSL in Kafka (i.e. keep ssl.client.auth=none in Kafka).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (STORM-3123) Storm Kafka Monitor does not work with Kafka over two-way SSL
[
https://issues.apache.org/jira/browse/STORM-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691203#comment-16691203
]
Jungtaek Lim commented on STORM-3123:
-
[~arunmahadevan] Thanks for the quick follow-up! Also merged into 1.x-branch.
Now looks like Github mirror is not in sync, but I pushed into ASF git repo
anyway so mirror will eventually sync up.
> Storm Kafka Monitor does not work with Kafka over two-way SSL
> -
>
> Key: STORM-3123
> URL: https://issues.apache.org/jira/browse/STORM-3123
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-monitor
>Affects Versions: 1.2.2
>Reporter: Vipin Rathor
>Assignee: Vipin Rathor
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0, 1.2.3
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Storm Kafka Monitor has no option to read / parse SSL truststore/keystore
> properties which are required to connect to Kafka running over two-way SSL.
> As a fix, it needs to understand the following additional Kafka properties:
> {code:java}
> ssl.truststore.location=
> ssl.truststore.password=
> ssl.keystore.location=
> ssl.keystore.password=
> ssl.key.password=
> {code}
> Since, JVM has a fallback mechanism for loading SSL truststore, Storm Kafka
> Monitor would always endup using some truststore and would eventually work
> with one-way SSL (which is also a default for Kafka setup).
> Since there is no such fallback for SSL keystore, Storm Kafka Monitor would
> start without a keystore and would eventually throw this error (in SSL debug
> mode):
> {code:java}
> Warning: no suitable certificate found - continuing without client
> authentication
> *** Certificate chain
>
> ***
> {code}
> At this time, Kafka broker would complain about above like this:
> {code:java}
> kafka-network-thread-1002-SSL-7, READ: TLSv1.2 Handshake, length = 141
> *** Certificate chain
>
> ***
> kafka-network-thread-1002-SSL-7, fatal error: 42: null cert chain
> javax.net.ssl.SSLHandshakeException: null cert chain
> {code}
> Therefore, in the absence of this fix, the only available workaround is to
> stick to one-way SSL in Kafka (i.e. keep ssl.client.auth=none in Kafka).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (STORM-3123) Storm Kafka Monitor does not work with Kafka over two-way SSL
[
https://issues.apache.org/jira/browse/STORM-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-3123:
Fix Version/s: 1.2.3
> Storm Kafka Monitor does not work with Kafka over two-way SSL
> -
>
> Key: STORM-3123
> URL: https://issues.apache.org/jira/browse/STORM-3123
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-monitor
>Affects Versions: 1.2.2
>Reporter: Vipin Rathor
>Assignee: Vipin Rathor
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0, 1.2.3
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Storm Kafka Monitor has no option to read / parse SSL truststore/keystore
> properties which are required to connect to Kafka running over two-way SSL.
> As a fix, it needs to understand the following additional Kafka properties:
> {code:java}
> ssl.truststore.location=
> ssl.truststore.password=
> ssl.keystore.location=
> ssl.keystore.password=
> ssl.key.password=
> {code}
> Since, JVM has a fallback mechanism for loading SSL truststore, Storm Kafka
> Monitor would always endup using some truststore and would eventually work
> with one-way SSL (which is also a default for Kafka setup).
> Since there is no such fallback for SSL keystore, Storm Kafka Monitor would
> start without a keystore and would eventually throw this error (in SSL debug
> mode):
> {code:java}
> Warning: no suitable certificate found - continuing without client
> authentication
> *** Certificate chain
>
> ***
> {code}
> At this time, Kafka broker would complain about above like this:
> {code:java}
> kafka-network-thread-1002-SSL-7, READ: TLSv1.2 Handshake, length = 141
> *** Certificate chain
>
> ***
> kafka-network-thread-1002-SSL-7, fatal error: 42: null cert chain
> javax.net.ssl.SSLHandshakeException: null cert chain
> {code}
> Therefore, in the absence of this fix, the only available workaround is to
> stick to one-way SSL in Kafka (i.e. keep ssl.client.auth=none in Kafka).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-3123) Storm Kafka Monitor does not work with Kafka over two-way SSL
[
https://issues.apache.org/jira/browse/STORM-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-3123.
-
Resolution: Fixed
Assignee: Vipin Rathor
Fix Version/s: 2.0.0
Thanks [~vrathor-hw] and [~arunmahadevan] for the patch, I merged into master.
[~arunmahadevan] I marked [~vrathor-hw] as assignee since he originally worked
on. Please reassign to you it doesn't make sense to you.
And please raise another PR for 1.x-branch as well? There's merge conflict when
cherry-picking.
Thanks again!
> Storm Kafka Monitor does not work with Kafka over two-way SSL
> -
>
> Key: STORM-3123
> URL: https://issues.apache.org/jira/browse/STORM-3123
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-monitor
>Affects Versions: 1.2.2
>Reporter: Vipin Rathor
>Assignee: Vipin Rathor
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Storm Kafka Monitor has no option to read / parse SSL truststore/keystore
> properties which are required to connect to Kafka running over two-way SSL.
> As a fix, it needs to understand the following additional Kafka properties:
> {code:java}
> ssl.truststore.location=
> ssl.truststore.password=
> ssl.keystore.location=
> ssl.keystore.password=
> ssl.key.password=
> {code}
> Since, JVM has a fallback mechanism for loading SSL truststore, Storm Kafka
> Monitor would always endup using some truststore and would eventually work
> with one-way SSL (which is also a default for Kafka setup).
> Since there is no such fallback for SSL keystore, Storm Kafka Monitor would
> start without a keystore and would eventually throw this error (in SSL debug
> mode):
> {code:java}
> Warning: no suitable certificate found - continuing without client
> authentication
> *** Certificate chain
>
> ***
> {code}
> At this time, Kafka broker would complain about above like this:
> {code:java}
> kafka-network-thread-1002-SSL-7, READ: TLSv1.2 Handshake, length = 141
> *** Certificate chain
>
> ***
> kafka-network-thread-1002-SSL-7, fatal error: 42: null cert chain
> javax.net.ssl.SSLHandshakeException: null cert chain
> {code}
> Therefore, in the absence of this fix, the only available workaround is to
> stick to one-way SSL in Kafka (i.e. keep ssl.client.auth=none in Kafka).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (STORM-3202) Include offset information to spout metrics and remove storm-kafka-monitor
Jungtaek Lim created STORM-3202: --- Summary: Include offset information to spout metrics and remove storm-kafka-monitor Key: STORM-3202 URL: https://issues.apache.org/jira/browse/STORM-3202 Project: Apache Storm Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Jungtaek Lim Assignee: Jungtaek Lim To provide offset information on Kafka spout (old and new), we have storm-kafka-monitor module which is being run by UI (shell). This approach requires UI doing too many things - basically UI process does most of things via interacting with Nimbus - and also running external shell process in UI process per opening topology page doesn't look right. We could just let Spout include offset information into spout metric, and let UI leverage the information. I have been thinking about this approach but forgot about addressing it while thinking about generalizing the format. Now I think we don't have to put too much effort to generalize format, because Kafka spout is used mainly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3200) Improve debugging keytab errors
[ https://issues.apache.org/jira/browse/STORM-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3200. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~agresch], I merged into master. > Improve debugging keytab errors > > > Key: STORM-3200 > URL: https://issues.apache.org/jira/browse/STORM-3200 > Project: Apache Storm > Issue Type: Bug >Reporter: Aaron Gresch >Assignee: Aaron Gresch >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > [https://github.com/apache/storm/blob/master/external/storm-hdfs-blobstore/src/main/java/org/apache/storm/hdfs/blobstore/HdfsBlobStore.java#L165] > > This line leaves all the useful information about the failure out of the log. > Should be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2578) Apply new code style to storm-elasticsearch
[ https://issues.apache.org/jira/browse/STORM-2578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2578. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~jingc], I merged into master. Keep up the good work. > Apply new code style to storm-elasticsearch > --- > > Key: STORM-2578 > URL: https://issues.apache.org/jira/browse/STORM-2578 > Project: Apache Storm > Issue Type: Task > Components: storm-elasticsearch >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jing Chen >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3196) 'storm list' should include topology owner in output
[
https://issues.apache.org/jira/browse/STORM-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-3196.
-
Resolution: Fixed
Fix Version/s: 2.0.0
Thanks [~jtolar], I merged into master.
We are moving toward to Storm 2.0.0 hence we are unlikely to add new
features/improvements to the 1.x version lines. But please submit a pull
request for 1.x-branch when you rely on 1.x and would like to push the feature
to 1.x version too.
> 'storm list' should include topology owner in output
>
>
> Key: STORM-3196
> URL: https://issues.apache.org/jira/browse/STORM-3196
> Project: Apache Storm
> Issue Type: Improvement
>Reporter: Jacob Tolar
>Assignee: Jacob Tolar
>Priority: Minor
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> It would be helpful if 'storm list' included the owner in the output.
> For example:
> {code:java}
> Topology_name Status Num_tasks Num_workers Uptime_secs Owner
> ---
> my_topology ACTIVE 205 15 123456 jtolar
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (STORM-3195) IllegalArgumentException when rolling upgrade nimbus from 1.x to 2.x
[
https://issues.apache.org/jira/browse/STORM-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581928#comment-16581928
]
Jungtaek Lim commented on STORM-3195:
-
[~ethanli]
I'm wondering the symptom itself: shouldn't worker count as well as acker
executor number be integer? No one would say worker count is 1.66 and acker
executor count is 2.33, so the value itself looks somewhat incorrect to me.
> IllegalArgumentException when rolling upgrade nimbus from 1.x to 2.x
>
>
> Key: STORM-3195
> URL: https://issues.apache.org/jira/browse/STORM-3195
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Ethan Li
>Priority: Major
>
> {code:java}
> 2018-08-15 15:49:06.550 o.a.s.d.n.Nimbus timer [ERROR] Error while processing
> event
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Don't know
> how to convert 1.76041667 to int
> at
> org.apache.storm.daemon.nimbus.Nimbus.lambda$launchServer$48(Nimbus.java:2837)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at org.apache.storm.StormTimer$1.run(StormTimer.java:110)
> ~[storm-client-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:226)
> [storm-client-2.0.0.y.jar:2.0.0.y]
> Caused by: java.lang.IllegalArgumentException: Don't know how to convert
> 1.76041667 to int
> at org.apache.storm.utils.ObjectReader.getInt(ObjectReader.java:78)
> ~[storm-client-2.0.0.y.jar:2.0.0.y]
> at org.apache.storm.daemon.StormCommon.addAcker(StormCommon.java:256)
> ~[storm-client-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.daemon.StormCommon.systemTopologyImpl(StormCommon.java:523)
> ~[storm-client-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.daemon.StormCommon.systemTopology(StormCommon.java:440)
> ~[storm-client-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.daemon.StormCommon.stormTaskInfoImpl(StormCommon.java:544)
> ~[storm-client-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.daemon.StormCommon.stormTaskInfo(StormCommon.java:460)
> ~[storm-client-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.computeExecutors(Nimbus.java:1732)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.getOrUpdateExecutors(Nimbus.java:1357)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.computeExecutorToComponent(Nimbus.java:1752)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.readTopologyDetails(Nimbus.java:1582)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.getResourcesForTopology(Nimbus.java:2034)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.getClusterInfoImpl(Nimbus.java:2676)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.sendClusterMetricsToExecutors(Nimbus.java:2701)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.lambda$launchServer$48(Nimbus.java:2834)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> ... 2 more
> {code}
> We were rolling upgrading our storm cluster to 2.x and we found this error
> from nimbus.log. It's not necessary to do rolling upgrade but good news is we
> basically made it work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-3184) Storm supervisor log showing keystore and truststore password in plaintext
[
https://issues.apache.org/jira/browse/STORM-3184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-3184.
-
Resolution: Fixed
Assignee: Arun Mahadevan
Fix Version/s: 1.2.3
2.0.0
Thanks [~arunmahadevan], I merged into master and 1.x-branch.
> Storm supervisor log showing keystore and truststore password in plaintext
> --
>
> Key: STORM-3184
> URL: https://issues.apache.org/jira/browse/STORM-3184
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.1.1
>Reporter: Arpit Khare
>Assignee: Arun Mahadevan
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0, 1.2.3
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> When we enable SSL for Apache storm, the superviosr log shows the keystore
> and truststore password in the plaintext
> log name : /var/log/storm/supervisor.log
> {code}
> 2018-05-28 16:21:12.594 o.a.s.d.s.Supervisor main [INFO] Starting supervisor
> for storm version '1.1.1.3.1.1.0-35'.
> 2018-05-28 16:21:12.595 o.a.s.d.s.Supervisor main [INFO] Starting Supervisor
> with conf {storm.messaging.netty.min_wait_ms=100,
> storm.zookeeper.auth.user=null, storm.messaging.netty.buffer_s
> ize=5242880,
> client.jartransformer.class=org.apache.storm.hack.StormShadeTransformer,
> storm.exhibitor.port=8080, pacemaker.auth.method=NONE, ui.filter=null,
> worker.profiler.enabled=false
> ui.https.key.password=pass123
> ui.https.keystore.password=pass123
> {code}
> For the below properties created in custom-storm-site section in Ambari while
> enabling SSL.
> {code}
> ui.https.key.password=pass123
> ui.https.keystore.password=pass123
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (STORM-3145) UNCOMMITTED_EARLIEST and UNCOMMITTED_LATEST doesn't work as expected
[
https://issues.apache.org/jira/browse/STORM-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim closed STORM-3145.
---
Resolution: Won't Fix
Please send any question to the user mailing list, u...@storm.apache.org
http://storm.apache.org/getting-help.html for more details to access mailing
list.
> UNCOMMITTED_EARLIEST and UNCOMMITTED_LATEST doesn't work as expected
>
>
> Key: STORM-3145
> URL: https://issues.apache.org/jira/browse/STORM-3145
> Project: Apache Storm
> Issue Type: Question
> Components: storm-kafka-client, trident
>Reporter: Prakash N M
>Priority: Blocker
> Labels: storm-kafka-client, trident
>
> I am using storm-kafka-client 1.2.2.
> When I ran my topology I got NPE as mentioned in
> https://issues.apache.org/jira/browse/STORM-3046.
> So I modified the KafkaTridentSpoutEmitter.java#seek with patch mentioned in
> the Jira STORM-3046. Below is the modified code.
> {noformat}
> private long seek(TopicPartition tp, KafkaTridentSpoutBatchMetadata
> lastBatchMeta, long transactionId) {
> if (isFirstPoll(tp, transactionId)) {
> if (firstPollOffsetStrategy == EARLIEST) {
> LOG.debug("First poll for topic partition [{}], seeking to partition
> beginning", tp);
> kafkaConsumer.seekToBeginning(Collections.singleton(tp));
> } else if (firstPollOffsetStrategy == LATEST) {
> LOG.debug("First poll for topic partition [{}], seeking to partition end",
> tp);
> kafkaConsumer.seekToEnd(Collections.singleton(tp));
> } else if (lastBatchMeta != null) {
> LOG.debug("First poll for topic partition [{}], using last batch metadata",
> tp);
> kafkaConsumer.seek(tp, lastBatchMeta.getLastOffset() + 1); // seek next
> offset after last offset from previous batch
> } else if (firstPollOffsetStrategy == UNCOMMITTED_EARLIEST) {
> LOG.debug("First poll for topic partition [{}] with no last batch metadata,
> seeking to partition beginning", tp);
> kafkaConsumer.seekToBeginning(Collections.singleton(tp));
> } else if (firstPollOffsetStrategy == UNCOMMITTED_LATEST) {
> LOG.debug("First poll for topic partition [{}] with no last batch metadata,
> seeking to partition end", tp);
> kafkaConsumer.seekToEnd(Collections.singleton(tp));
> }
> firstPollTransaction.put(tp, transactionId);
> } else if (lastBatchMeta != null) {
> kafkaConsumer.seek(tp, lastBatchMeta.getLastOffset() + 1); // seek
> next offset after last offset from previous batch
> LOG.debug("First poll for topic partition [{}], using last batch
> metadata", tp);
> } else {
> long initialFetchOffset = firstPollTransaction.get(tp);
> OffsetAndMetadata lastCommittedOffset = kafkaConsumer.committed(tp);
> kafkaConsumer.seek(tp, lastCommittedOffset.offset());
> LOG.debug("First poll for topic partition [{}], no last batch
> metadata present."
> + " Using stored initial fetch offset [{}]", tp,
> initialFetchOffset);
> }
> final long fetchOffset = kafkaConsumer.position(tp);
> LOG.debug("Set [fetchOffset = {}] for partition [{}]", fetchOffset, tp);
> return fetchOffset;
> }{noformat}
> Now after code change - when the offset strategy is UNCOMMITTED_EARLIEST for
> the very first time when it is the first poll for the spout, lastBatchMeta is
> null and the kafka consumer seeks to the beginning offset. Should I be
> checking for last commit and starting from there? Likewise for the next fetch
> (when lastBatchMeta is not null), kafka consumer seeks to
> lastBatchMeta.getLastOffset() + 1. Should I be doing same here, checking for
> last commit and starting from there?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-3183) Topology Visualization is broken: shows components at most, but no more information
[ https://issues.apache.org/jira/browse/STORM-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3183. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~govindmenon], I merged into master. > Topology Visualization is broken: shows components at most, but no more > information > --- > > Key: STORM-3183 > URL: https://issues.apache.org/jira/browse/STORM-3183 > Project: Apache Storm > Issue Type: Bug > Components: storm-webapp >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Govind Menon >Priority: Blocker > Labels: pull-request-available > Fix For: 2.0.0 > > Attachments: Screen Shot 2018-08-06 at 3.48.23 PM.png > > Time Spent: 40m > Remaining Estimate: 0h > > Regardless of enabling ack it doesn't show stream between components. > > Please attach screenshot. > cc. [~govindmenon] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3182) Owner summary page in non-secured cluster shows 500 server error because of NullPointerException
[
https://issues.apache.org/jira/browse/STORM-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-3182.
-
Resolution: Fixed
Fix Version/s: 2.0.0
Thanks [~govindmenon] for providing quick fix, I merged into master.
> Owner summary page in non-secured cluster shows 500 server error because of
> NullPointerException
>
>
> Key: STORM-3182
> URL: https://issues.apache.org/jira/browse/STORM-3182
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-webapp
>Affects Versions: 2.0.0
>Reporter: Jungtaek Lim
>Assignee: Govind Menon
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Attachments: Screen Shot 2018-08-06 at 4.14.23 PM.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When opening owner summary page in non-secured cluster, all contents become
> empty except html template, and shows 500 error with below stack trace:
> {code}
> java.lang.NullPointerException
> at
> org.apache.storm.daemon.ui.filters.AuthorizedUserFilter.makeResponse(AuthorizedUserFilter.java:83)
> at
> org.apache.storm.daemon.ui.filters.AuthorizedUserFilter.filter(AuthorizedUserFilter.java:121)
> at
> org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:132)
> at
> org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:68)
> at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197)
> at
> org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
> {code}
> I'll attach the screenshot.
> I'm not sure it would work normally in secured cluster, since I don't have
> secured cluster.
> cc. [~govindmenon]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-3181) Topology stats doesn't show complete latency even for topology which ack is enabled
[ https://issues.apache.org/jira/browse/STORM-3181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3181. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~govindmenon] for providing quick fix, I merged into master. > Topology stats doesn't show complete latency even for topology which ack is > enabled > --- > > Key: STORM-3181 > URL: https://issues.apache.org/jira/browse/STORM-3181 > Project: Apache Storm > Issue Type: Bug > Components: storm-webapp >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Govind Menon >Priority: Blocker > Labels: pull-request-available > Fix For: 2.0.0 > > Attachments: Screen Shot 2018-08-06 at 3.49.51 PM.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > Please see attached screenshot. > cc. [~govindmenon] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3179) No data is available for Nimbus Summary in main UI page even Nimbus is running
[ https://issues.apache.org/jira/browse/STORM-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3179. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~govindmenon] for providing quick fix, I merged into master. > No data is available for Nimbus Summary in main UI page even Nimbus is running > -- > > Key: STORM-3179 > URL: https://issues.apache.org/jira/browse/STORM-3179 > Project: Apache Storm > Issue Type: Bug > Components: storm-webapp >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Govind Menon >Priority: Blocker > Labels: pull-request-available > Fix For: 2.0.0 > > Attachments: Screen Shot 2018-08-06 at 2.28.21 PM.png > > Time Spent: 20m > Remaining Estimate: 0h > > After STORM-1311, "Nimbus Summary" is showing "No data available in table", > even it runs with Nimbus. > Will attach screenshot which shows the behavior. > cc. [~govindmenon] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3180) Total executors in Cluster Summary in main UI page is not exposed even a topology is running
[ https://issues.apache.org/jira/browse/STORM-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3180. - Resolution: Fixed Fix Version/s: 2.0.0 Merged into master. > Total executors in Cluster Summary in main UI page is not exposed even a > topology is running > > > Key: STORM-3180 > URL: https://issues.apache.org/jira/browse/STORM-3180 > Project: Apache Storm > Issue Type: Bug > Components: storm-webapp >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Blocker > Labels: pull-request-available > Fix For: 2.0.0 > > Attachments: Screen Shot 2018-08-06 at 2.28.21 PM.png > > Time Spent: 50m > Remaining Estimate: 0h > > After STORM-1311, Total executors in Cluster Summary is showing nothing > (blank) even one topology is running. > Will attach screenshot which shows the behavior. > cc. [~govindmenon] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (STORM-3180) Total executors in Cluster Summary in main UI page is not exposed even a topology is running
[ https://issues.apache.org/jira/browse/STORM-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned STORM-3180: --- Assignee: Jungtaek Lim > Total executors in Cluster Summary in main UI page is not exposed even a > topology is running > > > Key: STORM-3180 > URL: https://issues.apache.org/jira/browse/STORM-3180 > Project: Apache Storm > Issue Type: Bug > Components: storm-webapp >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Blocker > Labels: pull-request-available > Attachments: Screen Shot 2018-08-06 at 2.28.21 PM.png > > Time Spent: 10m > Remaining Estimate: 0h > > After STORM-1311, Total executors in Cluster Summary is showing nothing > (blank) even one topology is running. > Will attach screenshot which shows the behavior. > cc. [~govindmenon] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (STORM-3183) Topology Visualization is broken: shows components at most, but no more information
Jungtaek Lim created STORM-3183: --- Summary: Topology Visualization is broken: shows components at most, but no more information Key: STORM-3183 URL: https://issues.apache.org/jira/browse/STORM-3183 Project: Apache Storm Issue Type: Bug Components: storm-webapp Affects Versions: 2.0.0 Reporter: Jungtaek Lim Attachments: Screen Shot 2018-08-06 at 3.48.23 PM.png Regardless of enabling ack it doesn't show stream between components. Please attach screenshot. cc. [~govindmenon] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-3182) Owner summary page in non-secured cluster shows 500 server error because of NullPointerException
[
https://issues.apache.org/jira/browse/STORM-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-3182:
Description:
When opening owner summary page in non-secured cluster, all contents become
empty except html template, and shows 500 error with below stack trace:
{code}
java.lang.NullPointerException
at
org.apache.storm.daemon.ui.filters.AuthorizedUserFilter.makeResponse(AuthorizedUserFilter.java:83)
at
org.apache.storm.daemon.ui.filters.AuthorizedUserFilter.filter(AuthorizedUserFilter.java:121)
at
org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:132)
at
org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:68)
at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197)
at
org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
{code}
I'll attach the screenshot.
I'm not sure it would work normally in secured cluster, since I don't have
secured cluster.
cc. [~govindmenon]
was:
When opening owner summary page in non-secured cluster, all contents become
empty except html template, and shows 500 error with below stack trace:
{code}
java.lang.NullPointerException
at
org.apache.storm.daemon.ui.filters.AuthorizedUserFilter.makeResponse(AuthorizedUserFilter.java:83)
at
org.apache.storm.daemon.ui.filters.AuthorizedUserFilter.filter(AuthorizedUserFilter.java:121)
at
org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:132)
at
org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:68)
at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197)
at
org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
{code}
I'll attach the screenshot.
I'm not sure it would work normally in secured cluster, since I don't have
secured cluster.
> Owner summary page in non-secured cluster shows 500 server error because of
> NullPointerException
>
>
> Key: STORM-3182
> URL: https://issues.apache.org/jira/browse/STORM-3182
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-webapp
>Affects Versions: 2.0.0
>Reporter: Jungtaek Lim
>Priority: Blocker
> Attachments: Screen Shot 2018-08-06 at 4.14.23 PM.png
>
>
> When opening owner summary page in non-secured cluster, all contents become
> empty except html template, and shows 500 error with below stack trace:
> {code}
> java.lang.NullPointerException
> at
> org.apache.storm.daemon.ui.filters.AuthorizedUserFilter.makeResponse(AuthorizedUserFilter.java:83)
> at
> org.apache.storm.daemon.ui.filters.AuthorizedUserFilter.filter(AuthorizedUserFilter.java:121)
> at
> org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:132)
> at
> org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:68)
> at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197)
> at
> org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
> {code}
> I'll attach the screenshot.
> I'm not sure it would work normally in secured cluster, since I don't have
> secured cluster.
> cc. [~govindmenon]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (STORM-3182) Owner summary page in non-secured cluster shows 500 server error because of NullPointerException
Jungtaek Lim created STORM-3182:
---
Summary: Owner summary page in non-secured cluster shows 500
server error because of NullPointerException
Key: STORM-3182
URL: https://issues.apache.org/jira/browse/STORM-3182
Project: Apache Storm
Issue Type: Bug
Components: storm-webapp
Affects Versions: 2.0.0
Reporter: Jungtaek Lim
Attachments: Screen Shot 2018-08-06 at 4.14.23 PM.png
When opening owner summary page in non-secured cluster, all contents become
empty except html template, and shows 500 error with below stack trace:
{code}
java.lang.NullPointerException
at
org.apache.storm.daemon.ui.filters.AuthorizedUserFilter.makeResponse(AuthorizedUserFilter.java:83)
at
org.apache.storm.daemon.ui.filters.AuthorizedUserFilter.filter(AuthorizedUserFilter.java:121)
at
org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:132)
at
org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:68)
at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197)
at
org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
{code}
I'll attach the screenshot.
I'm not sure it would work normally in secured cluster, since I don't have
secured cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (STORM-3181) Topology stats doesn't show complete latency even for topology which ack is enabled
[ https://issues.apache.org/jira/browse/STORM-3181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-3181: Description: Please see attached screenshot. cc. [~govindmenon] was:Please see attached screenshot. > Topology stats doesn't show complete latency even for topology which ack is > enabled > --- > > Key: STORM-3181 > URL: https://issues.apache.org/jira/browse/STORM-3181 > Project: Apache Storm > Issue Type: Bug > Components: storm-webapp >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Priority: Blocker > Attachments: Screen Shot 2018-08-06 at 3.49.51 PM.png > > > Please see attached screenshot. > cc. [~govindmenon] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-3180) Total executors in Cluster Summary in main UI page is not exposed even a topology is running
[
https://issues.apache.org/jira/browse/STORM-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569779#comment-16569779
]
Jungtaek Lim commented on STORM-3180:
-
"/api/v1/cluster/summary" returns
{noformat}
{
totalExecutors: 15,
stormVersion: "2.0.0-SNAPSHOT",
fragmentedCpu: 0,
totalMem: 4096,
availCpu: 250,
slotsTotal: 4,
slotsUsed: 1,
bugtracker-url: null,
topologies: 1,
totalCpu: 400,
cpuAssignedPercentUtil: "62.5",
availMem: 2176,
slotsFree: 3,
memAssignedPercentUtil: "53.125",
central-log-url: null,
user: null,
fragmentedMem: 0,
tasksTotal: 15,
schedulerDisplayResource: false,
supervisors: 1
}
{noformat}
totalExecutors should be executorsTotal to load the value from UI.
Actually there's no consistency among field names, some fields have total +
resource whereas some others have resource + total. It may be breaking change
to client which leverages REST API, so better to file a new issue and handle it
if we think it is worth to fix it.
> Total executors in Cluster Summary in main UI page is not exposed even a
> topology is running
>
>
> Key: STORM-3180
> URL: https://issues.apache.org/jira/browse/STORM-3180
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-webapp
>Affects Versions: 2.0.0
>Reporter: Jungtaek Lim
>Priority: Blocker
> Attachments: Screen Shot 2018-08-06 at 2.28.21 PM.png
>
>
> After STORM-1311, Total executors in Cluster Summary is showing nothing
> (blank) even one topology is running.
> Will attach screenshot which shows the behavior.
> cc. [~govindmenon]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (STORM-3180) Total executors in Cluster Summary in main UI page is not exposed even a topology is running
Jungtaek Lim created STORM-3180: --- Summary: Total executors in Cluster Summary in main UI page is not exposed even a topology is running Key: STORM-3180 URL: https://issues.apache.org/jira/browse/STORM-3180 Project: Apache Storm Issue Type: Bug Components: storm-webapp Affects Versions: 2.0.0 Reporter: Jungtaek Lim Attachments: Screen Shot 2018-08-06 at 2.28.21 PM.png After STORM-1311, Total executors in Cluster Summary is showing nothing (blank) even one topology is running. Will attach screenshot which shows the behavior. cc. [~govindmenon] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-3179) No data is available for Nimbus Summary in main UI page even Nimbus is running
[ https://issues.apache.org/jira/browse/STORM-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-3179: Attachment: Screen Shot 2018-08-06 at 2.28.21 PM.png > No data is available for Nimbus Summary in main UI page even Nimbus is running > -- > > Key: STORM-3179 > URL: https://issues.apache.org/jira/browse/STORM-3179 > Project: Apache Storm > Issue Type: Bug > Components: storm-webapp >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Priority: Blocker > Attachments: Screen Shot 2018-08-06 at 2.28.21 PM.png > > > After STORM-1311, "Nimbus Summary" is showing "No data available in table", > even it runs with Nimbus. > Will attach screenshot which shows the behavior. > cc. [~govindmenon] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (STORM-3179) No data is available for Nimbus Summary in main UI page even Nimbus is running
Jungtaek Lim created STORM-3179: --- Summary: No data is available for Nimbus Summary in main UI page even Nimbus is running Key: STORM-3179 URL: https://issues.apache.org/jira/browse/STORM-3179 Project: Apache Storm Issue Type: Bug Components: storm-webapp Affects Versions: 2.0.0 Reporter: Jungtaek Lim After STORM-1311, "Nimbus Summary" is showing "No data available in table", even it runs with Nimbus. Will attach screenshot which shows the behavior. cc. [~govindmenon] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3158) improve login failure message
[ https://issues.apache.org/jira/browse/STORM-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3158. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~agresch], I merged into master. > improve login failure message > - > > Key: STORM-3158 > URL: https://issues.apache.org/jira/browse/STORM-3158 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron Gresch >Assignee: Aaron Gresch >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > We had an issue where the DRPC server (on an older version of storm) was > misconfigured and constantly restarting without logging shutdown messages. > One of the warning messages that did log should have been an error and could > provide more useful information about fixing the issue. > warning message: > 2018-07-20 15:09:30.191 clojure-agent-send-off-pool-0 > b.s.s.a.k.ServerCallbackHandler [WARN] No password found for user: null > > hopefully more helpful: > 2018-07-20 15:44:39.835 clojure-agent-send-off-pool-0 > b.s.s.a.k.ServerCallbackHandler [ERROR] No password found for user: null, > validate klist matches jaas conf -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-3121) Fix flaky metrics tests in storm-core
[
https://issues.apache.org/jira/browse/STORM-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-3121:
Fix Version/s: 1.2.3
> Fix flaky metrics tests in storm-core
> -
>
> Key: STORM-3121
> URL: https://issues.apache.org/jira/browse/STORM-3121
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0, 1.2.3
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> The tests are flaky, but only rarely fail. I've only seen them fail on Travis
> when Travis is under load.
> Example failures:
> {code}
> classname: org.apache.storm.metrics-test / testname:
> test-custom-metric-with-multi-tasks
> expected: (clojure.core/= [1 0 0 0 0 0 2] (clojure.core/subvec
> (org.apache.storm.metrics-test/lookup-bucket-by-comp-id-&-metric-name! "2"
> "my-custom-metric") 0 N__3207__auto__))
> actual: (not (clojure.core/= [1 0 0 0 0 0 2] [1 0 0 0 0 0 0]))
> at: test_runner.clj:105
> {code}
> {code}
> classname: org.apache.storm.metrics-test / testname: test-builtin-metrics-2
> expected: (clojure.core/= [1 1] (clojure.core/subvec
> (org.apache.storm.metrics-test/lookup-bucket-by-comp-id-&-metric-name!
> "myspout" "__emit-count/default") 0 N__3207__auto__))
> actual: (not (clojure.core/= [1 1] [1 0]))
> at: test_runner.clj:105
> {code}
> The problem is that the tests increment metrics counters in the executor
> async loops, then expect the counters to end up in exact metrics buckets. The
> creation of a bucket is triggered by the metrics timer. The timer is included
> in time simulation and LocalCluster.waitForIdle, but the executor async loop
> isn't. There isn't any guarantee that the executor async loop gets to run
> when the test does a sequence like
> {code}
> Time.advanceClusterTime
> cluster.waitForIdle
> {code}
> because the waitForIdle check doesn't know about the executor async loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-3161) Local mode should force setting min replication count to 1
[ https://issues.apache.org/jira/browse/STORM-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3161. - Resolution: Fixed Fix Version/s: 1.2.3 2.0.0 Merged into master and 1.x-branch. > Local mode should force setting min replication count to 1 > -- > > Key: STORM-3161 > URL: https://issues.apache.org/jira/browse/STORM-3161 > Project: Apache Storm > Issue Type: Bug > Components: storm-server >Affects Versions: 2.0.0, 1.2.3 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0, 1.2.3 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > When topology.min.replication.count is set to more than 1, nimbus in local > mode never achieve condition for replication, hence stuck on handling blobs. > We should force set it to 1 in local mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-3161) Local mode should force setting min replication count to 1
[ https://issues.apache.org/jira/browse/STORM-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-3161: Component/s: (was: storm-client) storm-server > Local mode should force setting min replication count to 1 > -- > > Key: STORM-3161 > URL: https://issues.apache.org/jira/browse/STORM-3161 > Project: Apache Storm > Issue Type: Bug > Components: storm-server >Affects Versions: 2.0.0, 1.2.3 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Major > > When topology.min.replication.count is set to more than 1, nimbus in local > mode never achieve condition for replication, hence stuck on handling blobs. > We should force set it to 1 in local mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (STORM-3161) Local mode should force setting min replication count to 1
Jungtaek Lim created STORM-3161: --- Summary: Local mode should force setting min replication count to 1 Key: STORM-3161 URL: https://issues.apache.org/jira/browse/STORM-3161 Project: Apache Storm Issue Type: Bug Components: storm-client Affects Versions: 2.0.0, 1.2.3 Reporter: Jungtaek Lim Assignee: Jungtaek Lim When topology.min.replication.count is set to more than 1, nimbus in local mode never achieve condition for replication, hence stuck on handling blobs. We should force set it to 1 in local mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2953) Remove storm-kafka in 2.0.0
[ https://issues.apache.org/jira/browse/STORM-2953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2953. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~Srdo], I merged into master. > Remove storm-kafka in 2.0.0 > --- > > Key: STORM-2953 > URL: https://issues.apache.org/jira/browse/STORM-2953 > Project: Apache Storm > Issue Type: Task > Components: storm-kafka >Affects Versions: 2.0.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Blocker > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Remove storm-kafka from master. See the thread at > http://mail-archives.apache.org/mod_mbox/storm-dev/201802.mbox/%3CCAF5108hZBRf7Y%2BOFPif9kSw9%2BuZuPx5iBgSt1TOX5zMvF0-NfQ%40mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2972) Replace storm-kafka in storm-sql-kafka with storm-kafka-client
[ https://issues.apache.org/jira/browse/STORM-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2972. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~Srdo], I merged into master. > Replace storm-kafka in storm-sql-kafka with storm-kafka-client > -- > > Key: STORM-2972 > URL: https://issues.apache.org/jira/browse/STORM-2972 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-sql >Affects Versions: 2.0.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2972) Replace storm-kafka in storm-sql-kafka with storm-kafka-client
[ https://issues.apache.org/jira/browse/STORM-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545816#comment-16545816 ] Jungtaek Lim commented on STORM-2972: - Now we merged STORM-2406, so we just need to match Spout and Bolt for storm-kafka-client. > Replace storm-kafka in storm-sql-kafka with storm-kafka-client > -- > > Key: STORM-2972 > URL: https://issues.apache.org/jira/browse/STORM-2972 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-sql >Affects Versions: 2.0.0 >Reporter: Stig Rohde Døssing >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-3151) Negative Scheduling Resource/Overscheduling issue
[
https://issues.apache.org/jira/browse/STORM-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545742#comment-16545742
]
Jungtaek Lim commented on STORM-3151:
-
We're using "Fix version/s" as actual fix version, so it should be set when
patch is going to be merged in.
> Negative Scheduling Resource/Overscheduling issue
> -
>
> Key: STORM-3151
> URL: https://issues.apache.org/jira/browse/STORM-3151
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-server
>Affects Versions: 2.0.0
>Reporter: Zhengdai Hu
>Priority: Critical
>
> Possible overscheduling captured when follow steps are performed (Logging is
> added in STORM-3147)
> 1) launch nimbus & zookeeper
> 2) launch supervisor 1
> 3) launch topology 1 (I used org.apache.storm.starter.WordCountTopology)
> 4) launch supervisor 2
> 5) launch topology 2 (I used org.apache.storm.starter.ExclamationTopology)
> {noformat}
> 2018-07-13 12:58:43.196 o.a.s.d.n.Nimbus timer [WARN] Memory over-scheduled
> on 176ec6d4-2df3-40ca-95ca-c84a81dbcc22-172.130.97.212
> {noformat}
> Indicating there may be issues inside scheduler.
> It is discovered when I ported ClusterSummay to StormMetrics
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (STORM-3153) Restore storm sql provider tests affected via STORM-2406
Jungtaek Lim created STORM-3153: --- Summary: Restore storm sql provider tests affected via STORM-2406 Key: STORM-3153 URL: https://issues.apache.org/jira/browse/STORM-3153 Project: Apache Storm Issue Type: Improvement Components: storm-sql Affects Versions: 2.0.0 Reporter: Jungtaek Lim Assignee: Jungtaek Lim Current proposed patch for STORM-2406 gets rid of major parts of tests for provider, because we can't do same test newly changed code. To restore tests additional code change is needed. This issue is to track the effort of making change of Storm SQL to be able to restore previous tests. [https://github.com/apache/storm/blob/c9e9a7c294458c8bb1166e0646a5fa580661e21e/sql/storm-sql-external/storm-sql-hdfs/src/test/org/apache/storm/sql/hdfs/TestHdfsDataSourcesProvider.java#L90] [https://github.com/apache/storm/blob/c9e9a7c294458c8bb1166e0646a5fa580661e21e/sql/storm-sql-external/storm-sql-kafka/src/test/org/apache/storm/sql/kafka/TestKafkaDataSourcesProvider.java] [https://github.com/apache/storm/blob/c9e9a7c294458c8bb1166e0646a5fa580661e21e/sql/storm-sql-external/storm-sql-mongodb/src/test/org/apache/storm/sql/mongodb/TestMongoDataSourcesProvider.java] [https://github.com/apache/storm/blob/c9e9a7c294458c8bb1166e0646a5fa580661e21e/sql/storm-sql-external/storm-sql-redis/src/test/org/apache/storm/sql/redis/TestRedisDataSourcesProvider.java] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2947) Review and fix/remove deprecated things in Storm 2.0.0
[ https://issues.apache.org/jira/browse/STORM-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544809#comment-16544809 ] Jungtaek Lim commented on STORM-2947: - [~roshan_naik] It make sense, but could you file a new issue since [~Srdo] already picked this up and raised two pull requests which one was already merged and another one is close to be merged? Thanks in advance! > Review and fix/remove deprecated things in Storm 2.0.0 > -- > > Key: STORM-2947 > URL: https://issues.apache.org/jira/browse/STORM-2947 > Project: Apache Storm > Issue Type: Task > Components: storm-client, storm-hdfs, storm-kafka, storm-server, > storm-solr >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Major > Labels: pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > > We've been deprecating the things but haven't have time to replace/get rid of > them. It should be better if we have time to review and address them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2972) Replace storm-kafka in storm-sql-kafka with storm-kafka-client
[ https://issues.apache.org/jira/browse/STORM-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542999#comment-16542999 ] Jungtaek Lim commented on STORM-2972: - If we would really want to merge STORM-2406 in, we need to review and merge that first, so that we don't do redundant work. (Now we should deal with Trident, whereas after STORM-2406 we just need to use KafkaBolt.) > Replace storm-kafka in storm-sql-kafka with storm-kafka-client > -- > > Key: STORM-2972 > URL: https://issues.apache.org/jira/browse/STORM-2972 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-sql >Affects Versions: 2.0.0 >Reporter: Stig Rohde Døssing >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2974) Add a transactional non-opaque spout to storm-kafka-client
[ https://issues.apache.org/jira/browse/STORM-2974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542992#comment-16542992 ] Jungtaek Lim commented on STORM-2974: - [~Srdo] OK sounds nice. Thanks again! > Add a transactional non-opaque spout to storm-kafka-client > -- > > Key: STORM-2974 > URL: https://issues.apache.org/jira/browse/STORM-2974 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-kafka-client >Affects Versions: 2.0.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2974) Add a transactional non-opaque spout to storm-kafka-client
[ https://issues.apache.org/jira/browse/STORM-2974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542982#comment-16542982 ] Jungtaek Lim commented on STORM-2974: - [~Srdo] I guess you don't plan to port back to 1.x, right? If not please raise a pull request on 1.x-branch as well, since current patch contains lambda so can't cherry pick easily. > Add a transactional non-opaque spout to storm-kafka-client > -- > > Key: STORM-2974 > URL: https://issues.apache.org/jira/browse/STORM-2974 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-kafka-client >Affects Versions: 2.0.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2974) Add a transactional non-opaque spout to storm-kafka-client
[ https://issues.apache.org/jira/browse/STORM-2974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2974. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks for your patience, [~Srdo]! I merged into master. > Add a transactional non-opaque spout to storm-kafka-client > -- > > Key: STORM-2974 > URL: https://issues.apache.org/jira/browse/STORM-2974 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-kafka-client >Affects Versions: 2.0.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-3013) Deactivated topology restarts if data flows into Kafka
[
https://issues.apache.org/jira/browse/STORM-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-3013:
Fix Version/s: 1.2.3
> Deactivated topology restarts if data flows into Kafka
> --
>
> Key: STORM-3013
> URL: https://issues.apache.org/jira/browse/STORM-3013
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
>Affects Versions: 2.0.0, 1.2.1
>Reporter: Ajeesh B
>Assignee: Stig Rohde Døssing
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0, 1.2.3
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Hi, I have deactivated the storm topology & then if I produce any records
> into Kafka, Storm throws an exception. Exception follows,
> {code:java}
> 2018-03-28 09:50:23.804 o.a.s.d.executor Thread-83-kafkaLogs-executor[130
> 130] [INFO] Deactivating spout kafkaLogs:(130)
> 2018-03-28 09:51:01.289 o.a.s.util Thread-17-kafkaLogs-executor[139 139]
> [ERROR] Async loop died!
> java.lang.RuntimeException: java.lang.IllegalStateException: This consumer
> has already been closed.
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:522)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:487)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:477)
> ~[storm-core-1.2.1.jar:1.2.1]
> at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:70)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$fn__4975$fn__4990$fn__5021.invoke(executor.clj:634)
> ~[storm-core-1.2.1.jar:1.2.1]
> at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484)
> [storm-core-1.2.1.jar:1.2.1]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
> Caused by: java.lang.IllegalStateException: This consumer has already been
> closed.
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.acquireAndEnsureOpen(KafkaConsumer.java:1787)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.beginningOffsets(KafkaConsumer.java:1622)
> ~[stormjar.jar:?]
> at
> org.apache.storm.kafka.spout.metrics.KafkaOffsetMetric.getValueAndReset(KafkaOffsetMetric.java:79)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$metrics_tick$fn__4899.invoke(executor.clj:345)
> ~[storm-core-1.2.1.jar:1.2.1]
> at clojure.core$map$fn__4553.invoke(core.clj:2622) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
> at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?]
> at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?]
> at clojure.core$filter$fn__4580.invoke(core.clj:2679) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
> at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.7.0.jar:?]
> at clojure.lang.RT.next(RT.java:674) ~[clojure-1.7.0.jar:?]
> at clojure.core$next__4112.invoke(core.clj:64) ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6523.invoke(protocols.clj:170)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6478$G__6473__6487.invoke(protocols.clj:19)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6506.invoke(protocols.clj:101)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13)
> ~[clojure-1.7.0.jar:?]
> at clojure.core$reduce.invoke(core.clj:6519) ~[clojure-1.7.0.jar:?]
> at clojure.core$into.invoke(core.clj:6600) ~[clojure-1.7.0.jar:?]
> at org.apache.storm.daemon.executor$metrics_tick.invoke(executor.clj:349)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$fn__4975$tuple_action_fn__4981.invoke(executor.clj:522)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__4964.invoke(executor.clj:471)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4475.onEvent(disruptor.clj:41)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:509)
> ~[storm-core-1.2.1.jar:1.2.1]
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (STORM-3013) Deactivated topology restarts if data flows into Kafka
[
https://issues.apache.org/jira/browse/STORM-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542429#comment-16542429
]
Jungtaek Lim commented on STORM-3013:
-
Also merged into 1.x-branch.
> Deactivated topology restarts if data flows into Kafka
> --
>
> Key: STORM-3013
> URL: https://issues.apache.org/jira/browse/STORM-3013
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
>Affects Versions: 2.0.0, 1.2.1
>Reporter: Ajeesh B
>Assignee: Stig Rohde Døssing
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0, 1.2.3
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Hi, I have deactivated the storm topology & then if I produce any records
> into Kafka, Storm throws an exception. Exception follows,
> {code:java}
> 2018-03-28 09:50:23.804 o.a.s.d.executor Thread-83-kafkaLogs-executor[130
> 130] [INFO] Deactivating spout kafkaLogs:(130)
> 2018-03-28 09:51:01.289 o.a.s.util Thread-17-kafkaLogs-executor[139 139]
> [ERROR] Async loop died!
> java.lang.RuntimeException: java.lang.IllegalStateException: This consumer
> has already been closed.
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:522)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:487)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:477)
> ~[storm-core-1.2.1.jar:1.2.1]
> at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:70)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$fn__4975$fn__4990$fn__5021.invoke(executor.clj:634)
> ~[storm-core-1.2.1.jar:1.2.1]
> at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484)
> [storm-core-1.2.1.jar:1.2.1]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
> Caused by: java.lang.IllegalStateException: This consumer has already been
> closed.
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.acquireAndEnsureOpen(KafkaConsumer.java:1787)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.beginningOffsets(KafkaConsumer.java:1622)
> ~[stormjar.jar:?]
> at
> org.apache.storm.kafka.spout.metrics.KafkaOffsetMetric.getValueAndReset(KafkaOffsetMetric.java:79)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$metrics_tick$fn__4899.invoke(executor.clj:345)
> ~[storm-core-1.2.1.jar:1.2.1]
> at clojure.core$map$fn__4553.invoke(core.clj:2622) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
> at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?]
> at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?]
> at clojure.core$filter$fn__4580.invoke(core.clj:2679) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
> at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.7.0.jar:?]
> at clojure.lang.RT.next(RT.java:674) ~[clojure-1.7.0.jar:?]
> at clojure.core$next__4112.invoke(core.clj:64) ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6523.invoke(protocols.clj:170)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6478$G__6473__6487.invoke(protocols.clj:19)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6506.invoke(protocols.clj:101)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13)
> ~[clojure-1.7.0.jar:?]
> at clojure.core$reduce.invoke(core.clj:6519) ~[clojure-1.7.0.jar:?]
> at clojure.core$into.invoke(core.clj:6600) ~[clojure-1.7.0.jar:?]
> at org.apache.storm.daemon.executor$metrics_tick.invoke(executor.clj:349)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$fn__4975$tuple_action_fn__4981.invoke(executor.clj:522)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__4964.invoke(executor.clj:471)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4475.onEvent(disruptor.clj:41)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:509)
> ~[storm-core-1.2.1.jar:1.2.1]
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (STORM-3046) Getting a NPE leading worker to die when starting a topology.
[
https://issues.apache.org/jira/browse/STORM-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542425#comment-16542425
]
Jungtaek Lim commented on STORM-3046:
-
Also merged into 1.x-branch as well.
> Getting a NPE leading worker to die when starting a topology.
> -
>
> Key: STORM-3046
> URL: https://issues.apache.org/jira/browse/STORM-3046
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client, trident
>Affects Versions: 1.2.1
>Reporter: Kush Khandelwal
>Assignee: Stig Rohde Døssing
>Priority: Blocker
> Labels: kafka, pull-request-available, storm-kafka-client,
> trident
> Fix For: 2.0.0, 1.2.3
>
> Attachments: TestTopology.java
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I am using storm-core and storm-kafka-client version 1.2.1 and kafka clients
> version 1.1.0.
> We have an external kafka from where we get the messages.
> Whenever I try to run the topology, I get a NPE, which leads to the worker
> getting died.
> If I set poll strategy to earliest and the topic already contains some
> messages, it works fine.
> I have used a custom record translator which is working fine.
> Can someone please help me fix the issue?
> Thanks.
>
> Error -
> 10665 [Thread-58-spout-handle-rule-local-kafka-spout-executor[26 26]] ERROR
> o.a.s.util - Async loop died!
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:522)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:487)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:74)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$fn__5043$fn__5056$fn__5109.invoke(executor.clj:861)
> ~[storm-core-1.2.1.jar:1.2.1]
> at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484)
> [storm-core-1.2.1.jar:1.2.1]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
> Caused by: java.lang.NullPointerException
> at
> org.apache.storm.kafka.spout.trident.KafkaTridentSpoutEmitter.seek(KafkaTridentSpoutEmitter.java:193)
> ~[storm-kafka-client-1.2.1.jar:1.2.1]
> at
> org.apache.storm.kafka.spout.trident.KafkaTridentSpoutEmitter.emitPartitionBatch(KafkaTridentSpoutEmitter.java:127)
> ~[storm-kafka-client-1.2.1.jar:1.2.1]
> at
> org.apache.storm.kafka.spout.trident.KafkaTridentSpoutEmitter.emitPartitionBatch(KafkaTridentSpoutEmitter.java:51)
> ~[storm-kafka-client-1.2.1.jar:1.2.1]
> at
> org.apache.storm.trident.spout.OpaquePartitionedTridentSpoutExecutor$Emitter.emitBatch(OpaquePartitionedTridentSpoutExecutor.java:141)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.trident.spout.TridentSpoutExecutor.execute(TridentSpoutExecutor.java:82)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.trident.topology.TridentBoltExecutor.execute(TridentBoltExecutor.java:383)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$fn__5043$tuple_action_fn__5045.invoke(executor.clj:739)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__4964.invoke(executor.clj:468)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4475.onEvent(disruptor.clj:41)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:509)
> ~[storm-core-1.2.1.jar:1.2.1]
> ... 6 more
>
>
> Topology class -
>
>
>
>
> import org.apache.storm.Config;
> import org.apache.storm.LocalCluster;
> import org.apache.storm.StormSubmitter;
> import org.apache.storm.generated.*;
> import org.apache.storm.kafka.spout.KafkaSpoutConfig;
> import org.apache.storm.kafka.spout.trident.KafkaTridentSpoutOpaque;
> import org.apache.storm.trident.Stream;
> import org.apache.storm.trident.TridentState;
> import org.apache.storm.trident.TridentTopology;
> import org.apache.storm.tuple.Fields;
> import java.util.Properties;
>
> public class TestTopology {
>
> private static StormTopology buildTopology(Properties stormProperties) {
>
> Properties kafkaProperties = getProperties("/kafka.properties");
> TridentTopology topology = new TridentTopology();
> Fields stageArguments = new Fields("test", "issue");
> KafkaSpoutConfig kafkaSpoutConfig =
> KafkaSpoutConfig.builder(kafkaProperties.getProperty("bootstrap.servers"),
> "test")
> .setFirstPollOffsetStrategy(KafkaSpoutConfig.FirstPollOffsetStrategy.LATEST)
> .setProcessingGuarantee(KafkaSpoutConfig.ProcessingGuarantee.AT_MOST_ONCE)
>
[jira] [Updated] (STORM-3046) Getting a NPE leading worker to die when starting a topology.
[
https://issues.apache.org/jira/browse/STORM-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-3046:
Fix Version/s: 1.2.3
> Getting a NPE leading worker to die when starting a topology.
> -
>
> Key: STORM-3046
> URL: https://issues.apache.org/jira/browse/STORM-3046
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client, trident
>Affects Versions: 1.2.1
>Reporter: Kush Khandelwal
>Assignee: Stig Rohde Døssing
>Priority: Blocker
> Labels: kafka, pull-request-available, storm-kafka-client,
> trident
> Fix For: 2.0.0, 1.2.3
>
> Attachments: TestTopology.java
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I am using storm-core and storm-kafka-client version 1.2.1 and kafka clients
> version 1.1.0.
> We have an external kafka from where we get the messages.
> Whenever I try to run the topology, I get a NPE, which leads to the worker
> getting died.
> If I set poll strategy to earliest and the topic already contains some
> messages, it works fine.
> I have used a custom record translator which is working fine.
> Can someone please help me fix the issue?
> Thanks.
>
> Error -
> 10665 [Thread-58-spout-handle-rule-local-kafka-spout-executor[26 26]] ERROR
> o.a.s.util - Async loop died!
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:522)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:487)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:74)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$fn__5043$fn__5056$fn__5109.invoke(executor.clj:861)
> ~[storm-core-1.2.1.jar:1.2.1]
> at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484)
> [storm-core-1.2.1.jar:1.2.1]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
> Caused by: java.lang.NullPointerException
> at
> org.apache.storm.kafka.spout.trident.KafkaTridentSpoutEmitter.seek(KafkaTridentSpoutEmitter.java:193)
> ~[storm-kafka-client-1.2.1.jar:1.2.1]
> at
> org.apache.storm.kafka.spout.trident.KafkaTridentSpoutEmitter.emitPartitionBatch(KafkaTridentSpoutEmitter.java:127)
> ~[storm-kafka-client-1.2.1.jar:1.2.1]
> at
> org.apache.storm.kafka.spout.trident.KafkaTridentSpoutEmitter.emitPartitionBatch(KafkaTridentSpoutEmitter.java:51)
> ~[storm-kafka-client-1.2.1.jar:1.2.1]
> at
> org.apache.storm.trident.spout.OpaquePartitionedTridentSpoutExecutor$Emitter.emitBatch(OpaquePartitionedTridentSpoutExecutor.java:141)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.trident.spout.TridentSpoutExecutor.execute(TridentSpoutExecutor.java:82)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.trident.topology.TridentBoltExecutor.execute(TridentBoltExecutor.java:383)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$fn__5043$tuple_action_fn__5045.invoke(executor.clj:739)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__4964.invoke(executor.clj:468)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4475.onEvent(disruptor.clj:41)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:509)
> ~[storm-core-1.2.1.jar:1.2.1]
> ... 6 more
>
>
> Topology class -
>
>
>
>
> import org.apache.storm.Config;
> import org.apache.storm.LocalCluster;
> import org.apache.storm.StormSubmitter;
> import org.apache.storm.generated.*;
> import org.apache.storm.kafka.spout.KafkaSpoutConfig;
> import org.apache.storm.kafka.spout.trident.KafkaTridentSpoutOpaque;
> import org.apache.storm.trident.Stream;
> import org.apache.storm.trident.TridentState;
> import org.apache.storm.trident.TridentTopology;
> import org.apache.storm.tuple.Fields;
> import java.util.Properties;
>
> public class TestTopology {
>
> private static StormTopology buildTopology(Properties stormProperties) {
>
> Properties kafkaProperties = getProperties("/kafka.properties");
> TridentTopology topology = new TridentTopology();
> Fields stageArguments = new Fields("test", "issue");
> KafkaSpoutConfig kafkaSpoutConfig =
> KafkaSpoutConfig.builder(kafkaProperties.getProperty("bootstrap.servers"),
> "test")
> .setFirstPollOffsetStrategy(KafkaSpoutConfig.FirstPollOffsetStrategy.LATEST)
> .setProcessingGuarantee(KafkaSpoutConfig.ProcessingGuarantee.AT_MOST_ONCE)
> .setRecordTranslator(new RecordTranslator(), stageArguments)
>
[jira] [Resolved] (STORM-3141) NPE in WorkerState.transferLocalBatch when receiving messages for a task that isn't the first task assigned to the executor
[
https://issues.apache.org/jira/browse/STORM-3141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-3141.
-
Resolution: Fixed
Fix Version/s: 2.0.0
Looks like Bobby forgot to close this. Marked as resolved. Thanks again [~Srdo]!
> NPE in WorkerState.transferLocalBatch when receiving messages for a task that
> isn't the first task assigned to the executor
> ---
>
> Key: STORM-3141
> URL: https://issues.apache.org/jira/browse/STORM-3141
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-client
>Affects Versions: 2.0.0
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Critical
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> {code}
> 2018-07-02 20:32:28.944 [Worker-Transfer] ERROR org.apache.storm.utils.Utils
> - Async loop died!
> java.lang.NullPointerException: null
> at
> org.apache.storm.daemon.worker.WorkerState.transferLocalBatch(WorkerState.java:538)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:71)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.messaging.local.Context$LocalClient.send(Context.java:194)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.utils.TransferDrainer.send(TransferDrainer.java:53)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.daemon.worker.WorkerTransfer.flush(WorkerTransfer.java:100)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.utils.JCQueue.consumeImpl(JCQueue.java:146)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.utils.JCQueue.consume(JCQueue.java:110)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.utils.JCQueue.consume(JCQueue.java:101)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.daemon.worker.WorkerTransfer.lambda$makeTransferThread$0(WorkerTransfer.java:82)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.utils.Utils$2.run(Utils.java:353)
> [storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
> 2018-07-02 20:32:28.945 [Worker-Transfer] ERROR org.apache.storm.utils.Utils
> - Async loop died!
> java.lang.RuntimeException: java.lang.NullPointerException
> at org.apache.storm.utils.Utils$2.run(Utils.java:368)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
> Caused by: java.lang.NullPointerException
> at
> org.apache.storm.daemon.worker.WorkerState.transferLocalBatch(WorkerState.java:538)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:71)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.messaging.local.Context$LocalClient.send(Context.java:194)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.utils.TransferDrainer.send(TransferDrainer.java:53)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.daemon.worker.WorkerTransfer.flush(WorkerTransfer.java:100)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.utils.JCQueue.consumeImpl(JCQueue.java:146)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.utils.JCQueue.consume(JCQueue.java:110)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.utils.JCQueue.consume(JCQueue.java:101)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.daemon.worker.WorkerTransfer.lambda$makeTransferThread$0(WorkerTransfer.java:82)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.utils.Utils$2.run(Utils.java:353)
> ~[storm-client-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> ... 1 more
> {code}
> When tuples are received, the method looks up which JCQueue to send to. It
> does this with the tuple destination task id. The map it looks in only has
> the queues by the starting task id of the executor's range, so if the
> destination is e.g. task 4 for an executor with assignment [3, 4], we hit an
> NPE.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-3134) upload-credentials imporvements
[ https://issues.apache.org/jira/browse/STORM-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3134. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~revans2], I merged into master. > upload-credentials imporvements > --- > > Key: STORM-3134 > URL: https://issues.apache.org/jira/browse/STORM-3134 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > There are some things that would be nice to do with AutoCredentials to make > debugging and usage simpler. > > First it would be ideal if we could use the topology conf as a starting point > for AutoCredentials instead of requiring the user to provide all configs from > the get go. > > It would also be nice to warn users if they are overriding the list of > AutoCreds in a way that makes it so some of the creds might not be > interpreted properly (meaning they are uploading creds that there is no > plugin on the worker to interpret them) > > At the same time it would also be really nice to have some help with > debugging, both as an admin, but also as a developer to be able to see what > creds currently exist (not necessarily the content of the credentials, but > the keys). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-2947) Review and fix/remove deprecated things in Storm 2.0.0
[ https://issues.apache.org/jira/browse/STORM-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2947: This shouldn't be a blocker, whether it might be nice to address... > Review and fix/remove deprecated things in Storm 2.0.0 > -- > > Key: STORM-2947 > URL: https://issues.apache.org/jira/browse/STORM-2947 > Project: Apache Storm > Issue Type: Task > Components: storm-client, storm-hdfs, storm-kafka, storm-server, > storm-solr >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Major > > We've been deprecating the things but haven't have time to replace/get rid of > them. It should be better if we have time to review and address them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-2963) Updates to Performance.md
[ https://issues.apache.org/jira/browse/STORM-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2963: I wouldn't think this is a blocker for Storm 2.0.0, since Performance.md catches up all performance patches, and there's no further performance patch (except minor) after STORM-2958. Will remove epic link. > Updates to Performance.md > -- > > Key: STORM-2963 > URL: https://issues.apache.org/jira/browse/STORM-2963 > Project: Apache Storm > Issue Type: Documentation >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Roshan Naik >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-3100) Minor optimization: Replace HashMap with an array backed data structure for faster lookups
[ https://issues.apache.org/jira/browse/STORM-3100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16537969#comment-16537969 ] Jungtaek Lim commented on STORM-3100: - I would remove epic link for this, since this is rather improvement, not blocker for Storm 2.0.0. There's still a chance for this to be included in Storm 2.0.0 but shouldn't explicitly mark as a blocker. > Minor optimization: Replace HashMap with an array backed data > structure for faster lookups > -- > > Key: STORM-3100 > URL: https://issues.apache.org/jira/browse/STORM-3100 > Project: Apache Storm > Issue Type: Improvement >Reporter: Roshan Naik >Assignee: Roshan Naik >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > * Introduce _CustomIndexArray_: An array backed data structure to speedup > HashMap use cases *in critical path*. It needs to supported -ve > indexing and a user defined (on construction) Upper and Lower Index range. > Does not need to be dynamically resizable given the nature of use cases we > have. > * Use this data structure for _GeneralTopologyContext._taskToComponent_ > mapping which is looked up in the critical path _Task.getOutgoingTasks._ This > lookup happens at least once for every emit and consequently can happen > millions of times per second. > * Also use this for _JCQueue.localReceiveQueues_ where the basic idea is > already in use but not in a reusable manner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-2945) Nail down and document how to support background emits in Spouts and Bolts
[ https://issues.apache.org/jira/browse/STORM-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2945: Removing epic link since I don't think this is a blocker for Storm 2.0.0. > Nail down and document how to support background emits in Spouts and Bolts > -- > > Key: STORM-2945 > URL: https://issues.apache.org/jira/browse/STORM-2945 > Project: Apache Storm > Issue Type: Documentation >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3056) Add a test for quickly rebinding to a port
[ https://issues.apache.org/jira/browse/STORM-3056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3056. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~raghavgautam], I merged into master. It doesn't seem to cleanly cherry picked into 1.x-branch as well as 1.1.x-branch. It would be nice if you could submit a patch for 1.x-branch but only applying to master branch would be OK. > Add a test for quickly rebinding to a port > -- > > Key: STORM-3056 > URL: https://issues.apache.org/jira/browse/STORM-3056 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.1.2, 1.2.1 >Reporter: Raghav Kumar Gautam >Assignee: Raghav Kumar Gautam >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > We need to add a test for the bug fix of STORM-3039. We try to rebind to port > 6700 a few times and expect it to be usable quickly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3136) Fix flaky integration test, and make it more readable
[ https://issues.apache.org/jira/browse/STORM-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3136. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~Srdo], I merged into master. > Fix flaky integration test, and make it more readable > - > > Key: STORM-3136 > URL: https://issues.apache.org/jira/browse/STORM-3136 > Project: Apache Storm > Issue Type: Bug > Components: integration-test >Affects Versions: 2.0.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 50m > Remaining Estimate: 0h > > The integration test appears flaky, e.g. > https://travis-ci.org/apache/storm/jobs/397471420. > We should try to fix this. I also find the integration test code hard to read > and understand, so I'd like to do some cleanup. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3143) Unnecessary inclusion of empty match result in Json
[
https://issues.apache.org/jira/browse/STORM-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-3143.
-
Resolution: Fixed
Thanks [~zhengdai], I merged into master.
> Unnecessary inclusion of empty match result in Json
> ---
>
> Key: STORM-3143
> URL: https://issues.apache.org/jira/browse/STORM-3143
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-webapp
>Affects Versions: 2.0.0
>Reporter: Zhengdai Hu
>Assignee: Zhengdai Hu
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> `FindNMatches()` didn't correctly filter out empty match result in
> `substringSearch()` and hence send back an empty map to user. I don't know if
> this the desired behavior but a fix to current behavior will make metrics for
> logviewer easier to implement.
> An example of current behavior:
> {code:json}
> {
> "fileOffset": 1,
> "searchString": "sdf",
> "matches": [
> {
> "searchString": "sdf",
> "fileName": "word-count-1-1530815972/6701/worker.log",
> "matches": [],
> "port": "6701",
> "isDaemon": "no",
> "startByteOffset": 0
> }
> ]
> }
> {code}
> Desired behavior:
> {code:json}
> {
> "fileOffset": 1,
> "searchString": "sdf",
> "matches": []
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-3028) HdfsSpout does not handle empty files in case of ack enabled
[ https://issues.apache.org/jira/browse/STORM-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3028. - Resolution: Fixed Fix Version/s: 1.2.3 2.0.0 Thanks [~ghajos], I merged into master and 1.x-branch. > HdfsSpout does not handle empty files in case of ack enabled > > > Key: STORM-3028 > URL: https://issues.apache.org/jira/browse/STORM-3028 > Project: Apache Storm > Issue Type: Bug > Components: storm-hdfs >Affects Versions: 1.2.1 >Reporter: Gergely Hajós >Assignee: Gergely Hajós >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0, 1.2.3 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > When ackers are present and an empty file is read, HdfsSpout waits for > acknowledgement to close the file. As nothing was emitted no ack arrives so > HdfsSpout stays in "fileReadCompletely" state forever. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3127) Avoid potential race condition
[ https://issues.apache.org/jira/browse/STORM-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3127. - Resolution: Fixed Thanks [~zhengdai], I merged into master. > Avoid potential race condition > --- > > Key: STORM-3127 > URL: https://issues.apache.org/jira/browse/STORM-3127 > Project: Apache Storm > Issue Type: Bug > Components: storm-server >Affects Versions: 2.0.0 >Reporter: Zhengdai Hu >Assignee: Zhengdai Hu >Priority: Minor > Fix For: 2.0.0 > > > PortAndAssignment and its call back is added after update to a blob is > invoked asynchronously. It is not guaranteed that the new dependent worker > will be registered before blob informs its update to listening workers. > This can be fixed by moving addReference call up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3126) Avoid unnecessary force kill when invoking storm kill_workers
[ https://issues.apache.org/jira/browse/STORM-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3126. - Resolution: Fixed Thanks [~zhengdai], I merged into master. > Avoid unnecessary force kill when invoking storm kill_workers > - > > Key: STORM-3126 > URL: https://issues.apache.org/jira/browse/STORM-3126 > Project: Apache Storm > Issue Type: Bug > Components: storm-server >Affects Versions: 2.0.0 >Reporter: Zhengdai Hu >Assignee: Zhengdai Hu >Priority: Minor > Fix For: 2.0.0 > > > Supervisor tries to force kill a worker before checking if it has died, > leading to unnecessary force kill call. This is minor but does help clean up > logs a little bit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3125) Refactoring methods in components for Supervisor and DRPC
[ https://issues.apache.org/jira/browse/STORM-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3125. - Resolution: Fixed Thanks [~zhengdai], I merged into master. > Refactoring methods in components for Supervisor and DRPC > - > > Key: STORM-3125 > URL: https://issues.apache.org/jira/browse/STORM-3125 > Project: Apache Storm > Issue Type: Improvement > Components: storm-server >Affects Versions: 2.0.0 >Reporter: Zhengdai Hu >Assignee: Zhengdai Hu >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > This is a supplement issue page to STORM-3099, separating out refactoring > work from metrics addition. > A few misc bug discovered during refactoring have been incorporate in this > issue as well. See links for more information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3046) Getting a NPE leading worker to die when starting a topology.
[
https://issues.apache.org/jira/browse/STORM-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-3046.
-
Resolution: Fixed
Fix Version/s: 2.0.0
Thanks [~Srdo], I merged into master.
The patch wasn't applied to 1.x-branch cleanly. Could you please raise a PR for
1.x-branch as well? Thanks in advance!
> Getting a NPE leading worker to die when starting a topology.
> -
>
> Key: STORM-3046
> URL: https://issues.apache.org/jira/browse/STORM-3046
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client, trident
>Affects Versions: 1.2.1
>Reporter: Kush Khandelwal
>Assignee: Stig Rohde Døssing
>Priority: Blocker
> Labels: kafka, pull-request-available, storm-kafka-client,
> trident
> Fix For: 2.0.0
>
> Attachments: TestTopology.java
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I am using storm-core and storm-kafka-client version 1.2.1 and kafka clients
> version 1.1.0.
> We have an external kafka from where we get the messages.
> Whenever I try to run the topology, I get a NPE, which leads to the worker
> getting died.
> If I set poll strategy to earliest and the topic already contains some
> messages, it works fine.
> I have used a custom record translator which is working fine.
> Can someone please help me fix the issue?
> Thanks.
>
> Error -
> 10665 [Thread-58-spout-handle-rule-local-kafka-spout-executor[26 26]] ERROR
> o.a.s.util - Async loop died!
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:522)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:487)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:74)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$fn__5043$fn__5056$fn__5109.invoke(executor.clj:861)
> ~[storm-core-1.2.1.jar:1.2.1]
> at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484)
> [storm-core-1.2.1.jar:1.2.1]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
> Caused by: java.lang.NullPointerException
> at
> org.apache.storm.kafka.spout.trident.KafkaTridentSpoutEmitter.seek(KafkaTridentSpoutEmitter.java:193)
> ~[storm-kafka-client-1.2.1.jar:1.2.1]
> at
> org.apache.storm.kafka.spout.trident.KafkaTridentSpoutEmitter.emitPartitionBatch(KafkaTridentSpoutEmitter.java:127)
> ~[storm-kafka-client-1.2.1.jar:1.2.1]
> at
> org.apache.storm.kafka.spout.trident.KafkaTridentSpoutEmitter.emitPartitionBatch(KafkaTridentSpoutEmitter.java:51)
> ~[storm-kafka-client-1.2.1.jar:1.2.1]
> at
> org.apache.storm.trident.spout.OpaquePartitionedTridentSpoutExecutor$Emitter.emitBatch(OpaquePartitionedTridentSpoutExecutor.java:141)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.trident.spout.TridentSpoutExecutor.execute(TridentSpoutExecutor.java:82)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.trident.topology.TridentBoltExecutor.execute(TridentBoltExecutor.java:383)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$fn__5043$tuple_action_fn__5045.invoke(executor.clj:739)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__4964.invoke(executor.clj:468)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4475.onEvent(disruptor.clj:41)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:509)
> ~[storm-core-1.2.1.jar:1.2.1]
> ... 6 more
>
>
> Topology class -
>
>
>
>
> import org.apache.storm.Config;
> import org.apache.storm.LocalCluster;
> import org.apache.storm.StormSubmitter;
> import org.apache.storm.generated.*;
> import org.apache.storm.kafka.spout.KafkaSpoutConfig;
> import org.apache.storm.kafka.spout.trident.KafkaTridentSpoutOpaque;
> import org.apache.storm.trident.Stream;
> import org.apache.storm.trident.TridentState;
> import org.apache.storm.trident.TridentTopology;
> import org.apache.storm.tuple.Fields;
> import java.util.Properties;
>
> public class TestTopology {
>
> private static StormTopology buildTopology(Properties stormProperties) {
>
> Properties kafkaProperties = getProperties("/kafka.properties");
> TridentTopology topology = new TridentTopology();
> Fields stageArguments = new Fields("test", "issue");
> KafkaSpoutConfig kafkaSpoutConfig =
> KafkaSpoutConfig.builder(kafkaProperties.getProperty("bootstrap.servers"),
> "test")
>
[jira] [Resolved] (STORM-3047) Ensure Trident emitter refreshPartitions is only called with partitions assigned to the emitter
[ https://issues.apache.org/jira/browse/STORM-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3047. - Resolution: Fixed Fix Version/s: 1.1.4 1.2.3 Thanks [~Srdo], I merged into 1.x-branch and 1.1.x-branch respectively. > Ensure Trident emitter refreshPartitions is only called with partitions > assigned to the emitter > --- > > Key: STORM-3047 > URL: https://issues.apache.org/jira/browse/STORM-3047 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.1.2, 1.2.1 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Major > Labels: pull-request-available > Fix For: 1.2.3, 1.1.4 > > Time Spent: 20m > Remaining Estimate: 0h > > This is a backport of the changes made to > OpaquePartitionedTridentSpoutExecutor in > https://github.com/apache/storm/pull/2300/files. > The description of the issue is copied here for convenience: > The changes in https://github.com/apache/storm/pull/2009 released in 1.1.0 > made some changes to the OpaquePartitionedTridentSpoutExecutor that likely > broke IOpaquePartitionedTridentSpout implementations other than > storm-kafka-client. The changed code used to request sorted partitions from > the spout via getOrderedPartitions, do a round-robin partitioning, and assign > partitions via refreshPartitions > https://github.com/apache/storm/blob/v1.0.4/storm-core/src/jvm/org/apache/storm/trident/spout/OpaquePartitionedTridentSpoutExecutor.java#L100. > The new code just passes the output of getOrderedPartitions into > refreshPartitions > https://github.com/apache/storm/blob/v1.1.0/storm-core/src/jvm/org/apache/storm/trident/spout/OpaquePartitionedTridentSpoutExecutor.java#L120. > It looks to me like refreshPartitions is passed the list of all partitions > assigned to any spout task, rather than just the partitions assigned to the > current task. > The proposed fix will use getOrderedPartitions to get the sorted partitions > list, pass the list into getPartitionsForTask, and pass the resulting list of > assigned partitions back into refreshPartitions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3013) Deactivated topology restarts if data flows into Kafka
[
https://issues.apache.org/jira/browse/STORM-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-3013.
-
Resolution: Fixed
Fix Version/s: 2.0.0
Thanks [~Srdo], I merged into master.
The patch wasn't applied to 1.x-branch cleanly. Could you please raise a PR for
1.x-branch as well? Thanks in advance!
> Deactivated topology restarts if data flows into Kafka
> --
>
> Key: STORM-3013
> URL: https://issues.apache.org/jira/browse/STORM-3013
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
>Affects Versions: 2.0.0, 1.2.1
>Reporter: Ajeesh B
>Assignee: Stig Rohde Døssing
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Hi, I have deactivated the storm topology & then if I produce any records
> into Kafka, Storm throws an exception. Exception follows,
> {code:java}
> 2018-03-28 09:50:23.804 o.a.s.d.executor Thread-83-kafkaLogs-executor[130
> 130] [INFO] Deactivating spout kafkaLogs:(130)
> 2018-03-28 09:51:01.289 o.a.s.util Thread-17-kafkaLogs-executor[139 139]
> [ERROR] Async loop died!
> java.lang.RuntimeException: java.lang.IllegalStateException: This consumer
> has already been closed.
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:522)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:487)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:477)
> ~[storm-core-1.2.1.jar:1.2.1]
> at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:70)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$fn__4975$fn__4990$fn__5021.invoke(executor.clj:634)
> ~[storm-core-1.2.1.jar:1.2.1]
> at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484)
> [storm-core-1.2.1.jar:1.2.1]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
> Caused by: java.lang.IllegalStateException: This consumer has already been
> closed.
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.acquireAndEnsureOpen(KafkaConsumer.java:1787)
> ~[stormjar.jar:?]
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.beginningOffsets(KafkaConsumer.java:1622)
> ~[stormjar.jar:?]
> at
> org.apache.storm.kafka.spout.metrics.KafkaOffsetMetric.getValueAndReset(KafkaOffsetMetric.java:79)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$metrics_tick$fn__4899.invoke(executor.clj:345)
> ~[storm-core-1.2.1.jar:1.2.1]
> at clojure.core$map$fn__4553.invoke(core.clj:2622) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
> at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?]
> at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?]
> at clojure.core$filter$fn__4580.invoke(core.clj:2679) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
> at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.7.0.jar:?]
> at clojure.lang.RT.next(RT.java:674) ~[clojure-1.7.0.jar:?]
> at clojure.core$next__4112.invoke(core.clj:64) ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6523.invoke(protocols.clj:170)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6478$G__6473__6487.invoke(protocols.clj:19)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6506.invoke(protocols.clj:101)
> ~[clojure-1.7.0.jar:?]
> at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13)
> ~[clojure-1.7.0.jar:?]
> at clojure.core$reduce.invoke(core.clj:6519) ~[clojure-1.7.0.jar:?]
> at clojure.core$into.invoke(core.clj:6600) ~[clojure-1.7.0.jar:?]
> at org.apache.storm.daemon.executor$metrics_tick.invoke(executor.clj:349)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$fn__4975$tuple_action_fn__4981.invoke(executor.clj:522)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__4964.invoke(executor.clj:471)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4475.onEvent(disruptor.clj:41)
> ~[storm-core-1.2.1.jar:1.2.1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:509)
> ~[storm-core-1.2.1.jar:1.2.1]
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-3121) Fix flaky metrics tests in storm-core
[
https://issues.apache.org/jira/browse/STORM-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-3121.
-
Resolution: Fixed
Fix Version/s: 2.0.0
Thanks [~Srdo], I merged into master.
> Fix flaky metrics tests in storm-core
> -
>
> Key: STORM-3121
> URL: https://issues.apache.org/jira/browse/STORM-3121
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> The tests are flaky, but only rarely fail. I've only seen them fail on Travis
> when Travis is under load.
> Example failures:
> {code}
> classname: org.apache.storm.metrics-test / testname:
> test-custom-metric-with-multi-tasks
> expected: (clojure.core/= [1 0 0 0 0 0 2] (clojure.core/subvec
> (org.apache.storm.metrics-test/lookup-bucket-by-comp-id-&-metric-name! "2"
> "my-custom-metric") 0 N__3207__auto__))
> actual: (not (clojure.core/= [1 0 0 0 0 0 2] [1 0 0 0 0 0 0]))
> at: test_runner.clj:105
> {code}
> {code}
> classname: org.apache.storm.metrics-test / testname: test-builtin-metrics-2
> expected: (clojure.core/= [1 1] (clojure.core/subvec
> (org.apache.storm.metrics-test/lookup-bucket-by-comp-id-&-metric-name!
> "myspout" "__emit-count/default") 0 N__3207__auto__))
> actual: (not (clojure.core/= [1 1] [1 0]))
> at: test_runner.clj:105
> {code}
> The problem is that the tests increment metrics counters in the executor
> async loops, then expect the counters to end up in exact metrics buckets. The
> creation of a bucket is triggered by the metrics timer. The timer is included
> in time simulation and LocalCluster.waitForIdle, but the executor async loop
> isn't. There isn't any guarantee that the executor async loop gets to run
> when the test does a sequence like
> {code}
> Time.advanceClusterTime
> cluster.waitForIdle
> {code}
> because the waitForIdle check doesn't know about the executor async loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-3122) FNFE due to race condition between "async localizer" and "update blob" timer thread
[ https://issues.apache.org/jira/browse/STORM-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-3122. - Resolution: Fixed Fix Version/s: 1.2.3 Merged into 1.x-branch. > FNFE due to race condition between "async localizer" and "update blob" timer > thread > --- > > Key: STORM-3122 > URL: https://issues.apache.org/jira/browse/STORM-3122 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.x >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Critical > Labels: pull-request-available > Fix For: 1.2.3 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > There's race condition between "async localizer" and "update blob" timer > thread. > When worker is shutting down, reference count for blob will be 0 and > supervisor will remove actual blob file. There's also "update blob" timer > thread which tries to keep blobs updated for downloaded topologies. While > updating topology it should read some of blob files already downloaded > assuming these files should be downloaded before, and the assumption is > broken because of async localizer. > [~arunmahadevan] suggested an approach to fix this: "updateBlobsForTopology" > can just catch the FIleNotFoundException and skip updating the blobs in case > it can't find the stormconf, and the approach looks simplest fix so I'll > provide a patch based on suggestion. > Btw, it doesn't apply to master branch, since in master branch all blobs are > synced up separately (no need to read stormconf to enumerate topology related > blobs), and update logic is already fault-tolerance (skip to next sync when > it can't pull the blob). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (STORM-2915) How could I to get the fail Number in Bolt When I use Kafka Spout
[
https://issues.apache.org/jira/browse/STORM-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim closed STORM-2915.
---
Resolution: Invalid
The issue looked like a question which is better to be posted to user mailing
list.
> How could I to get the fail Number in Bolt When I use Kafka Spout
>
>
> Key: STORM-2915
> URL: https://issues.apache.org/jira/browse/STORM-2915
> Project: Apache Storm
> Issue Type: New Feature
> Components: storm-kafka-client
>Affects Versions: 1.0.2, 1.1.0, 1.0.3, 1.0.4, 1.1.1, 1.0.5
>Reporter: Gergo Hong
>Priority: Minor
>
> I want to get fail num in bolt , how could I to get it?
> if fail it retry, I see This
> if (!isScheduled || retryService.isReady(msgId)) {
> final String stream = tuple instanceof KafkaTuple ? ((KafkaTuple)
> tuple).getStream() : Utils.DEFAULT_STREAM_ID;
> if (!isAtLeastOnceProcessing()) {
> if (kafkaSpoutConfig.isTupleTrackingEnforced()) {
> collector.emit(stream, tuple, msgId);
> LOG.trace("Emitted tuple [{}] for record [{}] with msgId [{}]", tuple,
> record, msgId);
> } else {
> collector.emit(stream, tuple);
> LOG.trace("Emitted tuple [{}] for record [{}]", tuple, record);
> }
> } else {
> emitted.add(msgId);
> offsetManagers.get(tp).addToEmitMsgs(msgId.offset());
> if (isScheduled) { // Was scheduled for retry and re-emitted, so remove from
> schedule.
> retryService.remove(msgId);
> }
> collector.emit(stream, tuple, msgId);
> tupleListener.onEmit(tuple, msgId);
> LOG.trace("Emitted tuple [{}] for record [{}] with msgId [{}]", tuple,
> record, msgId);
> }
> return true;
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (STORM-2945) Nail down and document how to support background emits in Spouts and Bolts
[ https://issues.apache.org/jira/browse/STORM-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2945: Issue Type: Documentation (was: Bug) > Nail down and document how to support background emits in Spouts and Bolts > -- > > Key: STORM-2945 > URL: https://issues.apache.org/jira/browse/STORM-2945 > Project: Apache Storm > Issue Type: Documentation >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-2963) Updates to Performance.md
[ https://issues.apache.org/jira/browse/STORM-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2963: Issue Type: Documentation (was: Improvement) > Updates to Performance.md > -- > > Key: STORM-2963 > URL: https://issues.apache.org/jira/browse/STORM-2963 > Project: Apache Storm > Issue Type: Documentation >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Roshan Naik >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-2963) Updates to Performance.md
[ https://issues.apache.org/jira/browse/STORM-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2963: Fix Version/s: (was: 2.0.0) > Updates to Performance.md > -- > > Key: STORM-2963 > URL: https://issues.apache.org/jira/browse/STORM-2963 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Roshan Naik >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-3061) Upgrade Dependencies before 2.x release
[ https://issues.apache.org/jira/browse/STORM-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-3061: Affects Version/s: 2.0.0 > Upgrade Dependencies before 2.x release > --- > > Key: STORM-3061 > URL: https://issues.apache.org/jira/browse/STORM-3061 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Major > Labels: pull-request-available > Time Spent: 9h 20m > Remaining Estimate: 0h > > Storm has a lot of dependencies. It would be great to upgrade many of them > to newer versions ahead of a 2.x release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-3081) Storm kafka client not consuming messages properly
[
https://issues.apache.org/jira/browse/STORM-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-3081:
Component/s: (was: storm-kafka)
> Storm kafka client not consuming messages properly
> --
>
> Key: STORM-3081
> URL: https://issues.apache.org/jira/browse/STORM-3081
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
>Affects Versions: 1.2.1
>Reporter: Kush Khandelwal
>Priority: Blocker
>
> A single thread is pushing some messages to kafka serially and we consume it
> at the storm's end using storm-kafka-client.
> After a few requests, the consumer is not able to consume from the queue
> blocking the thread which waits for the response from storm's end.
> We added one more consumer with a different consumer group and there the
> messages are getting read properly.
> So we know there is some problem at the storm kafka client consumer's end.
> The kafka spout config is written like this -
> KafkaSpoutConfig kafkaSpoutConfig =
> KafkaSpoutConfig.builder(kafkaProperties.getProperty("bootstrap.servers"),
> stormProperties.getProperty("TOPIC"))
>
> .setFirstPollOffsetStrategy(KafkaSpoutConfig.FirstPollOffsetStrategy.LATEST)
> .setRecordTranslator(new MessageDeserializer(), arguments)
> .build();
> I can't seem to figure out the issue.
> Can someone please help me out?
> Thanks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (STORM-3110) Supervisor does not kill all worker processes in secure mode in case of user mismatch
[
https://issues.apache.org/jira/browse/STORM-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-3110:
Issue Type: Bug (was: Improvement)
> Supervisor does not kill all worker processes in secure mode in case of user
> mismatch
> -
>
> Key: STORM-3110
> URL: https://issues.apache.org/jira/browse/STORM-3110
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0, 1.2.3
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> While running in secure mode, supervisor sets the worker user (in workers
> local state) as the user that launched the topology.
>
> {code:java}
> SET worker-user 4d67a6be-4c80-4622-96af-f94706d58553 foo
> {code}
>
> However the worker OS process does not actually run as the user "foo"
> (instead runs as storm user) unless {{supervisor.run.worker.as.user}} is also
> set.
> If the supervisor's assignment changes, the supervisor in some cases checks
> if all processes are dead by matching the "pid+user". Here if the worker is
> running as a different user (say storm) the supervisor wrongly assumes that
> the worker process is dead.
> Later when supervisor tries to launch a worker at that same port, it throws a
> bind exception
> o.a.s.m.n.Server main [INFO] Create Netty Server Netty-server-localhost-6700,
> buffer_size: 5242880, maxWorkers: 1
> o.a.s.d.worker main [ERROR] Error on initialization of server mk-worker
> org.apache.storm.shade.org.jboss.netty.channel.ChannelException: Failed to
> bind to: 0.0.0.0/0.0.0.0:6700
> at
> org.apache.storm.shade.org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
> ~[storm-core-1.2.0.3.1.0.0-501.jar:1.2.0.3.1.0.0-501]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-3110) Supervisor does not kill all worker processes in secure mode in case of user mismatch
[
https://issues.apache.org/jira/browse/STORM-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-3110.
-
Resolution: Fixed
Fix Version/s: 1.2.3
2.0.0
Thanks [~arunmahadevan], I merged into master and 1.x-branch.
For master branch I squashed commits and pushed. The commit is
https://github.com/apache/storm/commit/82deb62dd3a6f58e5569244517bc93647fcab2a2
> Supervisor does not kill all worker processes in secure mode in case of user
> mismatch
> -
>
> Key: STORM-3110
> URL: https://issues.apache.org/jira/browse/STORM-3110
> Project: Apache Storm
> Issue Type: Improvement
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0, 1.2.3
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> While running in secure mode, supervisor sets the worker user (in workers
> local state) as the user that launched the topology.
>
> {code:java}
> SET worker-user 4d67a6be-4c80-4622-96af-f94706d58553 foo
> {code}
>
> However the worker OS process does not actually run as the user "foo"
> (instead runs as storm user) unless {{supervisor.run.worker.as.user}} is also
> set.
> If the supervisor's assignment changes, the supervisor in some cases checks
> if all processes are dead by matching the "pid+user". Here if the worker is
> running as a different user (say storm) the supervisor wrongly assumes that
> the worker process is dead.
> Later when supervisor tries to launch a worker at that same port, it throws a
> bind exception
> o.a.s.m.n.Server main [INFO] Create Netty Server Netty-server-localhost-6700,
> buffer_size: 5242880, maxWorkers: 1
> o.a.s.d.worker main [ERROR] Error on initialization of server mk-worker
> org.apache.storm.shade.org.jboss.netty.channel.ChannelException: Failed to
> bind to: 0.0.0.0/0.0.0.0:6700
> at
> org.apache.storm.shade.org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
> ~[storm-core-1.2.0.3.1.0.0-501.jar:1.2.0.3.1.0.0-501]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (STORM-3122) FNFE due to race condition between "async localizer" and "update blob" timer thread
[ https://issues.apache.org/jira/browse/STORM-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-3122: Priority: Critical (was: Major) > FNFE due to race condition between "async localizer" and "update blob" timer > thread > --- > > Key: STORM-3122 > URL: https://issues.apache.org/jira/browse/STORM-3122 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.x >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Critical > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > There's race condition between "async localizer" and "update blob" timer > thread. > When worker is shutting down, reference count for blob will be 0 and > supervisor will remove actual blob file. There's also "update blob" timer > thread which tries to keep blobs updated for downloaded topologies. While > updating topology it should read some of blob files already downloaded > assuming these files should be downloaded before, and the assumption is > broken because of async localizer. > [~arunmahadevan] suggested an approach to fix this: "updateBlobsForTopology" > can just catch the FIleNotFoundException and skip updating the blobs in case > it can't find the stormconf, and the approach looks simplest fix so I'll > provide a patch based on suggestion. > Btw, it doesn't apply to master branch, since in master branch all blobs are > synced up separately (no need to read stormconf to enumerate topology related > blobs), and update logic is already fault-tolerance (skip to next sync when > it can't pull the blob). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (STORM-3122) FNFE due to race condition between "async localizer" and "update blob" timer thread
Jungtaek Lim created STORM-3122: --- Summary: FNFE due to race condition between "async localizer" and "update blob" timer thread Key: STORM-3122 URL: https://issues.apache.org/jira/browse/STORM-3122 Project: Apache Storm Issue Type: Bug Components: storm-core Affects Versions: 1.x Reporter: Jungtaek Lim Assignee: Jungtaek Lim There's race condition between "async localizer" and "update blob" timer thread. When worker is shutting down, reference count for blob will be 0 and supervisor will remove actual blob file. There's also "update blob" timer thread which tries to keep blobs updated for downloaded topologies. While updating topology it should read some of blob files already downloaded assuming these files should be downloaded before, and the assumption is broken because of async localizer. [~arunmahadevan] suggested an approach to fix this: "updateBlobsForTopology" can just catch the FIleNotFoundException and skip updating the blobs in case it can't find the stormconf, and the approach looks simplest fix so I'll provide a patch based on suggestion. Btw, it doesn't apply to master branch, since in master branch all blobs are synced up separately (no need to read stormconf to enumerate topology related blobs), and update logic is already fault-tolerance (skip to next sync when it can't pull the blob). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
