[jira] [Closed] (STORM-2417) Storm kafka monitor throws IllegalArgumentException if there is no data in kafka topic
[ https://issues.apache.org/jira/browse/STORM-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyank Shah closed STORM-2417. --- Resolution: Won't Fix > Storm kafka monitor throws IllegalArgumentException if there is no data in > kafka topic > -- > > Key: STORM-2417 > URL: https://issues.apache.org/jira/browse/STORM-2417 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-monitor >Reporter: Priyank Shah >Assignee: Priyank Shah > > If a topology has a kafka spout in it and if kafka topic does not have any > data or topology has not committed offsets to zk node then > storm-kafka-monitor throws an IllegalArgumentException and it shows up in UI. > While this is not breaking any functionality showing a better message can be > displayed -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (STORM-2379) still refers 1.6 elastic
[
https://issues.apache.org/jira/browse/STORM-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927406#comment-15927406
]
Sree Vaddi edited comment on STORM-2379 at 3/16/17 3:29 AM:
Last week, I was at elasticon and I had a chance to talk with James Baiera, who
heads the elastic-hadoop.
Details at,
https://discuss.elastic.co/t/storm-1-0-x-elastic-5-0-x-integration-issue/76337.
>From other talks that I attended and talking to elastic.co employees, my
>understanding is that elastic.co, itself is pushing to use java REST client
>more over the Java Client.
This falls in line with our mind set here on this topic.
And REST client supports multiple versions of elastic, too.
How about using dockers for integration testing against multiple versions of
elastic.
https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
java rest client:
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/index.html
may be use this, docker-maven-plugin: {not sure, how reliable this is}
https://dmp.fabric8.io/
was (Author: sreevaddi):
Last week, I was at elasticon and I had a chance to talk with James Baiera, who
heads the elastic-hadoop.
Details at,
https://discuss.elastic.co/t/storm-1-0-x-elastic-5-0-x-integration-issue/76337.
>From other talks that I attended and talking to elastic.co employees, my
>understanding is that elastic.co, itself is pushing to use java REST client
>more over the Java Client.
This falls in line with our mind set here on this topic.
And REST client supports multiple versions of elastic, too.
How about using dockers for integration testing against multiple versions of
elastic.
https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
java rest client:
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/index.html
> still refers 1.6 elastic
>
>
> Key: STORM-2379
> URL: https://issues.apache.org/jira/browse/STORM-2379
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-elasticsearch
>Affects Versions: 1.0.1, 1.0.2, 1.0.3
> Environment: storm 1.0.x
> elastic 5.0.0 and higher
>Reporter: Sree Vaddi
>Priority: Blocker
>
> following documentation:
> https://storm.apache.org/releases/1.0.1/storm-elasticsearch.html
> https://github.com/apache/storm/blob/master/external/storm-elasticsearch/pom.xml#L40
> this causes errors while writing to elastic 5.x
> {code:language=java}
> java.lang.NoClassDefFoundError: org/elasticsearch/common/base/Preconditions
> at
> org.apache.storm.elasticsearch.common.EsConfig.(EsConfig.java:62)
> ~[storm-elasticsearch-1.0.2.jar:1.0.2]
> at
> org.apache.storm.elasticsearch.common.EsConfig.(EsConfig.java:49)
> ~[storm-elasticsearch-1.0.2.jar:1.0.2]
> Caused by: java.lang.ClassNotFoundException:
> org.elasticsearch.common.base.Preconditions
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> ~[?:1.8.0_112]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_112]
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> ~[?:1.8.0_112]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_112]
> 261538 [elasticsearch[Ringleader][generic][T#2]] INFO o.e.c.transport -
> [Ringleader] failed to get node info for
> [#transport#-1][svaddi][inet[localhost/127.0.0.1:9200]], disconnecting...
> org.elasticsearch.transport.ReceiveTimeoutTransportException:
> [][inet[localhost/127.0.0.1:9200]][cluster:monitor/nodes/info] request_id
> [26] timed out after [5005ms]
> at
> org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:529)
> ~[elasticsearch-1.6.0.jar:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_112]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
> elastic logs:
> {code:language=java}
> [2017-02-23T15:47:04,487][WARN ][o.e.t.n.Netty4Transport ] [Qt9qlNV]
> exception caught on transport layer [[id: 0x8f15e875, L:/127.0.0.1:9300 -
> R:/127.0.0.1:52031]], closing connection
> java.lang.IllegalStateException: Received message from unsupported version:
> [1.0.0] minimal compatible version is: [5.0.0]
> at
> org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1199)
> ~[elasticsearch-5.0.0.jar:5.0.0]
> at
> org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
> ~[transport-netty4-5.0.0.jar:5.0.0]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
>
[jira] [Comment Edited] (STORM-2379) still refers 1.6 elastic
[
https://issues.apache.org/jira/browse/STORM-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927406#comment-15927406
]
Sree Vaddi edited comment on STORM-2379 at 3/16/17 3:26 AM:
Last week, I was at elasticon and I had a chance to talk with James Baiera, who
heads the elastic-hadoop.
Details at,
https://discuss.elastic.co/t/storm-1-0-x-elastic-5-0-x-integration-issue/76337.
>From other talks that I attended and talking to elastic.co employees, my
>understanding is that elastic.co, itself is pushing to use java REST client
>more over the Java Client.
This falls in line with our mind set here on this topic.
And REST client supports multiple versions of elastic, too.
How about using dockers for integration testing against multiple versions of
elastic.
https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
java rest client:
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/index.html
was (Author: sreevaddi):
Last week, I was at elasticon and I had a chance to talk with James Baiera, who
heads the elastic-hadoop.
Details at,
https://discuss.elastic.co/t/storm-1-0-x-elastic-5-0-x-integration-issue/76337.
>From other talks that I attended and talking to elastic.co employees, my
>understanding is that elastic.co, itself is pushing to use java REST client
>more over the Java Client.
This falls in line with our mind set here on this topic.
And REST support multiple versions of elastic, too.
How about using dockers for integration testing against multiple versions of
elastic.
https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
java rest client:
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/index.html
> still refers 1.6 elastic
>
>
> Key: STORM-2379
> URL: https://issues.apache.org/jira/browse/STORM-2379
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-elasticsearch
>Affects Versions: 1.0.1, 1.0.2, 1.0.3
> Environment: storm 1.0.x
> elastic 5.0.0 and higher
>Reporter: Sree Vaddi
>Priority: Blocker
>
> following documentation:
> https://storm.apache.org/releases/1.0.1/storm-elasticsearch.html
> https://github.com/apache/storm/blob/master/external/storm-elasticsearch/pom.xml#L40
> this causes errors while writing to elastic 5.x
> {code:language=java}
> java.lang.NoClassDefFoundError: org/elasticsearch/common/base/Preconditions
> at
> org.apache.storm.elasticsearch.common.EsConfig.(EsConfig.java:62)
> ~[storm-elasticsearch-1.0.2.jar:1.0.2]
> at
> org.apache.storm.elasticsearch.common.EsConfig.(EsConfig.java:49)
> ~[storm-elasticsearch-1.0.2.jar:1.0.2]
> Caused by: java.lang.ClassNotFoundException:
> org.elasticsearch.common.base.Preconditions
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> ~[?:1.8.0_112]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_112]
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> ~[?:1.8.0_112]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_112]
> 261538 [elasticsearch[Ringleader][generic][T#2]] INFO o.e.c.transport -
> [Ringleader] failed to get node info for
> [#transport#-1][svaddi][inet[localhost/127.0.0.1:9200]], disconnecting...
> org.elasticsearch.transport.ReceiveTimeoutTransportException:
> [][inet[localhost/127.0.0.1:9200]][cluster:monitor/nodes/info] request_id
> [26] timed out after [5005ms]
> at
> org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:529)
> ~[elasticsearch-1.6.0.jar:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_112]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
> elastic logs:
> {code:language=java}
> [2017-02-23T15:47:04,487][WARN ][o.e.t.n.Netty4Transport ] [Qt9qlNV]
> exception caught on transport layer [[id: 0x8f15e875, L:/127.0.0.1:9300 -
> R:/127.0.0.1:52031]], closing connection
> java.lang.IllegalStateException: Received message from unsupported version:
> [1.0.0] minimal compatible version is: [5.0.0]
> at
> org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1199)
> ~[elasticsearch-5.0.0.jar:5.0.0]
> at
> org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
> ~[transport-netty4-5.0.0.jar:5.0.0]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
> [netty-transport-4.1.5.Final.jar:4.1.5.Final]
> at
>
[jira] [Commented] (STORM-2379) still refers 1.6 elastic
[
https://issues.apache.org/jira/browse/STORM-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927406#comment-15927406
]
Sree Vaddi commented on STORM-2379:
---
Last week, I was at elasticon and I had a chance to talk with James Baiera, who
heads the elastic-hadoop.
Details at,
https://discuss.elastic.co/t/storm-1-0-x-elastic-5-0-x-integration-issue/76337.
>From other talks that I attended and talking to elastic.co employees, my
>understanding is that elastic.co, itself is pushing to use java REST client
>more over the Java Client.
This falls in line with our mind set here on this topic.
And REST support multiple versions of elastic, too.
How about using dockers for integration testing against multiple versions of
elastic.
https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
java rest client:
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/index.html
> still refers 1.6 elastic
>
>
> Key: STORM-2379
> URL: https://issues.apache.org/jira/browse/STORM-2379
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-elasticsearch
>Affects Versions: 1.0.1, 1.0.2, 1.0.3
> Environment: storm 1.0.x
> elastic 5.0.0 and higher
>Reporter: Sree Vaddi
>Priority: Blocker
>
> following documentation:
> https://storm.apache.org/releases/1.0.1/storm-elasticsearch.html
> https://github.com/apache/storm/blob/master/external/storm-elasticsearch/pom.xml#L40
> this causes errors while writing to elastic 5.x
> {code:language=java}
> java.lang.NoClassDefFoundError: org/elasticsearch/common/base/Preconditions
> at
> org.apache.storm.elasticsearch.common.EsConfig.(EsConfig.java:62)
> ~[storm-elasticsearch-1.0.2.jar:1.0.2]
> at
> org.apache.storm.elasticsearch.common.EsConfig.(EsConfig.java:49)
> ~[storm-elasticsearch-1.0.2.jar:1.0.2]
> Caused by: java.lang.ClassNotFoundException:
> org.elasticsearch.common.base.Preconditions
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> ~[?:1.8.0_112]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_112]
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> ~[?:1.8.0_112]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_112]
> 261538 [elasticsearch[Ringleader][generic][T#2]] INFO o.e.c.transport -
> [Ringleader] failed to get node info for
> [#transport#-1][svaddi][inet[localhost/127.0.0.1:9200]], disconnecting...
> org.elasticsearch.transport.ReceiveTimeoutTransportException:
> [][inet[localhost/127.0.0.1:9200]][cluster:monitor/nodes/info] request_id
> [26] timed out after [5005ms]
> at
> org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:529)
> ~[elasticsearch-1.6.0.jar:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_112]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {code}
> elastic logs:
> {code:language=java}
> [2017-02-23T15:47:04,487][WARN ][o.e.t.n.Netty4Transport ] [Qt9qlNV]
> exception caught on transport layer [[id: 0x8f15e875, L:/127.0.0.1:9300 -
> R:/127.0.0.1:52031]], closing connection
> java.lang.IllegalStateException: Received message from unsupported version:
> [1.0.0] minimal compatible version is: [5.0.0]
> at
> org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1199)
> ~[elasticsearch-5.0.0.jar:5.0.0]
> at
> org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
> ~[transport-netty4-5.0.0.jar:5.0.0]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
> [netty-transport-4.1.5.Final.jar:4.1.5.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
> [netty-transport-4.1.5.Final.jar:4.1.5.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350)
> [netty-transport-4.1.5.Final.jar:4.1.5.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
> [netty-codec-4.1.5.Final.jar:4.1.5.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280)
> [netty-codec-4.1.5.Final.jar:4.1.5.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396)
> [netty-codec-4.1.5.Final.jar:4.1.5.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
> [netty-codec-4.1.5.Final.jar:4.1.5.Final]
> at
[jira] [Updated] (STORM-2417) Storm kafka monitor throws IllegalArgumentException if there is no data in kafka topic
[ https://issues.apache.org/jira/browse/STORM-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyank Shah updated STORM-2417: Summary: Storm kafka monitor throws IllegalArgumentException if there is no data in kafka topic (was: Storm kafka monitor throws null pointer exception if there is no data in kafka topic) > Storm kafka monitor throws IllegalArgumentException if there is no data in > kafka topic > -- > > Key: STORM-2417 > URL: https://issues.apache.org/jira/browse/STORM-2417 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-monitor >Reporter: Priyank Shah >Assignee: Priyank Shah > > If a topology has a kafka spout in it and if kafka topic does not have any > data the storm-kafka-monitor throws a NPE and it shows up in UI. While this > is not breaking any functionality showing a NPE in ui is not a user friendly > experience. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (STORM-2417) Storm kafka monitor throws null pointer exception if there is no data in kafka topic
Priyank Shah created STORM-2417: --- Summary: Storm kafka monitor throws null pointer exception if there is no data in kafka topic Key: STORM-2417 URL: https://issues.apache.org/jira/browse/STORM-2417 Project: Apache Storm Issue Type: Bug Components: storm-kafka-monitor Reporter: Priyank Shah Assignee: Priyank Shah If a topology has a kafka spout in it and if kafka topic does not have any data the storm-kafka-monitor throws a NPE and it shows up in UI. While this is not breaking any functionality showing a NPE in ui is not a user friendly experience. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (STORM-2416) Release Packaging Improvements
P. Taylor Goetz created STORM-2416: -- Summary: Release Packaging Improvements Key: STORM-2416 URL: https://issues.apache.org/jira/browse/STORM-2416 Project: Apache Storm Issue Type: Improvement Components: build Reporter: P. Taylor Goetz Assignee: P. Taylor Goetz This issue is to address distribution packaging improvements discussed on the dev@ list: 1. Move remaining examples to "examples" directory. 2. Package examples as source-only, to be compiled by users 3. Remove connector jars from binary distribution (since they are available in Maven, and we want to discourage users from hand-crafting topology jars) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (STORM-2402) KafkaSpout sub-classes should be able to customize tuple processing
[
https://issues.apache.org/jira/browse/STORM-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Klessing closed STORM-2402.
--
Resolution: Invalid
As comments on the PR on github showed there is a cleaner solution already
available. This request can be safely discarged.
> KafkaSpout sub-classes should be able to customize tuple processing
> ---
>
> Key: STORM-2402
> URL: https://issues.apache.org/jira/browse/STORM-2402
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
>Affects Versions: 1.0.3
>Reporter: Daniel Klessing
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> We need a {{KafkaSpout}} that writes unprocessable records to a
> "dead-letter-topic". For this to function we sub-classed {{KafkaSpout}} and
> added the corresponding code. Without the incoming patch sub-classses can not
> have access to the actual tuples/records but just the {{KafkaSpoutMessageId}}
> in {{ack()}} and {{fail()}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (STORM-2290) Upgrading zookeeper to 3.4.9 for stability
[ https://issues.apache.org/jira/browse/STORM-2290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925853#comment-15925853 ] Anthony Milbourne edited comment on STORM-2290 at 3/15/17 10:27 AM: [~kabhwan] Thanks, I have created a new issue (STORM-2415) for the ZOOKEEPER-1506 related problems. was (Author: amilbourne): @Jungtaek Lim Thanks, I have created a new issue (STORM-2415) for the ZOOKEEPER-1506 related problems. > Upgrading zookeeper to 3.4.9 for stability > -- > > Key: STORM-2290 > URL: https://issues.apache.org/jira/browse/STORM-2290 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 1.0.2 >Reporter: Sachin Goyal > Time Spent: 20m > Remaining Estimate: 0h > > We should upgrade zookeeper to 3.4.9 as it brings in a lot of stability > improvements (http://zookeeper.apache.org/releases.html) and storm is still > using 3.4.6 (https://github.com/apache/storm/blob/master/pom.xml) > One serious issue affecting zookeeper 3.4.6 is > https://issues.apache.org/jira/browse/ZOOKEEPER-1506 which prohibits > zookeeper from getting a quorum and hence affects storm's stability as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (STORM-2290) Upgrading zookeeper to 3.4.9 for stability
[ https://issues.apache.org/jira/browse/STORM-2290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925853#comment-15925853 ] Anthony Milbourne edited comment on STORM-2290 at 3/15/17 10:26 AM: @Jungtaek Lim Thanks, I have created a new issue (STORM-2415) for the ZOOKEEPER-1506 related problems. was (Author: amilbourne): Jungtaek Lim Thanks, I have created a new issue (STORM-2415) for the ZOOKEEPER-1506 related problems. > Upgrading zookeeper to 3.4.9 for stability > -- > > Key: STORM-2290 > URL: https://issues.apache.org/jira/browse/STORM-2290 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 1.0.2 >Reporter: Sachin Goyal > Time Spent: 20m > Remaining Estimate: 0h > > We should upgrade zookeeper to 3.4.9 as it brings in a lot of stability > improvements (http://zookeeper.apache.org/releases.html) and storm is still > using 3.4.6 (https://github.com/apache/storm/blob/master/pom.xml) > One serious issue affecting zookeeper 3.4.6 is > https://issues.apache.org/jira/browse/ZOOKEEPER-1506 which prohibits > zookeeper from getting a quorum and hence affects storm's stability as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (STORM-2290) Upgrading zookeeper to 3.4.9 for stability
[ https://issues.apache.org/jira/browse/STORM-2290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925853#comment-15925853 ] Anthony Milbourne commented on STORM-2290: -- Thanks, I have created a new issue (STORM-2415) for the ZOOKEEPER-1506 related problems. > Upgrading zookeeper to 3.4.9 for stability > -- > > Key: STORM-2290 > URL: https://issues.apache.org/jira/browse/STORM-2290 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 1.0.2 >Reporter: Sachin Goyal > Time Spent: 20m > Remaining Estimate: 0h > > We should upgrade zookeeper to 3.4.9 as it brings in a lot of stability > improvements (http://zookeeper.apache.org/releases.html) and storm is still > using 3.4.6 (https://github.com/apache/storm/blob/master/pom.xml) > One serious issue affecting zookeeper 3.4.6 is > https://issues.apache.org/jira/browse/ZOOKEEPER-1506 which prohibits > zookeeper from getting a quorum and hence affects storm's stability as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (STORM-2415) Storm fails to properly handle Zookeeper hosts going down
[
https://issues.apache.org/jira/browse/STORM-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925850#comment-15925850
]
Anthony Milbourne edited comment on STORM-2415 at 3/15/17 10:22 AM:
This issue was created as a result of discussion on STORM-2290.
was (Author: amilbourne):
This issue was created as a result of discussion on the previous issue.
> Storm fails to properly handle Zookeeper hosts going down
> -
>
> Key: STORM-2415
> URL: https://issues.apache.org/jira/browse/STORM-2415
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.3
> Environment: All
>Reporter: Anthony Milbourne
>
> We run a storm cluster (v.1.0.3) on AWS and have 3 Zookeepers supporting it.
> Because AWS sometimes terminates VMs, we sometimes lose a Zookeeper instance.
> When this happens, the hostname cannot be resolved for that zookeeper
> instance as AWS has taken the VM away. We noticed that in this case storm
> fails to connect to zookeeper – even though there are still 2 Zookeeper
> instances running. It fails with an exception something like:
> {noformat}
> java.net.UnknownHostException: zookeeper3
> at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.storm.shade.org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
>
> at
> org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
>
> at
> org.apache.storm.shade.org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
>
> at
> org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:150)
>
> at
> org.apache.storm.shade.org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
>
> at
> org.apache.storm.shade.org.apache.curator.HandleHolder.getZooKeeper(HandleHolder.java:55)
>
> at
> org.apache.storm.shade.org.apache.curator.ConnectionState.reset(ConnectionState.java:218)
>
> at
> org.apache.storm.shade.org.apache.curator.ConnectionState.start(ConnectionState.java:103)
>
> at
> org.apache.storm.shade.org.apache.curator.CuratorZookeeperClient.start(CuratorZookeeperClient.java:190)
>
> at
> org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:259)
>
> at org.apache.storm.zookeeper$mk_client.doInvoke(zookeeper.clj:86)
> at clojure.lang.RestFn.invoke(RestFn.java:494)
> at
> org.apache.storm.cluster_state.zookeeper_state_factory$_mkState.invoke(zookeeper_state_factory.clj:28)
>
> at org.apache.storm.cluster_state.zookeeper_state_factory.mkState(Unknown
> Source)
>
> {noformat}
> Having done some research it looks like this error is caused by a bug in the
> Zookeeper client library. There is an issue for it here:
> [https://issues.apache.org/jira/browse/ZOOKEEPER-1576]
> This issue has been resolved in the version 3.5.x branch of Zookeeper.
> However, after 2.5 years and 3 releases the 3.5.x branch of Zookeeper is
> still in Alpha .
> Despite the fact that it is in alpha, there is a branch of Curator (v.3.x.x)
> that uses it, but Storm uses Curator version 2.x.x – possibly because it
> doesn’t rely on alpha code. So the bug is still unpatched in Storm.
> I realise that an upgrade to alpha code may be too much of a risk, but this
> problem is a serious issue for those running Storm in a containerised or
> cloud environment - so perhaps it may be worth considering?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (STORM-2415) Storm fails to properly handle Zookeeper hosts going down
[
https://issues.apache.org/jira/browse/STORM-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925850#comment-15925850
]
Anthony Milbourne commented on STORM-2415:
--
This issue was created as a result of discussion on the previous issue.
> Storm fails to properly handle Zookeeper hosts going down
> -
>
> Key: STORM-2415
> URL: https://issues.apache.org/jira/browse/STORM-2415
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.3
> Environment: All
>Reporter: Anthony Milbourne
>
> We run a storm cluster (v.1.0.3) on AWS and have 3 Zookeepers supporting it.
> Because AWS sometimes terminates VMs, we sometimes lose a Zookeeper instance.
> When this happens, the hostname cannot be resolved for that zookeeper
> instance as AWS has taken the VM away. We noticed that in this case storm
> fails to connect to zookeeper – even though there are still 2 Zookeeper
> instances running. It fails with an exception something like:
> {noformat}
> java.net.UnknownHostException: zookeeper3
> at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.storm.shade.org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
>
> at
> org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
>
> at
> org.apache.storm.shade.org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
>
> at
> org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:150)
>
> at
> org.apache.storm.shade.org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
>
> at
> org.apache.storm.shade.org.apache.curator.HandleHolder.getZooKeeper(HandleHolder.java:55)
>
> at
> org.apache.storm.shade.org.apache.curator.ConnectionState.reset(ConnectionState.java:218)
>
> at
> org.apache.storm.shade.org.apache.curator.ConnectionState.start(ConnectionState.java:103)
>
> at
> org.apache.storm.shade.org.apache.curator.CuratorZookeeperClient.start(CuratorZookeeperClient.java:190)
>
> at
> org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:259)
>
> at org.apache.storm.zookeeper$mk_client.doInvoke(zookeeper.clj:86)
> at clojure.lang.RestFn.invoke(RestFn.java:494)
> at
> org.apache.storm.cluster_state.zookeeper_state_factory$_mkState.invoke(zookeeper_state_factory.clj:28)
>
> at org.apache.storm.cluster_state.zookeeper_state_factory.mkState(Unknown
> Source)
>
> {noformat}
> Having done some research it looks like this error is caused by a bug in the
> Zookeeper client library. There is an issue for it here:
> [https://issues.apache.org/jira/browse/ZOOKEEPER-1576]
> This issue has been resolved in the version 3.5.x branch of Zookeeper.
> However, after 2.5 years and 3 releases the 3.5.x branch of Zookeeper is
> still in Alpha .
> Despite the fact that it is in alpha, there is a branch of Curator (v.3.x.x)
> that uses it, but Storm uses Curator version 2.x.x – possibly because it
> doesn’t rely on alpha code. So the bug is still unpatched in Storm.
> I realise that an upgrade to alpha code may be too much of a risk, but this
> problem is a serious issue for those running Storm in a containerised or
> cloud environment - so perhaps it may be worth considering?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (STORM-2415) Storm fails to properly handle Zookeeper hosts going down
Anthony Milbourne created STORM-2415:
Summary: Storm fails to properly handle Zookeeper hosts going down
Key: STORM-2415
URL: https://issues.apache.org/jira/browse/STORM-2415
Project: Apache Storm
Issue Type: Bug
Components: storm-core
Affects Versions: 1.0.3
Environment: All
Reporter: Anthony Milbourne
We run a storm cluster (v.1.0.3) on AWS and have 3 Zookeepers supporting it.
Because AWS sometimes terminates VMs, we sometimes lose a Zookeeper instance.
When this happens, the hostname cannot be resolved for that zookeeper instance
as AWS has taken the VM away. We noticed that in this case storm fails to
connect to zookeeper – even though there are still 2 Zookeeper instances
running. It fails with an exception something like:
{noformat}
java.net.UnknownHostException: zookeeper3
at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at
org.apache.storm.shade.org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
at
org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
at
org.apache.storm.shade.org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
at
org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:150)
at
org.apache.storm.shade.org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
at
org.apache.storm.shade.org.apache.curator.HandleHolder.getZooKeeper(HandleHolder.java:55)
at
org.apache.storm.shade.org.apache.curator.ConnectionState.reset(ConnectionState.java:218)
at
org.apache.storm.shade.org.apache.curator.ConnectionState.start(ConnectionState.java:103)
at
org.apache.storm.shade.org.apache.curator.CuratorZookeeperClient.start(CuratorZookeeperClient.java:190)
at
org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:259)
at org.apache.storm.zookeeper$mk_client.doInvoke(zookeeper.clj:86)
at clojure.lang.RestFn.invoke(RestFn.java:494)
at
org.apache.storm.cluster_state.zookeeper_state_factory$_mkState.invoke(zookeeper_state_factory.clj:28)
at org.apache.storm.cluster_state.zookeeper_state_factory.mkState(Unknown
Source)
{noformat}
Having done some research it looks like this error is caused by a bug in the
Zookeeper client library. There is an issue for it here:
[https://issues.apache.org/jira/browse/ZOOKEEPER-1576]
This issue has been resolved in the version 3.5.x branch of Zookeeper. However,
after 2.5 years and 3 releases the 3.5.x branch of Zookeeper is still in Alpha .
Despite the fact that it is in alpha, there is a branch of Curator (v.3.x.x)
that uses it, but Storm uses Curator version 2.x.x – possibly because it
doesn’t rely on alpha code. So the bug is still unpatched in Storm.
I realise that an upgrade to alpha code may be too much of a risk, but this
problem is a serious issue for those running Storm in a containerised or cloud
environment - so perhaps it may be worth considering?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
