[jira] [Updated] (STORM-2844) KafkaSpout Throws IllegalStateException After Committing to Kafka When First Poll Strategy Set to EARLIEST
[ https://issues.apache.org/jira/browse/STORM-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated STORM-2844: -- Labels: pull-request-available (was: ) > KafkaSpout Throws IllegalStateException After Committing to Kafka When First > Poll Strategy Set to EARLIEST > -- > > Key: STORM-2844 > URL: https://issues.apache.org/jira/browse/STORM-2844 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.2.0 >Reporter: Hugo Louro >Assignee: Hugo Louro >Priority: Critical > Labels: pull-request-available > > This > [code|https://github.com/apache/storm/blob/1.x-branch/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java#L407-L409], > which was committed to fix > [STORM-2666|https://issues.apache.org/jira/browse/STORM-2666] throws > IllegalStateException when the KafkaSpout commits to Kafka and is restarted > with the same consumer group id and first poll strategy is set to EARLIEST. > For example consider the following sequence: > # KafkaSpout with consumer_group_id=TEST polls and commits offsets 1-5 > # KafkaSpout with consumer_group_id=TEST is restarted with first poll > strategy set to EARLIEST > ==> IllegalStateException will be thrown > This bug could be a blocker. I am setting it to Critical because assigning a > different consumer id serves as a workaround to the problem. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2847) Exception thrown after rebalance IllegalArgumentException
[ https://issues.apache.org/jira/browse/STORM-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated STORM-2847: -- Fix Version/s: 2.0.0 > Exception thrown after rebalance IllegalArgumentException > - > > Key: STORM-2847 > URL: https://issues.apache.org/jira/browse/STORM-2847 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.2.0 >Reporter: Evan Rosebrook >Assignee: Stig Rohde Døssing > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > After rebalance the storm-kafka-client spout attempts to check the current > position of partitions that are no longer assigned to the current spout. This > occurs in a topology with multiple spout instances. > java.lang.IllegalArgumentException: You can only check the position for > partitions assigned to this consumer. at > org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:1262) > at > org.apache.storm.kafka.spout.KafkaSpout.commitOffsetsForAckedTuples(KafkaSpout.java:473) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2851) org.apache.storm.kafka.spout.KafkaSpout.doSeekRetriableTopicPartitions sometimes throws ConcurrentModificationException
[ https://issues.apache.org/jira/browse/STORM-2851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing resolved STORM-2851. --- Resolution: Fixed Fix Version/s: 1.2.0 > org.apache.storm.kafka.spout.KafkaSpout.doSeekRetriableTopicPartitions > sometimes throws ConcurrentModificationException > --- > > Key: STORM-2851 > URL: https://issues.apache.org/jira/browse/STORM-2851 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.2.0 > Environment: Using Storm 1.2.0 preview binaries shared by Stig Rohde > Døssing & Jungtaek Lim through the "[Discuss] Release Storm 1.2.0" discussion > is Storm Developer's mailing list > With one Nimbus Vm, 6 Supervisor VMs, 3 Zookeeper VMs, 15 topologies, talking > with a 5 VMs Kafka Brokers set (based on Kafka 0.10.2), all with ORACLE > Server JRE 8 update 152. > About 15 topologies, handling around 1 million Kafka messages per minute, and > connected to Redis, OpenTSDB & HBase. >Reporter: Alexandre Vermeerbergen >Assignee: Stig Rohde Døssing > Labels: pull-request-available > Fix For: 2.0.0, 1.2.0 > > Time Spent: 4h 10m > Remaining Estimate: 0h > > Hello, > We have been running Storm 1.2.0 preview on our pre-production supervision > system. > We noticed that in the logs of our topology to logs persistency in Hbase, we > got the following exceptions (about 4 times in a 48 hours period): > java.util.ConcurrentModificationException at > java.util.HashMap$HashIterator.nextNode(HashMap.java:1442) > at java.util.HashMap$KeyIterator.next(HashMap.java:1466) > at > org.apache.storm.kafka.spout.KafkaSpout.doSeekRetriableTopicPartitions(KafkaSpout.java:347) > > at > org.apache.storm.kafka.spout.KafkaSpout.pollKafkaBroker(KafkaSpout.java:320) > at org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:245) > at > org.apache.storm.daemon.executor$fn__4963$fn__4978$fn__5009.invoke(executor.clj:647) > > at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484) > at clojure.lang.AFn.run(AFn.java:22) > at java.lang.Thread.run(Thread.java:748) > It looks like there's something to fix here, such as making the map > thread-safe, or managing the exclusivity of modification of this map at a > caller level. > Note: this topology is using Storm Kafka Client spout with default properties > (unlike other topologies we have based on autocommit). However, it's the one > which deals with highest rate of messages (line of logs coming from about > 1 VMs, a nice scale test for Storm :)) > Could it be fixed in Storm 1.2.0 final version? > Best regards, > Alexandre Vermeerbergen -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2851) org.apache.storm.kafka.spout.KafkaSpout.doSeekRetriableTopicPartitions sometimes throws ConcurrentModificationException
[ https://issues.apache.org/jira/browse/STORM-2851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated STORM-2851: -- Fix Version/s: 2.0.0 > org.apache.storm.kafka.spout.KafkaSpout.doSeekRetriableTopicPartitions > sometimes throws ConcurrentModificationException > --- > > Key: STORM-2851 > URL: https://issues.apache.org/jira/browse/STORM-2851 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.2.0 > Environment: Using Storm 1.2.0 preview binaries shared by Stig Rohde > Døssing & Jungtaek Lim through the "[Discuss] Release Storm 1.2.0" discussion > is Storm Developer's mailing list > With one Nimbus Vm, 6 Supervisor VMs, 3 Zookeeper VMs, 15 topologies, talking > with a 5 VMs Kafka Brokers set (based on Kafka 0.10.2), all with ORACLE > Server JRE 8 update 152. > About 15 topologies, handling around 1 million Kafka messages per minute, and > connected to Redis, OpenTSDB & HBase. >Reporter: Alexandre Vermeerbergen >Assignee: Stig Rohde Døssing > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 4h > Remaining Estimate: 0h > > Hello, > We have been running Storm 1.2.0 preview on our pre-production supervision > system. > We noticed that in the logs of our topology to logs persistency in Hbase, we > got the following exceptions (about 4 times in a 48 hours period): > java.util.ConcurrentModificationException at > java.util.HashMap$HashIterator.nextNode(HashMap.java:1442) > at java.util.HashMap$KeyIterator.next(HashMap.java:1466) > at > org.apache.storm.kafka.spout.KafkaSpout.doSeekRetriableTopicPartitions(KafkaSpout.java:347) > > at > org.apache.storm.kafka.spout.KafkaSpout.pollKafkaBroker(KafkaSpout.java:320) > at org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:245) > at > org.apache.storm.daemon.executor$fn__4963$fn__4978$fn__5009.invoke(executor.clj:647) > > at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484) > at clojure.lang.AFn.run(AFn.java:22) > at java.lang.Thread.run(Thread.java:748) > It looks like there's something to fix here, such as making the map > thread-safe, or managing the exclusivity of modification of this map at a > caller level. > Note: this topology is using Storm Kafka Client spout with default properties > (unlike other topologies we have based on autocommit). However, it's the one > which deals with highest rate of messages (line of logs coming from about > 1 VMs, a nice scale test for Storm :)) > Could it be fixed in Storm 1.2.0 final version? > Best regards, > Alexandre Vermeerbergen -- This message was sent by Atlassian JIRA (v6.4.14#64029)
