[
https://issues.apache.org/jira/browse/STORM-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16284070#comment-16284070
]
Stig Rohde Døssing commented on STORM-2847:
-------------------------------------------
Thanks [~erosebrook]. Your configuration and versions look fine to me.
I've spent some time going over the code, and I'm having a hard time spotting
the problem.
The exception you're getting from
https://github.com/apache/storm/blob/e2e3f5d19a8671e3759a04b94135fd6643b3aa61/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java#L475
is saying that the TopicPartition parameter to KafkaConsumer.position was not
assigned to the consumer. I don't understand how that can happen, because the
TopicPartition parameter is based on the offsetManagers map, and the key set of
that map is set to be exactly the partitions that are assigned to the consumer,
at
https://github.com/apache/storm/blob/e2e3f5d19a8671e3759a04b94135fd6643b3aa61/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java#L174
and
https://github.com/apache/storm/blob/e2e3f5d19a8671e3759a04b94135fd6643b3aa61/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java#L197.
Could you try to enable debug logging for the org.apache.storm.kafka.spout log
domain and provoking the error again? I particularly want to see if the
partitions logged at
https://github.com/apache/storm/blob/e2e3f5d19a8671e3759a04b94135fd6643b3aa61/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java#L164
match the ones logged at
https://github.com/apache/storm/blob/e2e3f5d19a8671e3759a04b94135fd6643b3aa61/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java#L469.
I'm also curious if the error happens at random, or does it happen at
particular moments?
> Exception thrown after rebalance IllegalArgumentException
> ---------------------------------------------------------
>
> Key: STORM-2847
> URL: https://issues.apache.org/jira/browse/STORM-2847
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
> Affects Versions: 1.2.0
> Reporter: Evan Rosebrook
> Assignee: Stig Rohde Døssing
>
> After rebalance the storm-kafka-client spout attempts to check the current
> position of partitions that are no longer assigned to the current spout. This
> occurs in a topology with multiple spout instances.
> java.lang.IllegalArgumentException: You can only check the position for
> partitions assigned to this consumer. at
> org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:1262)
> at
> org.apache.storm.kafka.spout.KafkaSpout.commitOffsetsForAckedTuples(KafkaSpout.java:473)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
