[
https://issues.apache.org/jira/browse/STORM-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikita Gorbachevski updated STORM-3090:
---------------------------------------
Summary: The same offset value is used by different topics. (was: Offset
out of range unpredictable behavior)
> The same offset value is used by different topics.
> --------------------------------------------------
>
> Key: STORM-3090
> URL: https://issues.apache.org/jira/browse/STORM-3090
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka
> Affects Versions: 1.0.5
> Reporter: Nikita Gorbachevski
> Priority: Major
>
> Our topology uses KafkaSpout to fetch messages from kafka topics. We have
> ~150 topics with ~12 partitions, 8 storm executors and tasks on 2 storm nodes.
> Storm version 1.0.5, Kafka brokers version 10.0.2, Kafka clients version
> 0.9.0.1. We do not delete Kafka topics.
> At some moment in time i observed huge amount of repetitive WARN messages in
> worker.log
> 2018-05-29 14:36:57.928 o.a.s.k.KafkaUtils Thread-15-kafka-spout-executor[18
> 18] [WARN] Partition\{host1:9092, topic=topic_1, partition=10} Got fetch
> request with offset out of range: [9248]
> 2018-05-29 14:36:57.929 o.a.s.k.KafkaUtils Thread-23-kafka-spout-executor[16
> 16] [WARN] Partition\{host=host2:9092, topic=topic_2, partition=0} Got fetch
> request with offset out of range: [22650006]
> 2018-05-29 14:36:57.930 o.a.s.k.KafkaUtils Thread-15-kafka-spout-executor[18
> 18] [WARN] Partition\{host=host2:9092, topic=topic_1, partition=2} Got fetch
> request with offset out of range: [6780]
> 2018-05-29 14:36:57.930 o.a.s.k.KafkaUtils Thread-23-kafka-spout-executor[16
> 16] [WARN] Partition\{host=host3:9092, topic=topic_3, partition=4} Got fetch
> request with offset out of range: [1011584]
> 2018-05-29 14:36:57.932 o.a.s.k.KafkaUtils Thread-7-kafka-spout-executor[12
> 12] [WARN] Partition\{host1:9092, topic=topic4, partition=4} Got fetch
> request with offset out of range: [9266]
> 2018-05-29 14:36:57.933 o.a.s.k.KafkaUtils Thread-7-kafka-spout-executor[12
> 12] [WARN] Partition\{host=host2:9092, topic=topic5, partition=4} Got fetch
> request with offset out of range: [9266]
> 2018-05-29 14:36:57.935 o.a.s.k.KafkaUtils Thread-23-kafka-spout-executor[16
> 16] [WARN] Partition\{host1:9092, topic=topic6, partition=4} Got fetch
> request with offset out of range: [1011584]
> 2018-05-29 14:36:57.935 o.a.s.k.KafkaUtils Thread-29-kafka-spout-executor[14
> 14] [WARN] Partition\{host1:9092, topic=topic7, partition=2} Got fetch
> request with offset out of range: [337]
> 2018-05-29 14:36:57.936 o.a.s.k.KafkaUtils Thread-15-kafka-spout-executor[18
> 18] [WARN] Partition\{host=host2:9092, topic=topic6, partition=10} Got fetch
> request with offset out of range: [9248]
> For some reason the same constant offset value is used for the same partition
> of different topics.
> I enabled DEBUG mode and observed log files more precisely.
> 2018-05-29 14:37:03.573 o.a.s.k.PartitionManager
> Thread-7-kafka-spout-executor[12 12] [DEBUG] Wrote last completed offset
> (1572936) to ZK for Partition\{host=host3:9092, topic=topic1, partition=8}
> for topology: topology1
> 2018-05-29 14:37:03.577 o.a.s.k.PartitionManager
> Thread-7-kafka-spout-executor[12 12] [DEBUG] Wrote last completed offset
> (1572936) to ZK for Partition\{host=host1:9092, topic=topic2, partition=8}
> for topology: topology1
> 2018-05-29 14:37:03.578 o.a.s.k.PartitionManager
> Thread-7-kafka-spout-executor[12 12] [DEBUG] Wrote last completed offset
> (1572936) to ZK for Partition\{host=host2:9092, topic=topic3, partition=8}
> for topology: topology1
> ...
> 2018-05-29 14:38:07.581 o.a.s.k.PartitionManager
> Thread-23-kafka-spout-executor[16 16] [DEBUG] Wrote last completed offset
> (61292573) to ZK for Partition\{host=host1:9092, topic=topic4, partition=8}
> for topology: topology1
> 2018-05-29 14:38:07.582 o.a.s.k.PartitionManager
> Thread-23-kafka-spout-executor[16 16] [DEBUG] Wrote last completed offset
> (61292573) to ZK for Partition\{host=host2:9092, topic=topic5, partition=8}
> for topology: topology1
> 2018-05-29 14:38:07.584 o.a.s.k.PartitionManager
> Thread-23-kafka-spout-executor[16 16] [DEBUG] Wrote last completed offset
> (61292573) to ZK for Partition\{host=host3:9092, topic=topic6, partition=8}
> for topology: topology1
> I noticed that some part of all the topics was split in two independent
> groups. Each group consisted of 31 topics.
> All the topics in each group were using the same offset value for each
> partition. However that value wasn't constant and vary between 8 different
> values.
> Each of these 8 values was correct for a particular topics from the group.
> Moreover this value was growing over the time and all the topics updated it
> synchronously.
> Most of the topics(55 from 62) from each group had a corresponding 'offset
> out or range' WARNING message,
> but with the constant value. Other 7 topics were continuing to work correctly
> without WARNING messages but their offset value was changing as well.
> I went through the source code of storm-kafka and noticed that
> useStartOffsetTimeIfOffsetOutOfRange flag doesn't work in our case, because
> we don't have
> failed tuples and kafka offset is less than _emittedToOffset. However i don't
> understand to is it possible that _emittedToOffset got the same value
> for different topics. Do you probably have any ideas why this could happen?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
