[
https://issues.apache.org/jira/browse/KAFKA-1640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138733#comment-14138733
]
chenshangan commented on KAFKA-1640:
------------------------------------
The following is about the test:
offset info within each partition:
-------------Topic[org.mobile] Offsets--------------
[partition_id:0, earliest:1771669312, latest:1868984348, count:97315036]
[partition_id:1, earliest:1768183261, latest:1866445674, count:98262413]
[partition_id:2, earliest:1772497970, latest:1870503255, count:98005285]
[partition_id:3, earliest:1773379686, latest:1874249045, count:100869359]
[partition_id:4, earliest:1770481630, latest:1869468413, count:98986783]
[partition_id:5, earliest:1773971554, latest:1872383470, count:98411916]
[partition_id:6, earliest:1767987875, latest:1867386661, count:99398786]
[partition_id:7, earliest:1772896791, latest:1871889556, count:98992765]
[partition_id:8, earliest:1767845437, latest:1866253739, count:98408302]
[partition_id:9, earliest:1768493678, latest:1865977579, count:97483901]
[partition_id:10, earliest:1767675631, latest:1865135380, count:97459749]
[partition_id:11, earliest:1766688851, latest:1863677692, count:96988841]
[partition_id:12, earliest:1773892262, latest:1872170855, count:98278593]
[partition_id:13, earliest:1774271586, latest:1873379949, count:99108363]
[partition_id:14, earliest:1767316658, latest:1865490492, count:98173834]
[partition_id:15, earliest:1775086549, latest:1875130663, count:100044114]
invalid offset info:
partition_id:5, offset:1826483041
offset between the earliest offset and invalid offset : 1826483041 - 1773971554
= 52511487
so if I consume messages before offset 52511475 relative to the earliest offset
and after offset 52511475, it's ok. If I consume messages contain offset
52511475, it will fail without any error message and you can never consume
messages starting from this offset.
> consumer will fail if encounter special offset
> ----------------------------------------------
>
> Key: KAFKA-1640
> URL: https://issues.apache.org/jira/browse/KAFKA-1640
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.8.1.1
> Reporter: chenshangan
> Assignee: Neha Narkhede
>
> I'm running topology on storm which consume messages from kafka. Sometimes I
> encounter exceptions like offset out of range. I did not pay much attention
> on this problem before and just restarted the topology to consume messages
> from latest offset because of various reasons. Recently, more services are
> dependent on kafka, and this problem still exists. According to the invalid
> offset information, I did a test and finally found the problem. There exists
> an offset which consumer can never start from and can not skip over it unless
> you provide next valid offset to start from. I can not figure out why.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
