[
https://issues.apache.org/jira/browse/KAFKA-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482623#comment-16482623
]
Jason Gustafson commented on KAFKA-6873:
----------------------------------------
If a fetch offset lands in the middle of a batch of records (either old or new
message format), the broker will return the full batch, which may include
offsets lower than the fetched offset. It is expected that clients will filter
these messages.
> Broker is not returning data including requested offset
> -------------------------------------------------------
>
> Key: KAFKA-6873
> URL: https://issues.apache.org/jira/browse/KAFKA-6873
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 1.1.0
> Environment: ubuntu
> Reporter: Adam Dratwinski
> Priority: Blocker
>
> After upgrading Kafka to 1.1.0 from 0.9.x I experience issues related with
> broker returning incomplete responses. This happens for my all log compacted
> topics. I am using Golang client (Sarama).
> I debugged the issue and found that for some requests brokers return
> FetchResponse with all messages having offsets lower then requested. For
> example, I request for offset 1078831, I get FetchResponse with only one
> message having offset 1078830, which produces missing blocks error. If I
> request the next offset (1078832), then I get a block with many messages,
> starting with much higher offset (e.g 1083813). There is a big gap in offsets
> between these records, probably because I am using log compacted topics, but
> all expected messages are there.
> Sarama client treats this as consumer error:
> {quote}kafka: response did not contain all the expected topic/partition blocks
> {quote}
> For build-in java client this issue is not happening. Looks like it is less
> restrict regarding the data order, and when the offset is missing in the
> returned block, it just simply request the next offset.
> I reported this issue at Shopify/sarama Github project (see
> [https://github.com/Shopify/sarama/issues/1087)], where I got response, that
> this seems to be Kafka bug, as according to the documentation, in this
> situation broker should never return only messages having lower offsets.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
