It's a know problem... (eg: https://issues.apache.org/jira/browse/KAFKA-6607)
Also `lastOffset = endOffset - 2` does not necessarily do the trick, because if you have multiple in-flight transactions (from different producers), there might be multiple commit markers of different transactions at the end of the topic. -Matthias On 3/6/19 9:08 AM, Zach Cox wrote: > Hi - we've observed that after committing a transaction, commit marker > records are appended to all topic partitions written to in the transaction, > these commit markers are assigned offsets in the partition, and > KafkaConsumer.endOffsets will return the offsets of commit markers if they > are the last records appended to the partition, although the consumer will > never consume records with those commit marker offsets. > > This can lead to problems, for example on startup a system gets current end > offsets for a topic's partitions and tracks the current offsets a consumer > has consumed, it will never be "caught up" to the last offsets since they > are the commit marker offsets, until newer records are written to those > partitions. A workaround to this specific problem is to assume lastOffset = > endOffset - 2, but at best this is surprising until you learn about commit > markers. > > How have others dealt with this? Are we missing anything important in the > above? Could the docs include more details about commit markers? Should > KafkaConsumer.endOffsets account for commit markers somehow? > > Thanks, > Zach >
signature.asc
Description: OpenPGP digital signature