[
https://issues.apache.org/jira/browse/KAFKA-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004454#comment-15004454
]
Jason Gustafson commented on KAFKA-2500:
----------------------------------------
[~willf] Apologies for the lack of progress on this issue. It does not look
like it will get into 0.9 unfortunately. With all of the movement in the new
consumer internals and with an already large consumer API, the thought was to
hold hold back on this issue to have enough time to think it through (there may
be some impact, for example, from KIP-32 on how offsets are exposed through the
API). Although it may not be much use to you if it's not in the release, you
can have a look at a quick patch I wrote this morning to expose the high
watermark in ConsumerRecords:
https://github.com/hachikuji/kafka/commit/9eb745ce49039ee11b2c0f34a6fb92d11f261a42.
I think this will be one of the major areas of focus for the next release.
> Make logEndOffset available in the 0.8.3 Consumer
> -------------------------------------------------
>
> Key: KAFKA-2500
> URL: https://issues.apache.org/jira/browse/KAFKA-2500
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer
> Affects Versions: 0.9.0.0
> Reporter: Will Funnell
> Assignee: Jason Gustafson
> Priority: Critical
> Fix For: 0.9.0.0
>
>
> Originally created in the old consumer here:
> https://issues.apache.org/jira/browse/KAFKA-1977
> The requirement is to create a snapshot from the Kafka topic but NOT do
> continual reads after that point. For example you might be creating a backup
> of the data to a file.
> This ticket covers the addition of the functionality to the new consumer.
> In order to achieve that, a recommended solution by Joel Koshy and Jay Kreps
> was to expose the high watermark, as maxEndOffset, from the FetchResponse
> object through to each MessageAndMetadata object in order to be aware when
> the consumer has reached the end of each partition.
> The submitted patch achieves this by adding the maxEndOffset to the
> PartitionTopicInfo, which is updated when a new message arrives in the
> ConsumerFetcherThread and then exposed in MessageAndMetadata.
> See here for discussion:
> http://search-hadoop.com/m/4TaT4TpJy71
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
