Hi James, What are you trying to do exactly? If all you are trying to do is monitor how far behind a consumer is getting you could use the ConsumerOffsetChecker. As described in the link below. http://community.spiceworks.com/how_to/77610-how-far-behind-is-your-kafka-consumer
Each message being processed will also have the offset and partition attached to it so with that data. I suppose that information plus info from a fetch response you could determine this with in an application. https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchResponse Does that help? On Fri, May 8, 2015 at 6:04 PM, James Cheng <jch...@tivo.com> wrote: > Hi, > > I want to use the high level consumer to read all partitions for a topic, > and know when I have reached "the end". I know "the end" might be a little > vague, since items keep showing up, but I'm trying to get as close as > possible. I know that more messages might show up later, but I want to know > when I've received all the items that are currently available in the topic. > > Is there a standard/recommended way to do this? > > I know one way to do it is to first issue an OffsetRequest for each > partition, which would get me the last offset, and then use that > information in my high level consumer to detect when I've reached that a > message with that offset. Which is exactly what the SimpleConsumer example > does ( > https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example). > That involves finding the leader for the partition, etc etc. Not hard, but > a bunch of steps. > > I noticed that kafkacat has an option similar to what I'm looking for: > -e Exit successfully when last message received > > Looking at the code, it appears that a FetchRequest returns the > HighwaterMarkOffset mark for a partition, and the API docs confirm that: > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchResponse > > Does the Java high-level consumer expose the HighwaterMarkOffset in any > way? I looked but I couldn't find such a thing. > > Thanks, > -James > >
