I have used what Gwen has suggested but to avoid false positive:
While consuming records keep track of *last* consumed offset and compare
with latest offset on broker for consumed topic when you get TimeOut
Exception for that particular partition for given topic (e.g JMX Bean
*LogEndOffset
*for
Thanks everyone.
To answer Charlie's question:
I'm doing some simple stream processing. I have Topics A,B, and C, all using
log compaction and all recordings having primary keys. The data in Topic A is
essentially a routing table that tells me which primary keys in Topics B and C
I should pay
@Gwen- But that only works for topics that have low enough traffic that you
would ever actually hit that timeout.
The Confluent schema registry needs to do something similar to make sure it
has fully consumed the topic it stores data in so it doesn't serve stale
data. We know in our case we'll
For Flume, we use the timeout configuration and catch the exception, with
the assumption that no messages for few seconds == the end.
On Sat, May 9, 2015 at 2:04 AM, James Cheng jch...@tivo.com wrote:
Hi,
I want to use the high level consumer to read all partitions for a topic,
and know when
Hi James,
What are you trying to do exactly? If all you are trying to do is monitor
how far behind a consumer is getting you could use the ConsumerOffsetChecker.
As described in the link below.
http://community.spiceworks.com/how_to/77610-how-far-behind-is-your-kafka-consumer
Each message being
I've created a patch to expose the high end watermark, having this exact
requirement.
Still waiting for it to be accepted, but are using this in production at
the moment and it works quite nicely:
https://issues.apache.org/jira/browse/KAFKA-1977
On Sat, 9 May 2015 at 18:43 Charlie Knudsen
Hi,
I want to use the high level consumer to read all partitions for a topic, and
know when I have reached the end. I know the end might be a little vague,
since items keep showing up, but I'm trying to get as close as possible. I know
that more messages might show up later, but I want to know
