You can store offsets wherever you prefer, and it's separate from processes
you mentioned. Unfortunately custom offset storage support has to be
entirely on client side, one cannot extend (easily) Kafka broker with
support for different offset storage. This has as a consequence that
existing Kafka
Thank you for the detailed explanation.
Is it essential that offsets be stored in Kafka, or could they be stored
outside the kafka/zookeeper system? Will it affect how logs are managed,
and when “older” messages are purged? Or are they two independent systems?
On 11/2/15, 03:51, "Stevo Slavić"
Hi David,
My Bad. I did not understand your question correctly. Thanks Stevo for
detailed explanation.
Just incase, if you want checkout kafka-based offset management, this is a
very good presentation :
http://www.slideshare.net/jjkoshy/offset-management-in-kafka
Thanks,
Mayuresh
On Mon, Nov 2
Here is a bit longer and more detailed, not necessarily better
understandable explanation.
When using Kafka for offsets storage, consumer offsets get stored in
(compacted) consumer offsets Kafka topic ("__consumer_offsets"). Every
partition of consumer offsets topic could store offsets from multip
Hello David,
In short, problem is not with your topic, it is with consumer offsets topic
initialization.
You could modify your code to just retry fetching offsets (until successful
where successful is also return of -1 offset, or timeout), or alternatively
you could trigger consumer offsets topic
Yes. I know all of that. I guess my original message was not clear.
The topic metadata indicates that there is a leader.
Please see my comments interspersed below.
Thanks
David Corbin
On 10/29/15, 17:53, "Mayuresh Gharat" wrote:
> NotCoordinatorForConsumerException is for consumer side.
I a
NotCoordinatorForConsumerException is for consumer side. I think if you
are using simpleConsumer, then you have to do your own offset management.
TopicMetadata tells you whether there is a leader for the topic partitions,
replicas, ISR.
Thanks,
Mayuresh
On Wed, Oct 28, 2015 at 8:23 AM, David C
