After a bit more looking into this we found out that the Offsetmanager
is single threaded and due to our setup (few, powerful servers: rather
bad for kafka I know..) it seems we are limiting ourselves severely by
using kafka offsets.
Any more insight is still welcome of course.
-------- Forwarded Message --------
Subject: unknown (kafka) offsets after restart
Date: Fri, 6 May 2016 14:12:24 +0200
From: Jörg Wagner <joerg.wagn...@1und1.de>
Reply-To: users@kafka.apache.org
To: users@kafka.apache.org
We're using Kafka 0.8.2 and are puzzled by the offset behaviour when
they are stored in kafka topics.
Upon restart of the Kafka cluster (e.g. due to reconfiguration) it can
happen that the offsets are unknown and therefore stop consumers from
consuming without knowing their offset.
kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group
--zookeeper localhost:2181
Could not fetch offset for [topic,19] due to
kafka.common.OffsetsLoadInProgressException.
Group Topic Pid Offset
logSize Lag Owner
group topic 0 unknown 326606 unknown none
I currently have no other solution to this than to wait.. but it takes a
very long time (hours.. the servers are hopelessly underutilized!), I
would be grateful for any advice.
Thanks
Jörg
