Is the log compaction thread correctly working? The offsets are stored in a log compacted topic, and we have seen issues where the log cleaner thread dies and therefore the offsets topic just grows forever, which means it will take a long time to read in the topic.
You can look in the log-cleaner.log debuglog file to see if there are any error messages there. -James > On May 6, 2016, at 6:28 AM, Jörg Wagner <joerg.wagn...@1und1.de> wrote: > > After a bit more looking into this we found out that the Offsetmanager is > single threaded and due to our setup (few, powerful servers: rather bad for > kafka I know..) it seems we are limiting ourselves severely by using kafka > offsets. > > Any more insight is still welcome of course. > > > -------- Forwarded Message -------- > Subject: unknown (kafka) offsets after restart > Date: Fri, 6 May 2016 14:12:24 +0200 > From: Jörg Wagner <joerg.wagn...@1und1.de> > Reply-To: users@kafka.apache.org > To: users@kafka.apache.org > > > > We're using Kafka 0.8.2 and are puzzled by the offset behaviour when > they are stored in kafka topics. > > Upon restart of the Kafka cluster (e.g. due to reconfiguration) it can > happen that the offsets are unknown and therefore stop consumers from > consuming without knowing their offset. > > kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group > --zookeeper localhost:2181 > Could not fetch offset for [topic,19] due to > kafka.common.OffsetsLoadInProgressException. > > Group Topic Pid Offset > logSize Lag Owner > group topic 0 unknown 326606 unknown none > > I currently have no other solution to this than to wait.. but it takes a > very long time (hours.. the servers are hopelessly underutilized!), I > would be grateful for any advice. > > Thanks > Jörg > > >
