Hi Sean, Unless this has changed, this is the expected behavior when there's no offset associated with a topic-partition under a given consumer group, in which case the Kafka broker will return -1: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetFetchRequest "Note that if there is no offset associated with a topic-partition under that consumer group the broker does not set an error code (since it is not really an error), but returns empty metadata and sets the offset field to -1."
I addressed that issue in another Kafka Exporter by forcibly setting the lag to -1 so that I can monitor simple consumers, which don't support discovery, that are not consuming yet from newly added partitions: https://github.com/danielqsj/kafka_exporter/pull/17 On Tue, 4 Feb 2020 at 20:10, Sean Glover <[email protected]> wrote: > > Hi, > > I maintain Kafka Lag Exporter (an offset lag/latency monitoring tool). I > use the Apache Kafka Java AdminClient and KafkaConsumer to retrieve offset > information to calculate lag and other metrics, similar to how the > kafka-consumer-groups.sh bin script works. A few users have reported Kafka > Lag Exporter crashing due to an internal AdminClient exception caused by > the response from OffsetFetchRequest: "java.lang.IllegalArgumentException: > Invalid negative offset" (see stack trace below). > > We found out that the user is using Azure EventHub For Kafka and that > partition offsets for consumer groups are initialized with offset -1. I've > never heard of this practice before so I thought I'd ask if this is an > initialization default, or a default that users/services do commonly when > users don't want to use Kafka managed offsets. Since > kafka-consumer-groups.sh relies on AdminClient, this issue can be > reproduced using that script as well. > > Kafka Lag Exporter issue: > https://github.com/lightbend/kafka-lag-exporter/issues/120 > > Stack trace: > > Caused by: java.lang.IllegalArgumentException: Invalid negative offset > at > org.apache.kafka.clients.consumer.OffsetAndMetadata.<init>(OffsetAndMetadata.java:50) > at > org.apache.kafka.clients.admin.KafkaAdminClient$25.handleResponse(KafkaAdminClient.java:3018) > at > org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.handleResponses(KafkaAdminClient.java:1076) > at > org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1204) > ... 1 common frames omitted > > Regards, > Sean -- Gabriele
