In our broker logs, we occasionally see errors like this: 2013-12-23 05:02:08,456 ERROR [kafka-request-handler-2] server.KafkaApis - [KafkaApi-45] Error when processing fetch request for partition [mytopic,0] offset 204243601 from consumer with correlation id 130341 kafka.common.OffsetOutOfRangeException: Request for offset 204243601 but we only have log segments in the range 204343397 to 207423640.
I assume this means there's a consumer that has fallen behind consuming messages, and the log retention policy has removed messages before they could be consumed by the consumer. However, I'm not 100% which consumer it is, and it looks like the only info we have is the correlation id of the consumer, e.g.: "from consumer with correlation id 130341" Is there a way to know which consumer this refers to? It seems there are far more correlation id's than there are consumers. Would it be possible to provide a bit more descriptive error message here, so we can immediately know which consumer is falling behind? We do see a corresponding entry in the consumer logs too: 2013-12-23 05:02:08,797 WARN [ConsumerFetcherThread-myconsumergroup-1387353494862-7aa0c61d-0-45] consumer.ConsumerFetcherThread - [ConsumerFetcherThread-myconsumergroup-1387353494862-7aa0c61d-0-45], Current offset 204243601 for partition [mytopic,0] out of range; reset offset to 204343397 But it would be nice to be able to also use the broker log to quickly find consumers with issues. Also, I'm not sure, why is logging the event as an ERROR in the broker, but a WARN in the consumer? Jason
