Jason, You can search the correlation id in the public access log on the servers to get the consumer information.
As for logging, I agree that we should use the same level on both sides. Could you file a jira for this? Guozhang On Mon, Dec 23, 2013 at 3:09 PM, Jason Rosenberg <j...@squareup.com> wrote: > In our broker logs, we occasionally see errors like this: > > 2013-12-23 05:02:08,456 ERROR [kafka-request-handler-2] server.KafkaApis - > [KafkaApi-45] Error when processing fetch request for partition [mytopic,0] > offset 204243601 from consumer with correlation id 130341 > kafka.common.OffsetOutOfRangeException: Request for offset 204243601 but we > only have log segments in the range 204343397 to 207423640. > > I assume this means there's a consumer that has fallen behind consuming > messages, and the log retention policy has removed messages before they > could be consumed by the consumer. > > However, I'm not 100% which consumer it is, and it looks like the only info > we have is the correlation id of the consumer, e.g.: > > "from consumer with correlation id 130341" > > Is there a way to know which consumer this refers to? It seems there are > far more correlation id's than there are consumers. Would it be possible > to provide a bit more descriptive error message here, so we can immediately > know which consumer is falling behind? > > We do see a corresponding entry in the consumer logs too: > > 2013-12-23 05:02:08,797 WARN > [ConsumerFetcherThread-myconsumergroup-1387353494862-7aa0c61d-0-45] > consumer.ConsumerFetcherThread - > [ConsumerFetcherThread-myconsumergroup-1387353494862-7aa0c61d-0-45], > Current offset 204243601 for partition [mytopic,0] out of range; reset > offset to 204343397 > > But it would be nice to be able to also use the broker log to quickly find > consumers with issues. > > Also, I'm not sure, why is logging the event as an ERROR in the broker, but > a WARN in the consumer? > > Jason > -- -- Guozhang
