Re: Monitoring connection with kafka client

Alex, In the above mentioned case I think monitoring the consumer lag will be helpful "kafka.consumer":name="([-.\w]+)-MaxLag",type="ConsumerFetcherManager". Another approach I used before was to use metrics library to send consumer stats to ganglia and use nagios alert on those stats, I

Re: Monitoring connection with kafka client

@Neha Narkede Though monitoring the health of Kafka & Zookeeper clusters directly is useful, it might not be enough. Consider the following scenario: You have a client consuming messages. Zookeeper and kafka nodes are healthy. The monitor says everything is ok. For some reason, a connection betwe

Re: Monitoring connection with kafka client

@Otis thanks for your answer. Of course it is not about manually watching connection. We already have monitoring tools in place. But I would expect the kafka client to provide means for get notified about potential connectivity issues. Actual behavior is this: the client retries to reconnect inde

Re: Monitoring connection with kafka client

Hi, We use our own SPM to monitor our Kafka brokers, producers, and consumers (and ZK) and have alerts and anomaly detection on several key Kafka and ZK metrics. When things break around ZK and/or Kafka. we find out pretty quickly because a lot of metrics suddenly s

Re: Monitoring connection with kafka client

If you want to know if the Kafka and zookeeper cluster is healthy or not, you'd want to monitor the cluster directly. Here are pointers for monitoring the Kafka brokers - http://kafka.apache.org/documentation.html#monitoring Thanks, Neha On Thu, Oct 16, 2014 at 3:09 AM, Alex Objelean wrote: > H

Monitoring connection with kafka client

Hi, I'm trying to monitor the kafka connection on the consumer side. In other words, if the broker cluster is unavailable (or zookeer dies), I would like to know about that problem as soon as possible. Unfortunately, I didn't find anything useful to achieve that when using kafka library. Are there