RE: how to programatically monitor Kafka availability

[Hohl, Ken]

We're doing this -

every half second
For each broker
get the value of this MBean: 
kafka.server:type=KafkaServer,name=BrokerState
if the value is 4 (this broker is the leader), assume the 
cluster is available

if either couldn't get the MBean from any broker, or none had a value 
of 4, assume the cluster is NOT available

Thanks.

Ken Hohl
Cars.com

-Original Message-
From: hsy...@gmail.com [mailto:hsy...@gmail.com] 
Sent: Thursday, December 17, 2015 1:02 PM
To: users@kafka.apache.org
Subject: Re: how to programatically monitor Kafka availability

Hey Hohl,

I use *partitionsFor
<http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#partitionsFor%28java.lang.String%29>*
method to monitor the partition info for particular topics



On Tue, Dec 15, 2015 at 11:27 AM, Hohl, Ken  wrote:

> We want to be able to monitor the ability to send messages to Kafka 
> topics.  We want to be aware of the inability to do so before the time 
> we attempt to send a message.  What we're looking for is something 
> like a heartbeat.  The reason we need this is that in our deployment 
> environment, Kafka and its clients will not be co-located.  As such, 
> network issues could cause Kafka to not be available to its client.
>
> We've considered using Zookeeper that's already managing the Kafka 
> cluster but have not been able to determine exactly how we would use it.
>
> We've also considered requesting a JMX MBean periodically and 
> concluding the cluster is not accessible if we can't get the MBean 
> from at least 1 broker.
>
> What is the recommended way of accomplishing what we're trying to do?
>
> Thanks.
>
> Ken Hohl
> Cars.com
>
>

Re: how to programatically monitor Kafka availability

[hsy...@gmail.com]

Hey Hohl,

I use *partitionsFor
*
method to monitor the partition info for particular topics



On Tue, Dec 15, 2015 at 11:27 AM, Hohl, Ken  wrote:

> We want to be able to monitor the ability to send messages to Kafka
> topics.  We want to be aware of the inability to do so before the time we
> attempt to send a message.  What we're looking for is something like a
> heartbeat.  The reason we need this is that in our deployment environment,
> Kafka and its clients will not be co-located.  As such, network issues
> could cause Kafka to not be available to its client.
>
> We've considered using Zookeeper that's already managing the Kafka cluster
> but have not been able to determine exactly how we would use it.
>
> We've also considered requesting a JMX MBean periodically and concluding
> the cluster is not accessible if we can't get the MBean from at least 1
> broker.
>
> What is the recommended way of accomplishing what we're trying to do?
>
> Thanks.
>
> Ken Hohl
> Cars.com
>
>

how to programatically monitor Kafka availability

[Hohl, Ken]

We want to be able to monitor the ability to send messages to Kafka topics.  We 
want to be aware of the inability to do so before the time we attempt to send a 
message.  What we're looking for is something like a heartbeat.  The reason we 
need this is that in our deployment environment, Kafka and its clients will not 
be co-located.  As such, network issues could cause Kafka to not be available 
to its client.

We've considered using Zookeeper that's already managing the Kafka cluster but 
have not been able to determine exactly how we would use it.

We've also considered requesting a JMX MBean periodically and concluding the 
cluster is not accessible if we can't get the MBean from at least 1 broker.

What is the recommended way of accomplishing what we're trying to do?

Thanks.

Ken Hohl
Cars.com