RE: how to programatically monitor Kafka availability

[Hohl, Ken]

We're doing this -

every half second
For each broker
get the value of this MBean: 
kafka.server:type=KafkaServer,name=BrokerState
if the value is 4 (this broker is the leader), assume the 
cluster is available

if either couldn't get the MBean from any broker, or none had a value 
of 4, assume the cluster is NOT available

Thanks.

Ken Hohl
Cars.com

-Original Message-
From: hsy...@gmail.com [mailto:hsy...@gmail.com] 
Sent: Thursday, December 17, 2015 1:02 PM
To: users@kafka.apache.org
Subject: Re: how to programatically monitor Kafka availability

Hey Hohl,

I use *partitionsFor
*
method to monitor the partition info for particular topics



On Tue, Dec 15, 2015 at 11:27 AM, Hohl, Ken  wrote:

> We want to be able to monitor the ability to send messages to Kafka 
> topics.  We want to be aware of the inability to do so before the time 
> we attempt to send a message.  What we're looking for is something 
> like a heartbeat.  The reason we need this is that in our deployment 
> environment, Kafka and its clients will not be co-located.  As such, 
> network issues could cause Kafka to not be available to its client.
>
> We've considered using Zookeeper that's already managing the Kafka 
> cluster but have not been able to determine exactly how we would use it.
>
> We've also considered requesting a JMX MBean periodically and 
> concluding the cluster is not accessible if we can't get the MBean 
> from at least 1 broker.
>
> What is the recommended way of accomplishing what we're trying to do?
>
> Thanks.
>
> Ken Hohl
> Cars.com
>
>

Exceptions when programmatically start multiple kafka brokers

[hsy...@gmail.com]

I'm trying to start 2 brokers in my kafka ingestion unit test and I got
exception

javax.management.InstanceAlreadyExistsException:
kafka.server:type=app-info,id=0
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at
org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:57)
at kafka.server.KafkaServer.startup(KafkaServer.scala:239)
at
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at
org.apache.apex.malhar.kafka.KafkaOperatorTestBase.startKafkaServer(KafkaOperatorTestBase.java:133)
at
org.apache.apex.malhar.kafka.KafkaOperatorTestBase.startKafkaServer(KafkaOperatorTestBase.java:143)
at
org.apache.apex.malhar.kafka.KafkaOperatorTestBase.beforeTest(KafkaOperatorTestBase.java:175)

It is caused by JMXMetrcsReporter?
It doesn't affect any function we want, but it is annoying.
How to disable it?

Thanks!

Closing socket for 1.2.4.5 because of error (kafka.network.Processor)

[Arathi Maddula]

Hi,

I found this error in my kafka.out. How can I find what is causing this error? 
Kafka jar : kafka_2.10-0.8.2.2.3.2.0-2950.jar


Closing socket for /1.2.3.4 because of error (kafka.network.Processor)
java.lang.NullPointerException
at kafka.network.Processor.run(SocketServer.scala:404)
at java.lang.Thread.run(Thread.java:745)

Thanks,
Arathi

Re: Measuring Kafka Producer request latency when it is less than 1ms

[Ismael Juma]

Hi Alexey,

Could you please report a bug in JIRA for the NaN result? We should handle
that better.

Thanks,
Ismael

On Mon, Dec 21, 2015 at 9:12 AM, Alexey Pirogov 
wrote:

> I'm looking for help with a question regarding measuring of producer
> request latency.
> I expected that "producer.request-latency-avg/max" will do a good job for
> me. But seems that if latencies less than 1ms in most cases, this metric
> will emit NaN(as it doesn't support float values).
> We need this metric for monitoring purpose.
>
> Is there any way to producer request latency statistic without adding
> callback or blocking of Future from KafkaProducer.send(...) method?
>
> P.S. Technically, we could treat NaN from "producer.request-latency-avg"
> metric as a special case in our monitoring tool, but it will required some
> specific configuration only for this metric.
>
> Thank you,
> Alexey
>

Re: Re: Measuring Kafka Producer request latency when it is less than 1ms

[Alexey Pirogov]

Ismael, thanks for reply.

Jire created https://issues.apache.org/jira/browse/KAFKA-3028.

Thank you,
Alexey

Measuring Kafka Producer request latency when it is less than 1ms

[Alexey Pirogov]

I'm looking for help with a question regarding measuring of producer
request latency.
I expected that "producer.request-latency-avg/max" will do a good job for
me. But seems that if latencies less than 1ms in most cases, this metric
will emit NaN(as it doesn't support float values).
We need this metric for monitoring purpose.

Is there any way to producer request latency statistic without adding
callback or blocking of Future from KafkaProducer.send(...) method?

P.S. Technically, we could treat NaN from "producer.request-latency-avg"
metric as a special case in our monitoring tool, but it will required some
specific configuration only for this metric.

Thank you,
Alexey

Unifying kafka-clients call signatures

[Pierre-Yves Ritschard]

Hi list,

I've been working on an issue at
https://issues.apache.org/jira/browse/KAFKA-3006 and it is now a good
time to ask for feedback.

The attached PR moves all signatures which accepted either arrays or
java.util.List to accept java.util.Collection. The aim is to provide
consumers of kafka-clients a unified way to work with sequences.

Some concern was raised in the issue wrt to potential source
compatibility issues when different versions of the kafka-clients JAR
end up on a given classpath. Any people who feel they might be impacted
is encouraged to mention it here to inform the decision (it would still
be possible to keep the other signatures around but it adds a load of
bloat and decreases legibility/clarity IMO).

Kafka 0.9.0 New Java Consumer API fetching duplicate records

[pradeep kumar]

Can someone please help me on this.
http://stackoverflow.com/questions/34405124/kafka-0-9-0-new-java-consumer-api-fetching-duplicate-records

Thanks,
Pradeep