Re: Kafka Monitoring

; > > > > > Then we append this to KAFKA_OPTS > > > > > > > > > > > > > > -javaagent:/jolokia-jvm-agent.jar=port=8074,host=0.0.0.0 > > > > > > > > > > > > > > > > > > &

Re: Kafka Monitoring

hen "exposed" in Docker image and "allowed" in > > AWS > > > > > > Security Group. > > > > > > Then our monitoring tool is querying the following endpoins > (just a > > > > HTTP > > > > > > GET query) t

Re: Kafka Monitoring

gt; > > > > > > :8074/jolokia/read/kafka.server:* > > > > > :8074/jolokia/read/kafka.controller:* > > > > > :8074/jolokia/read/kafka.log:* > > > > > :8074/jolokia/read/kafka.network:* > > > > > :8074/jolokia/read/ja

Re: Kafka Monitoring

> > > > > > > > M. > > > > > > > > > > > > On 8 December 2017 at 09:28, Irtiza Ali <i...@an10.io> wrote: > > > > > > > > > Hello Michal, > > > > > > > > > > Can you send me link

Re: Kafka Monitoring

; > > > > > > > > > Thank you > > > > Irtiza > > > > > > > > On Wed, Dec 6, 2017 at 8:00 PM, Michal Michalski < > > > > michal.michal...@zalando.ie> wrote: > > > > > > > > > Hi Irtiza, > > &g

Re: Kafka Monitoring

> > > > > > > > > Thank you > > > Irtiza > > > > > > On Wed, Dec 6, 2017 at 8:00 PM, Michal Michalski < > > > michal.michal...@zalando.ie> wrote: > > > > > > > Hi Irtiza, > > > > > > > > We

Re: Kafka Monitoring

> > michal.michal...@zalando.ie> wrote: > > > > > Hi Irtiza, > > > > > > We're using Jolokia and we had no problems with it. > > > It would be useful to know what exactly you did (how you "plugged in" > > > Jolokia, how you configured it, what

Re: Kafka Monitoring

; We're using Jolokia and we had no problems with it. > > It would be useful to know what exactly you did (how you "plugged in" > > Jolokia, how you configured it, what endpoint are you querying etc.) to > > help you. > > > > On 6 December 2017 at 10:36, Ir

Re: Kafka Monitoring

we had no problems with it. > It would be useful to know what exactly you did (how you "plugged in" > Jolokia, how you configured it, what endpoint are you querying etc.) to > help you. > > On 6 December 2017 at 10:36, Irtiza Ali <i...@an10.io> wrote: > > > Hello ev

Re: Kafka Monitoring

to disk. > > https://github.com/jmxtrans/jmxtrans/wiki > > Hope that helps! > > Thanks, > Subhash > > Sent from my iPhone > > > On Dec 6, 2017, at 5:36 AM, Irtiza Ali <i...@an10.io> wrote: > > > > Hello everyone, > > > > I am working pyth

Re: Kafka Monitoring

And Lenses - https://www.landoop.com/kafka-lenses/ On 06/12/2017, 11:21, "Abhimanyu Nagrath" <abhimanyunagr...@gmail.com> wrote: There are a couple of tools for this. 1. Linkedin Burrow - https://github.com/linkedin/Burrow 2. Yahoo Kafka Monitor

Re: Kafka Monitoring

> Hello everyone, > > I am working python based Kafka monitoring application. I am unable to > figure out how to retrieve the metrics using Jolokia. I have enable the > port for metrics retrieval to . > > I Have two questions > > 1) Is there something that I am not do

Re: Kafka Monitoring

of them is the KeyOutWriter which just writes to disk. https://github.com/jmxtrans/jmxtrans/wiki Hope that helps! Thanks, Subhash Sent from my iPhone > On Dec 6, 2017, at 5:36 AM, Irtiza Ali <i...@an10.io> wrote: > > Hello everyone, > > I am working python based Kafka monitor

Re: Kafka Monitoring

<i...@an10.io> wrote: > > Hello everyone, > > I am working python based Kafka monitoring application. I am unable to > figure out how to retrieve the metrics using Jolokia. I have enable the > port for metrics retrieval to . > > I Have two questions > > 1)

Kafka Monitoring

Hello everyone, I am working python based Kafka monitoring application. I am unable to figure out how to retrieve the metrics using Jolokia. I have enable the port for metrics retrieval to . I Have two questions 1) Is there something that I am not doing correctly. 2) Is there some other way

Re: Kafka Monitoring

There are a couple of tools for this. 1. Linkedin Burrow - https://github.com/linkedin/Burrow 2. Yahoo Kafka Monitoring Tool - https://github.com/yahoo/kafka-manager 3. Uber Kafka Monitoring Tool - https://github.com/uber/chaperone Regards, Abhimanyu On Wed, Dec 6, 2017 at 3:14 PM

Kafka Monitoring

Hello, I am working on the Kafka monitoring and writing an application for it. Can anyone tell me how to get the metrics in crude form. I don't want to use Datadog or JConsole some tools like these one. I am currently using jolokia to retrieve metrics. With Regards Irtiza Ali

Re: Kafka Monitoring..

Yeah…jmx is really the cheap/easy way to monitor kafka. You should also monitor OS metrics like pageScanD and pageScanK. Those were very helpful for us. -David On 11/9/17, 11:40 AM, "Andrew Otto" wrote: We’ve recently started using Prometheus, and use Prometheus JMX

Re: Kafka Monitoring..

We’ve recently started using Prometheus, and use Prometheus JMX Exporter to get Kafka metrics into prometheus. Here’s our JMX Exporter config:

Re: Kafka Monitoring..

JMX was pretty easy to setup for us. Look up the various jmx beans locally for your particular version of kafka (i.e. with jconsole..etc) https://github.com/jmxtrans/jmxtrans On 11/8/17, 7:10 PM, "chidigam ." wrote: Hi All, What is the simplest way of

Re: Kafka Monitoring..

Have you seen this thread ? http://search-hadoop.com/m/Kafka/uyzND1Q6wyNBj42g?subj=Kafka+Monitoring On Wed, Nov 8, 2017 at 5:10 PM, chidigam . <bhanu.ki...@gmail.com> wrote: > Hi All, > What is the simplest way of monitoring the metrics in kaka brokers? > Is there any opens

Kafka Monitoring..

Hi All, What is the simplest way of monitoring the metrics in kaka brokers? Is there any opensource available? Any help in this regards is appreciated. Regards Bhanu

Re: Kafka Monitoring

ation above should get you started. >>> Using Telegraf to collect system stats and graph them with Grafana is >>> really simple and powerful, as the Grafana community has a lot of >> pre-built >>> content you can steal and make quick wins with. >>> >>> Monitoring

Re: Kafka Monitoring

l and make quick wins with. > > > > Monitoring Kafka can be a beast, but there is a lot of useful data there > > for if(when?) there is a problem. The more time you spend with the > > metrics, the more you start to get a feel for the internals. > >

Re: Kafka Monitoring

; On Mon, Jun 19, 2017 at 6:52 PM, Muhammad Arshad < > muhammad.ars...@alticeusa.com> wrote: > > > Hi, > > > > wanted to see if there is Kafka monitoring which is available. I am > > looking to the following: > > > > > > > > how much data ca

Re: Kafka Monitoring

with the metrics, the more you start to get a feel for the internals. On Mon, Jun 19, 2017 at 6:52 PM, Muhammad Arshad < muhammad.ars...@alticeusa.com> wrote: > Hi, > > wanted to see if there is Kafka monitoring which is available. I am > looking to the following: > >

Re: Kafka Monitoring

rg" <users@kafka.apache.org> Subject: Kafka Monitoring Hi, wanted to see if there is Kafka monitoring which is available. I am looking to the following: how much data came in at a certain time. Thanks, Muhammad Faisal Arshad Manager, Enterprise Data Quality Data Services & Architecture

Re: Queries regarding kafka Monitoring tool burrow.

The lag numbers are never going to be exactly the same as what the CLI tool returns, as the broker is queried on an interval for the offset at the end of each partition. As far as crashing goes, I’d be interested to hear about specifics as we run it (obviously) and don’t have that problem. It

Re: Queries regarding kafka Monitoring tool burrow.

Hey Abhimanyu, Not directly answering your questions but in the past we used burrow at my current company and we had a horrible time with it. It would crash daily and its lag metrics were very different to what was returned when you would run the kafka-consumer-group describe command as you

Queries regarding kafka Monitoring tool burrow.

Hi , I am using burrow to monitor kafka Lags and I am having following queries : 1.On hitting the API /v2/kafka/local/consumer/group1/lag I am not able to view all the topics details present in that group and getting complete: false in the above JSON. What does this mean? Below mentioned is the

Re: Kafka monitoring

Hi, Maybe you are looking for something like https://github.com/uber/chaperone ? See also: * https://issues.apache.org/jira/browse/KAFKA-260 * https://sematext.com/blog/2016/06/07/kafka-consumer-lag-offsets-monitoring/ Otis -- Monitoring - Log Management - Alerting - Anomaly Detection Solr &

Re: Kafka monitoring

All consumers will "eventually" get the messages. What is it that you want to achieve by monitoring that? For the brokers you can monitor lag, for the producers you can have a counter that tracks messages sent and for consumers have one that tracks messages consumed. Although, just tracking lag

Kafka monitoring

Hi all, I have requirement to monitor — wheather every message sent by the producer is received by each and every kafka consumers or not. That is if any message is not delivered then it should be taken care of properly. I googled out but didn’t find any satisfactory answers. Please suggest.

Re: Kafka Monitoring using JMX Mbeans

g using Mbeans ?What kind of command > line > > > tools been used? > > > > > > > See https://sematext.com/spm/integrations/kafka-monitoring/ > > We use it for monitoring Kafka, ZooKeeper, Elasticsearch, Solr, etc. > > > > Otis > > -- > > Monitoring - Log Management - Alerting - Anomaly Detection > > Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > > >

Re: Kafka Monitoring using JMX Mbeans

ng using Mbeans ?What kind of command line > > tools been used? > > > > See https://sematext.com/spm/integrations/kafka-monitoring/ > We use it for monitoring Kafka, ZooKeeper, Elasticsearch, Solr, etc. > > Otis > -- > Monitoring - Log Management - Alerting -

Re: Kafka Monitoring using JMX Mbeans

Hi, On Mon, Apr 25, 2016 at 4:14 AM, Mudit Kumar <mudit.ku...@askme.in> wrote: > Hi, > > Have anyone setup any monitoring using Mbeans ?What kind of command line > tools been used? > See https://sematext.com/spm/integrations/kafka-monitoring/ We use it for monito

RE: Kafka Monitoring using JMX Mbeans

: Re: Kafka Monitoring using JMX Mbeans We used a dockerized zabbix, one of the advantages of zabbix is that it has, jmx readout, creation of items, graphs, alerts in one product. Also how long to keep the history can be set for each item. The interface is not very intuitive though. On Mon, Apr

Re: Kafka Monitoring using JMX Mbeans

We used a dockerized zabbix, one of the advantages of zabbix is that it has, jmx readout, creation of items, graphs, alerts in one product. Also how long to keep the history can be set for each item. The interface is not very intuitive though. On Mon, Apr 25, 2016 at 10:14 AM Mudit Kumar

Kafka Monitoring using JMX Mbeans

Hi, Have anyone setup any monitoring using Mbeans ?What kind of command line tools been used? Thanks, Mudit

Re: Kafka Monitoring using JMX

Hi, Not sure what you are using to get the data out, but if you use SPM http://sematext.com/spm/integrations/kafka-monitoring.html you don't have to worry about any of that... assuming you use either 0.7.x or 0.8.2.x. Kafka metrics have changed drastically in the past and it looks like more

Re: Kafka Monitoring using JMX

Hi Naidu You'll need to escape the with a \ in the Mbean names. I've run across this too and it was al pain. It can get a bit tricky if you're doing it in code because you need to account for double escapes and so forth. This is a bug in the version of Metrics that Kafka is using. There is a

Re: Kafka Monitoring using JMX

You can use this https://github.com/Stackdriver/jmxtrans-config-stackdriver/blob/master/jmxtrans/stackdriver/json-specify-instance/kafka.json as an example of how Mbean are named and how is being escaped with \, and just use different output writer for any thing prior to 0.8.2.1 version. After

Kafka Monitoring using JMX

We are planning to monitor health of our Kafka environment using JMX. I have looked at below links to find what is available via https://cwiki.apache.org/confluence/display/KAFKA/Available+Metrics  https://cwiki.apache.org/confluence/display/KAFKA/JMX+Reporters why some of the kafka objects

Re: kafka monitoring

Manager, Database Operations Match.com 8300 Douglas Avenue I Suite 800 I Dallas, TX 75225 -Original Message- From: Sa Li [mailto:sal...@gmail.com] Sent: Thursday, January 08, 2015 1:09 PM To: users@kafka.apache.org Subject: kafka monitoring Hello, All I understand

Re: kafka monitoring

...@gmail.com] Sent: Thursday, January 08, 2015 1:09 PM To: users@kafka.apache.org Subject: kafka monitoring Hello, All I understand many of you are using jmxtrans along with graphite/ganglia to pull out metrics, according to https://kafka.apache.org/081/ops.html, it says The easiest way

Re: kafka monitoring

, January 08, 2015 1:09 PM To: users@kafka.apache.org Subject: kafka monitoring Hello, All I understand many of you are using jmxtrans along with graphite/ganglia to pull out metrics, according to https://kafka.apache.org/081/ops.html, it says The easiest way to see the available metrics

Re: kafka monitoring

Hi, you need to set jmx remote port. you can set this by executing below line in terminal and start server. (or) add below line to kafka-run-class.sh and start server. export JMX_PORT= (jmx remote port) and connect jconsole by giving brokerip: On Fri, Jan 9, 2015 at 12:38 AM,

RE: kafka monitoring

- From: Sa Li [mailto:sal...@gmail.com] Sent: Thursday, January 08, 2015 1:09 PM To: users@kafka.apache.org Subject: kafka monitoring Hello, All I understand many of you are using jmxtrans along with graphite/ganglia to pull out metrics, according to https://kafka.apache.org/081/ops.html

Re: kafka monitoring

You need to export the JMX_PORT for kafka to use on startup before starting up /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop

Re: kafka monitoring

Hi Sa Li, You need to set environment variable $JMX_PORT to enable jmx while starting kafka. See to kafka-run-class.sh on how it is used. Then you can connect to hostname:jmxport using Jconsole. Thanks, Raja. On Thu, Jan 8, 2015 at 2:08 PM, Sa Li sal...@gmail.com wrote: Hello, All I

kafka monitoring system

Hi, all I am thinking to make a reliable monitoring system for our kafka production cluster. I read such from documents: Kafka uses Yammer Metrics for metrics reporting in both the server and the client. This can be configured to report stats using pluggable stats reporters to hook up to your

Re: kafka monitoring system

Hi Sa Li, Have a look at SPM for monitoring Kafka: http://sematext.com/spm http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/ https://sematext.atlassian.net/wiki/display/PUBSPM/SPM+FAQ#SPMFAQ-KafkaMonitoring Otis -- Monitoring * Alerting * Anomaly Detection *

Re: kafka monitoring system

Hi Sa Li, You can try to use jmxtrans+opentsdb to monitor kafka. Jmxtrans is collecting data with JMX and sending to opentsdb. Opentsdb is graphing and alerting. YuanJia Li From: Sa Li Date: 2014-12-23 08:41 To: users Subject: kafka monitoring system Hi, all I am thinking to make

Re: kafka monitoring system

data with JMX and sending to opentsdb. Opentsdb is graphing and alerting. YuanJia Li From: Sa Li Date: 2014-12-23 08:41 To: users Subject: kafka monitoring system Hi, all I am thinking to make a reliable monitoring system for our kafka production cluster. I read such from

Re: kafka monitoring system

-12-23 08:41 To: users Subject: kafka monitoring system Hi, all I am thinking to make a reliable monitoring system for our kafka production cluster. I read such from documents: Kafka uses Yammer Metrics for metrics reporting in both the server and the client. This can

kafka monitoring

I see this MBean: kafka.server:name=AllTopicsMessagesInPerSec,type=BrokerTopicMetrics Does it return number per broker or per cluster? If it's per broker how to get global value per cluster and vice versa? Thanks.

Re: kafka monitoring

This is per broker. What we do is use JMXtrans (https://github.com/jmxtrans/jmxtrans) to pull this data into statsd. To get all messages into the cluster at once, we sum over all the brokers in Graphite. On Thu, Oct 24, 2013 at 4:49 AM, Kane Kane kane.ist...@gmail.com wrote: I see this MBean:

Re: kafka monitoring

It is perBroker. It gives the count of messages that a broker has. As per my understanding to get it at cluster level,the messages count at each broker needs to be summed up. On Thu, Oct 24, 2013 at 2:19 PM, Kane Kane kane.ist...@gmail.com wrote: I see this MBean:

Re: Kafka Monitoring

Thanks a lot Jun. This is very helpful. Thanks, Raja. On Thu, Sep 5, 2013 at 1:12 AM, Jun Rao jun...@gmail.com wrote: Updated the doc at http://kafka.apache.org/documentation.html#monitoring Hopefully that answers your questions. Thanks, Jun On Tue, Sep 3, 2013 at 11:16 PM, Vadim

Kafka Monitoring

Good evening. I have read through section of monitoring. I tried to map each section to corresponding JMX attribute. I will appreciate if you answer a few questions bellow. Thanks so much in advance, Vadim What this JMX kafka.controller:type=KafkaController,name=ActiveControllerCount for?

Re: Kafka Monitoring, 0.7 vs. 0.8 JMX

0.8 JMX is different from 0.7. In 0.8, all jmx beans are exposed through metrics. One can attach a metric reporter for monitoring. Thanks, Jun On Tue, May 7, 2013 at 1:45 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, We're considering adding Kafka monitoring to SPM (see http

Re: Kafka Monitoring, 0.7 vs. 0.8 JMX

wrote: 0.8 JMX is different from 0.7. In 0.8, all jmx beans are exposed through metrics. One can attach a metric reporter for monitoring. Thanks, Jun On Tue, May 7, 2013 at 1:45 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, We're considering adding Kafka

Re: Kafka Monitoring, 0.7 vs. 0.8 JMX

  From: Jun Rao jun...@gmail.com To: users@kafka.apache.org users@kafka.apache.org Sent: Wednesday, May 8, 2013 11:20 AM Subject: Re: Kafka Monitoring, 0.7 vs. 0.8 JMX 0.8 JMX is different from 0.7. In 0.8, all jmx beans are exposed through metrics. One can attach

Re: Kafka Monitoring, 0.7 vs. 0.8 JMX

From the JmxReporter section of the metrics manual: Warning We don¹t recommend that you try to gather metrics from your production environment. JMX¹s RPC API is fragile and bonkers. For development purposes and browsing, though, it can be very useful. -Dragos On 5/8/13 2:10 PM, Otis

Re: Kafka Monitoring, 0.7 vs. 0.8 JMX

11:20 AM Subject: Re: Kafka Monitoring, 0.7 vs. 0.8 JMX 0.8 JMX is different from 0.7. In 0.8, all jmx beans are exposed through metrics. One can attach a metric reporter for monitoring. Thanks, Jun On Tue, May 7, 2013 at 1:45 PM, Otis Gospodnetic otis.gospodne...@gmail.com

Kafka Monitoring, 0.7 vs. 0.8 JMX

Hi, We're considering adding Kafka monitoring to SPM (see http://sematext.com/spm/index.html ). We use Kafka ourselves and would like to see our Kafka metrics in SPM along with Hadoop, HBase, Solr, and other metrics we monitor. My questions: * What do people currently use for monitoring Kafka