Hi All
Thank you for your time and effort. All have suggested my to look at Prometheus
etc... I am not looking for a solution. I need metrics on which I can do
alerting. We are implementing custom alerting mechanism. I am using JMX to get
all the metrics but as many of you suggested I do not need all of them. I need
very few to alert and I need details about that metrics on which I can create
alerts.
Thanks in advance
ThanksArunkumar Pichaimuthu, PMP
On Wednesday, April 25, 2018, 11:35:27 PM CDT, Anand, Uttam
<uttam.an...@bnsf.com> wrote:
You can use Prometheus for monitoring , AlertManager for alerting and Grafana
to view the real time graphs by using the metrics scraped by Prometheus.
For gathering the metrics of Kafka and Zookeeper , you can use jmx exporter.
Also, all metrics will not be useful for humans to read and understand and it
will be overhead for monitoring apps to process them. So you can whitelist the
metrics you actually want.
https://www.robustperception.io/monitoring-kafka-with-prometheus
You can add jmx exporter as environment variable in container also.
<https://www.robustperception.io/monitoring-kafka-with-prometheus>
Monitoring Kafka with Prometheus | Robust
Perception<https://www.robustperception.io/monitoring-kafka-with-prometheus/>
www.robustperception.io
We’ve previously looked at how to monitor Cassandra with Prometheus. Let’s see
the process for getting metrics from another popular Java application, Kafka.
Similar to what we did for Cassandra, we download Kafka, the JMX exporter and
the config file:
________________________________
From: Will Weber <rwawe...@gmail.com>
Sent: Wednesday, April 25, 2018 11:27:40 PM
To: users@kafka.apache.org
Subject: Re: Monitoring and Alerting
<center><p style="color:red; font-size:24px;"><b>EXTERNAL EMAIL</b></p></center>
"Thirding" the use of prometheus, recommend reading this blog post as well:
https://www.robustperception.io/monitoring-kafka-with-prometheus
On Wed, Apr 25, 2018 at 11:39 PM, Jonathan Bethune <
jonathan.beth...@instaclustr.com> wrote:
> I also recommend Prometheus. Works great with JMX for Kafka or any Java
> service.
>
> Datadog is also fine if you want simple and don't spending money. Easy to
> integrate with alerting systems and great visualization.
>
> They have a good blogpost about integrating with Kafka and ZooKeeper:
> https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/
>
> On 26 April 2018 at 08:02, Stanislav Antic <stanislav.an...@gmail.com>
> wrote:
>
> > I recommend setting up Prometheus which have Zookeeper exporter (helper
> > program which gives metrics from outside software) and you can use JMX
> > exporter with Kafka.
> >
> > You can find example configs in their repo, which is pretty good and you
> > also have already done Grafana dashboards (https://grafana.com/
> dashboards
> > ):
> > https://github.com/prometheus/jmx_exporter/tree/master/example_configs
> >
> >
> > On Thu, Apr 26, 2018 at 12:05 AM, Arunkumar <pm_arunku...@yahoo.com.
> > invalid>
> > wrote:
> >
> > > HI All
> > > I am working on setting up Monitoring and alerting for our production
> > > cluster. As of now we have a cluster of 3 zookeeper and 3 kafka Brokers
> > > which will expand later.
> > > We are planning for basic metrics (important ones) on which we need to
> > > alert. We are in a process of developing alerting system for our
> > cluster. I
> > > googled and did not get much detail as needed.
> > > I see many say the following ones are important and I understand these
> > are
> > > aggregate metrics across cluster (I may not be
> right).kafka.server:type=
> > > ReplicaManager,name=UnderReplicatedPartitionskafka.controller:type=
> > > KafkaController,name=OfflinePartitionsCountkafka.controller:type=
> > > KafkaController,name=ActiveControllerCount
> > > But I am looking for broker,topic etc metrics to alert. Can any one
> > > provide info or point to any docs on this regards will be of great
> > > help. Thanks in advance
> > >
> > > ThanksArunkumar Pichaimuthu, PMP
> >
> >
> >
> >
> > --
> > Stanislav Antic
> >
>
>
>
> --
>
> *Jonathan Bethune - **Senior Consultant*
>
> JP: +81 70 4069 4357
>
> <https://www.facebook.com/instaclustr> <https://twitter.com/instaclustr>
> <https://www.linkedin.com/company/instaclustr>
>
> Read our latest technical blog posts here
> <https://www.instaclustr.com/blog/>. This email has been sent on behalf
> of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA). This
> email and any attachments may contain confidential and legally
> privileged information. If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>
