There is no apparent issue in Kafka.Its running fine and data is flowing in and
out without any issue.
My topic partitions are all in-sync also.I think its more likely an issue in
extracting the metrics out of the metrics store, but I am not so familiar with
the architecture.
[jthakrar@dtord01flm01p ~]$ /usr/hdp/current/kafka-broker/bin/kafka-topics.sh
--zookeeper dtord01flm01p --describeTopic:DmmAccess_v12 PartitionCount:5
ReplicationFactor:3 Configs: Topic: DmmAccess_v12 Partition:
0 Leader: 8 Replicas: 8,15,16 Isr: 15,16,8 Topic:
DmmAccess_v12 Partition: 1 Leader: 2 Replicas: 2,16,9 Isr:
16,9,2 Topic: DmmAccess_v12 Partition: 2 Leader: 3 Replicas:
3,14,10 Isr: 3,14,10 Topic: DmmAccess_v12 Partition: 3
Leader: 4 Replicas: 4,18,21 Isr: 21,18,4 Topic:
DmmAccess_v12 Partition: 4 Leader: 5 Replicas: 5,19,20 Isr:
19,5,20Topic:slimlogs_v1 PartitionCount:3 ReplicationFactor:3
Configs: Topic: slimlogs_v1 Partition: 0 Leader: 12
Replicas: 12,2,3 Isr: 12,3,2 Topic: slimlogs_v1 Partition: 1
Leader: 14 Replicas: 14,3,4 Isr: 14,3,4 Topic:
slimlogs_v1 Partition: 2 Leader: 15 Replicas: 15,4,5 Isr:
15,4,5
From: Siddharth Wagle <[email protected]>
To: "[email protected]" <[email protected]>; Jayesh Thakrar
<[email protected]>
Cc: Sriharsha Chintalapani <[email protected]>
Sent: Wednesday, May 6, 2015 10:30 PM
Subject: Re: Kafka broker metrics not appearing in REST API
#yiv4123801269 #yiv4123801269 --P{margin-top:0;margin-bottom:0;}#yiv4123801269
--P {margin-top:0;margin-bottom:0;}#yiv4123801269 Hi Jayesh,
There is little to go by as far as debugging info here since based on what you
observed host and cluster metrics are being reported.
Could you take a look at Kafka logs and check for exception if any?
BR,Sid
From: Jayesh Thakrar <[email protected]>
Sent: Wednesday, May 06, 2015 8:07 PM
To: Siddharth Wagle; [email protected]
Subject: Re: Kafka broker metrics not appearing in REST API Hi Siddharth,
Yes, I am using Ambari 2.0 with Ambari Metrics service.The interesting thing is
that I got them for some time and not anymore.And I also know that the metrics
are being collected since i can see them on the dashboard.Any pointer for
troubleshooting?
And btw, it would be nice to have a count of messages received and not a
computed metric count / min.TSDB does a good job of giving me cumulative and
rate-per-sec graphs and numbers.
Thanks in advance,Jayesh
From: Siddharth Wagle <[email protected]>
To: "[email protected]" <[email protected]>; Jayesh Thakrar
<[email protected]>
Sent: Wednesday, May 6, 2015 10:03 PM
Subject: Re: Kafka broker metrics not appearing in REST API
#yiv4123801269 #yiv4123801269 -- --P
{margin-top:0;margin-bottom:0;}#yiv4123801269 Hi Jayesh,
Are you using Ambari 2.0 with Ambari Metrics service?
BR,Sid
From: Jayesh Thakrar <[email protected]>
Sent: Wednesday, May 06, 2015 7:53 PM
To: [email protected]
Subject: Kafka broker metrics not appearing in REST API Hi,
I have installed 2 clusters with Ambari and Storm and Kafka.After the install,
I was able to get metrics for both Storm and Kafka via REST API.This worked
fine for a week, but since the past 2 days, I have not been getting Kafka
metrics.
I need the metrics to push to an OpenTSDB cluster.I do get host metrics and
Nimbus metrics but not KAFKA_BROKER metrics.
I did have maintenance turned on for some time, but maintenance is turned off
now.
[jthakrar@dtord01hdp0101d ~]$ curl --user admin:admin
'http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/NIMBUS?fields=metrics'{
"href" :
"http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/NIMBUS?fields=metrics";,
"ServiceComponentInfo" : { "cluster_name" : "ord_flume_kafka_prod",
"component_name" : "NIMBUS", "service_name" : "STORM" }, "metrics" : {
"storm" : { "nimbus" : { "freeslots" : 54.0, "supervisors" :
27.0, "topologies" : 0.0, "totalexecutors" : 0.0,
"totalslots" : 54.0, "totaltasks" : 0.0, "usedslots" : 0.0 }
} }}
[jthakrar@dtord01hdp0101d ~]$ curl --user admin:admin
'http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/KAFKA_BROKER?fields=metrics'{
"href" :
"http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/KAFKA_BROKER?fields=metrics";,
"ServiceComponentInfo" : { "cluster_name" : "ord_flume_kafka_prod",
"component_name" : "KAFKA_BROKER", "service_name" : "KAFKA" }}
[jthakrar@dtord01hdp0101d ~]$ curl --user admin:admin
'http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/SUPERVISOR?fields=metrics'{
"href" :
"http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/SUPERVISOR?fields=metrics";,
"ServiceComponentInfo" : { "cluster_name" : "ord_flume_kafka_prod",
"component_name" : "SUPERVISOR", "service_name" : "STORM" }
