Since you say Metric collector crashed, one possibility is that there less physical memory on the metric collector host than what is configured for hbase.
Memory settings are present in, ams-hbase-env. -Sid Sent by Outlook<http://taps.io/outlookmobile> for Android On Wed, May 6, 2015 at 8:43 PM -0700, "Jayesh Thakrar" <[email protected]<mailto:[email protected]>> wrote: There is no apparent issue in Kafka. Its running fine and data is flowing in and out without any issue. My topic partitions are all in-sync also. I think its more likely an issue in extracting the metrics out of the metrics store, but I am not so familiar with the architecture. [jthakrar@dtord01flm01p ~]$ /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper dtord01flm01p --describe Topic:DmmAccess_v12 PartitionCount:5 ReplicationFactor:3 Configs: Topic: DmmAccess_v12 Partition: 0 Leader: 8 Replicas: 8,15,16 Isr: 15,16,8 Topic: DmmAccess_v12 Partition: 1 Leader: 2 Replicas: 2,16,9 Isr: 16,9,2 Topic: DmmAccess_v12 Partition: 2 Leader: 3 Replicas: 3,14,10 Isr: 3,14,10 Topic: DmmAccess_v12 Partition: 3 Leader: 4 Replicas: 4,18,21 Isr: 21,18,4 Topic: DmmAccess_v12 Partition: 4 Leader: 5 Replicas: 5,19,20 Isr: 19,5,20 Topic:slimlogs_v1 PartitionCount:3 ReplicationFactor:3 Configs: Topic: slimlogs_v1 Partition: 0 Leader: 12 Replicas: 12,2,3 Isr: 12,3,2 Topic: slimlogs_v1 Partition: 1 Leader: 14 Replicas: 14,3,4 Isr: 14,3,4 Topic: slimlogs_v1 Partition: 2 Leader: 15 Replicas: 15,4,5 Isr: 15,4,5 ________________________________ From: Siddharth Wagle <[email protected]> To: "[email protected]" <[email protected]>; Jayesh Thakrar <[email protected]> Cc: Sriharsha Chintalapani <[email protected]> Sent: Wednesday, May 6, 2015 10:30 PM Subject: Re: Kafka broker metrics not appearing in REST API Hi Jayesh, There is little to go by as far as debugging info here since based on what you observed host and cluster metrics are being reported. Could you take a look at Kafka logs and check for exception if any? BR, Sid ________________________________ From: Jayesh Thakrar <[email protected]> Sent: Wednesday, May 06, 2015 8:07 PM To: Siddharth Wagle; [email protected] Subject: Re: Kafka broker metrics not appearing in REST API Hi Siddharth, Yes, I am using Ambari 2.0 with Ambari Metrics service. The interesting thing is that I got them for some time and not anymore. And I also know that the metrics are being collected since i can see them on the dashboard. Any pointer for troubleshooting? And btw, it would be nice to have a count of messages received and not a computed metric count / min. TSDB does a good job of giving me cumulative and rate-per-sec graphs and numbers. Thanks in advance, Jayesh ________________________________ From: Siddharth Wagle <[email protected]> To: "[email protected]" <[email protected]>; Jayesh Thakrar <[email protected]> Sent: Wednesday, May 6, 2015 10:03 PM Subject: Re: Kafka broker metrics not appearing in REST API Hi Jayesh, Are you using Ambari 2.0 with Ambari Metrics service? BR, Sid ________________________________ From: Jayesh Thakrar <[email protected]> Sent: Wednesday, May 06, 2015 7:53 PM To: [email protected] Subject: Kafka broker metrics not appearing in REST API Hi, I have installed 2 clusters with Ambari and Storm and Kafka. After the install, I was able to get metrics for both Storm and Kafka via REST API. This worked fine for a week, but since the past 2 days, I have not been getting Kafka metrics. I need the metrics to push to an OpenTSDB cluster. I do get host metrics and Nimbus metrics but not KAFKA_BROKER metrics. I did have maintenance turned on for some time, but maintenance is turned off now. [jthakrar@dtord01hdp0101d ~]$ curl --user admin:admin 'http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/NIMBUS?fields=metrics' { "href" : "http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/NIMBUS?fields=metrics";, "ServiceComponentInfo" : { "cluster_name" : "ord_flume_kafka_prod", "component_name" : "NIMBUS", "service_name" : "STORM" }, "metrics" : { "storm" : { "nimbus" : { "freeslots" : 54.0, "supervisors" : 27.0, "topologies" : 0.0, "totalexecutors" : 0.0, "totalslots" : 54.0, "totaltasks" : 0.0, "usedslots" : 0.0 } } } } [jthakrar@dtord01hdp0101d ~]$ curl --user admin:admin 'http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/KAFKA_BROKER?fields=metrics' { "href" : "http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/KAFKA_BROKER?fields=metrics";, "ServiceComponentInfo" : { "cluster_name" : "ord_flume_kafka_prod", "component_name" : "KAFKA_BROKER", "service_name" : "KAFKA" } } [jthakrar@dtord01hdp0101d ~]$ curl --user admin:admin 'http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/SUPERVISOR?fields=metrics' { "href" : "http://dtord01flm01p:8080/api/v1/clusters/ord_flume_kafka_prod/components/SUPERVISOR?fields=metrics";, "ServiceComponentInfo" : { "cluster_name" : "ord_flume_kafka_prod", "component_name" : "SUPERVISOR", "service_name" : "STORM" }
