oktay tuncay created AMBARI-25101:
-------------------------------------
Summary: HBase metrics not available
Key: AMBARI-25101
URL: https://issues.apache.org/jira/browse/AMBARI-25101
Project: Ambari
Issue Type: Bug
Components: ambari-metrics
Affects Versions: 2.7.1
Reporter: oktay tuncay
Attachments: AbstractTimelineMetricsSink-diff.png
TL;DR: I believe this is a bug introduced by the change of the class
responsible for fetching the available AMS Collectors. This bug occurs when
using more than one AMS Collector, details below.
-------------------------------------------------------------------------
This issue affects all metrics that use
org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.
After increasing the verbosity of loggin to DEBUG for the class
org.apache.hadoop.metrics2,*, it is possible to see that the metrics are not
being sent due a malformed URI in the connection string:
The malformed URI is composed by both AMS Collectors adresses separated by
comma, so I removed one of them as a test and restarted the services. After
this no more errors were shown in the logs and the metrics were sent to the AMS
DB (they weren't present before). Also, the dashboards started to show some
graphs.
Since this configuration was working in the previous versions (HDP 2.6 + Ambari
2.5) and, according to documentation, is still supported[1] with the new
versions (HDP 3.0 + Ambari 2.7) I checked the code of
HadoopTimelineMetricsSink[2] to verify what could be causing this. By analyzing
the code and correlating with the error messages in the log, we can see that
the connection process is handled by another class: AbstractTimelineMetricsSink.
The AbstractTimelineMetricsSink had modifications from Ambari 2.5[3] to Ambari
2.7[4]. The following snippet highlights the main differences in the connection
handling methods used by HadoopTimelineMetricsSink.
Based on that, I believe this is a bug introduced by the change of
AbstractTimelinetMetricsSink class. This bug occurs when using more than one
AMS Collector.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
