[
https://issues.apache.org/jira/browse/AMBARI-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15904958#comment-15904958
]
Chuan Jin commented on AMBARI-20392:
------------------------------------
In order to solve my problem, the metrics send to the collector are separated
into several groups. Each metric group will make a single query to HBase table,
the results returned are combined together as response.
{code:title=PhoenixHBaseAccessor}
List<String> metricNames = condition.getMetricNames();
Map<String,List<String>> metricGroup= new HashMap<>();
for(String metric: metricNames){
String type = metric.split("[_.]")[0];
if(!metricGroup.containsKey(type)){
metricGroup.put(type,new ArrayList<String>());
}
metricGroup.get(type).add(metric);
}
for(List<String> metric : metricGroup.values()){
condition.setMetricNames(metric);
stmt = PhoenixTransactSQL.prepareGetAggregateSqlStmt(conn, condition);
....
}
{code}
> Get aggregate metric records from HBase encounters performance issues
> ---------------------------------------------------------------------
>
> Key: AMBARI-20392
> URL: https://issues.apache.org/jira/browse/AMBARI-20392
> Project: Ambari
> Issue Type: Improvement
> Components: ambari-metrics
> Affects Versions: 2.4.2
> Reporter: Chuan Jin
>
> I have a mini cluster ( ~6 nodes) managed by Ambari, and use a distributed
> HBase (~3 nodes) to hold metrics collected from these nodes. After I deploy
> YARN serivce, then I notice that some widgets (Cluster Memory,Cluster
> Disk,...) cannot display properly in the YARN service dashboard page. And
> Ambari Server has continuous timeout exceptions, which complains that it
> doesn't get timeline metrics for connection refused.
> The request timeout parameter is 5s, which means the query of getting metrics
> from HBase takes more time than that. Then I use Phoenix shell to login and
> perform the same query in the HBase , and it takes nearly 30s to finish. But
> If I split the big query into small pieces , i mean, use less values in the
> "metric_name" field in the where ... in clause , then the result return in 1s
> after several small queries.
> The query performance in HBase is highly based on the design of rowkey and
> the proper usage for it. In the method of getting aggregate metrics, AMS
> collector query the METRIC_AGGREGATE table in a way that may cause the
> co-processor to scan several regions across different RS. If we add more
> metrics in the service dashboard, this situation will be worse.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
