-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50966/
-----------------------------------------------------------

(Updated Aug. 11, 2016, 10:30 p.m.)


Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.


Bugs: AMBARI-18093
    https://issues.apache.org/jira/browse/AMBARI-18093


Repository: ambari


Description (updated)
-------

Any hadoop metric that need not be aggregated across hosts can specify a 
skipAggregation = true flag in its Tags (Metadata). This metadata is captured 
by the AMS metadata and used to make a decision whether to aggregate a metric 
across hosts or not. 

Went with a "skip aggregation of metric" technique rather than a "don't even 
read metrics for aggregation that have skipAggregation = true" since the number 
of metrics can become quite large, and a METRIC_NAME NOT IN clause might become 
an issue.

As an example, used this 'skipAggregation' flag in the AMS controlled "per 
disk" metrics like disk_1_read_bytes, disk_2_read_count etc.

List of issues fixed

1. Skip Aggregation of certain metrics specified through metadata or configs.
2. Fix issue in METRIC_NAME where clause in DefaultCondition that causes 
mismatched metricNames to positions during PreparedStatement generation.

Sample 2min aggregator query before patch.
SELECT /+ NATIVE_TIME_RANGE(1470942120000) / METRIC_NAME, HOSTNAME, APP_ID, 
INSTANCE_ID, SERVER_TIME, START_TIME, UNITS, METRIC_SUM, METRIC_MAX, 
METRIC_MIN, METRIC_COUNT, 
METRICS  FROM METRIC_RECORD WHERE (METRIC_NAME NOT IN ('?') AND METRIC_NAME NOT 
LIKE '?' AND METRIC_NAME NOT LIKE '?') AND SERVER_TIME >= 1470942150000 AND 
SERVER_TIME < 1470942360000 ORDER BY METRIC_NAME, HOSTNAME, APP_ID, SERVER_TIME

0 = {String@6887} "disk_ _%" 
1 = {String@6888} "disk_ _ _%"
2 = {String@6889} "boottime"
3 = {Long@6890} "1470942150000"
4 = {Long@6891} "1470942360000"

Note that IN Clause will be given the value "disk_ _%" and LIKE clause will be 
given values - "disk_ _ _%" and "boottime" incorrectly.

3. Specify actual disk names in the metric name, instead of a number for 
individual disk metrics.
4. Fix issue in AMS embedded mode startup.


Diffs
-----

  
ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/TimelineMetric.java
 98f4978 
  
ambari-metrics/ambari-metrics-hadoop-sink/src/main/java/org/apache/hadoop/metrics2/sink/timeline/HadoopTimelineMetricsSink.java
 c534121 
  
ambari-metrics/ambari-metrics-hadoop-sink/src/test/java/org/apache/hadoop/metrics2/sink/timeline/HadoopTimelineMetricsSinkTest.java
 ea7f72d 
  
ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/host_info.py 
845b270 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java
 0f98db2 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricConfiguration.java
 1da68ba 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricClusterAggregatorSecond.java
 6731eb3 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TimelineMetricMetadataManager.java
 fd471fb 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/Condition.java
 4873c24 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultCondition.java
 0851e8f 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/EmptyCondition.java
 34174e2 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/SplitByMetricNamesCondition.java
 b8ca599 
  
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TestPhoenixTransactSQL.java
 a95655d 
  
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricClusterAggregatorSecondTest.java
 0f93bab 
  
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultConditionTest.java
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
 871e571 

Diff: https://reviews.apache.org/r/50966/diff/


Testing
-------

Manually tested.


Thanks,

Aravindan Vijayan

Reply via email to