> On June 9, 2017, 11:13 p.m., Sid Wagle wrote: > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TimelineMetricMetadataManager.java > > Lines 436 (patched) > > <https://reviews.apache.org/r/59951/diff/1/?file=1746901#file1746901line436> > > > > Why this check and why no else / exception?
Based on whether the passed in UUID has the hostname portion (20 bytes if read from host data tables) or not (16 bytes if read from aggregate table), we need to choose the correct subarray to get TimelineMetricMetadataKey. > On June 9, 2017, 11:13 p.m., Sid Wagle wrote: > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TimelineMetricMetadataManager.java > > Lines 482 (patched) > > <https://reviews.apache.org/r/59951/diff/1/?file=1746901#file1746901line482> > > > > Would it make sense to do a lookup for this, satizedMetricsCache for > > fast lookup. I believe that we have to loop around the set of metric names in the GET request. The only expensive operation is if the metric name has wildcard (%) and for that we cannot store pre-computed maps, since the metric names that it resolves might change over time. > On June 9, 2017, 11:13 p.m., Sid Wagle wrote: > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/source/cache/InternalMetricsCacheSizeOfEngine.java > > Line 26 (original), 26 (patched) > > <https://reviews.apache.org/r/59951/diff/1/?file=1746910#file1746910line26> > > > > This might conflict. Will remove this change. > On June 9, 2017, 11:13 p.m., Sid Wagle wrote: > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/uuid/HashBasedUuidGenStrategy.java > > Lines 39 (patched) > > <https://reviews.apache.org/r/59951/diff/1/?file=1746911#file1746911line39> > > > > Please add comment to exaplin the algorithm. Will do. > On June 9, 2017, 11:13 p.m., Sid Wagle wrote: > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/uuid/HashBasedUuidGenStrategy.java > > Lines 138 (patched) > > <https://reviews.apache.org/r/59951/diff/1/?file=1746911#file1746911line138> > > > > Any reason not to use a stemming library here? This is only removing > > special chars. Wouldn't it increase entropy? Stemming library does not support this explicitly. Also, the stem function is quite simple and specific for us. - Aravindan ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59951/#review177514 ----------------------------------------------------------- On June 9, 2017, 6:52 p.m., Aravindan Vijayan wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/59951/ > ----------------------------------------------------------- > > (Updated June 9, 2017, 6:52 p.m.) > > > Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle. > > > Bugs: AMBARI-21214 > https://issues.apache.org/jira/browse/AMBARI-21214 > > > Repository: ambari > > > Description > ------- > > Presently AMS stores (metricName, hostname, serverTime, appId, instanceId) > for every row. Although this makes SQL query easy to write, it adds > duplication when storing it in the Regions memory. The plan is to have a > unique integer ID for an application-instance metric on a host and use that > as a lookup table with a write through cache when inserting or reading data > in the METRIC_RECORD table. > > WORK DONE > Added new logic for UUID management > Integrated the UUID management into metadata manager. > Added UUID handling into write, read and aggregate paths. > > TODO > Fix downsampling use case for UUID change. > > > Diffs > ----- > > > ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/TimelineMetric.java > edace52 > > ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/TimelineMetricMetadata.java > 727becc > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricsService.java > f962f44 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessor.java > 15b0bb8 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricConfiguration.java > de33bd1 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStore.java > d052d54 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricsFilter.java > 1446ec2 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java > cb131d3 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineClusterMetric.java > b7d9110 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricAggregatorFactory.java > 2eb3553 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricAppAggregator.java > 9eaf456 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricClusterAggregator.java > 74d4013 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricClusterAggregatorSecond.java > 34b1f9b > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricHostAggregator.java > a17433b > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricReadHelper.java > 672f85f > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TimelineMetricHostMetadata.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TimelineMetricMetadataKey.java > 504b502 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TimelineMetricMetadataManager.java > 8a71756 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TimelineMetricMetadataSync.java > 6d519f6 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/Condition.java > 9aa64bd > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/ConditionBuilder.java > 32c1e84 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultCondition.java > a4f7014 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/EmptyCondition.java > 43ab88c > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java > d39230d > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/SplitByMetricNamesCondition.java > bb4dced > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/TopNCondition.java > 0f2a02c > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/source/cache/InternalMetricsCacheSizeOfEngine.java > 071dcd4 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/uuid/HashBasedUuidGenStrategy.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/uuid/MetricUuidGenStrategy.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/uuid/RandomUuidGenStrategy.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TimelineWebServices.java > 50cfb08 > > ambari-metrics/ambari-metrics-timelineservice/src/main/resources/metrics_def/AMBARI_SERVER.dat > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/resources/metrics_def/JOBHISTORYSERVER.dat > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/resources/metrics_def/MASTER_HBASE.dat > 9ba90f1 > > ambari-metrics/ambari-metrics-timelineservice/src/main/resources/metrics_def/SLAVE_HBASE.dat > 38b870f > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java > 41ddef5 > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/ITPhoenixHBaseAccessor.java > f6d69f6 > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/MetricTestHelper.java > 7eeb9c4 > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessorTest.java > bf9246d > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TestPhoenixTransactSQL.java > e988a61 > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TestTimelineMetricStore.java > ac2f9d7 > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricsAggregatorMemorySink.java > 53f6f6c > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerTest.java > a0fea40 > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/ITClusterAggregator.java > 86c9b40 > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/ITMetricAggregator.java > 75b3f91 > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricClusterAggregatorSecondTest.java > 6541b2c > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TestMetadataManager.java > 3adf770 > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TestMetadataSync.java > a524b13 > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/uuid/TimelineMetricUuidManagerTest.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/test/resources/test_data/full_whitelist.dat > PRE-CREATION > > > Diff: https://reviews.apache.org/r/59951/diff/1/ > > > Testing > ------- > > Manually tested on a cluster with all services sending metrics to AMS. > mvn clean test on amabri-metrics. > > [INFO] ambari-metrics ..................................... SUCCESS [ 1.230 > s] > [INFO] Ambari Metrics Common .............................. SUCCESS [ 4.732 > s] > [INFO] Ambari Metrics Hadoop Sink ......................... SUCCESS [ 5.545 > s] > [INFO] Ambari Metrics Flume Sink .......................... SUCCESS [ 4.121 > s] > [INFO] Ambari Metrics Kafka Sink .......................... SUCCESS [ 3.793 > s] > [INFO] Ambari Metrics Storm Sink .......................... SUCCESS [ 1.735 > s] > [INFO] Ambari Metrics Storm Sink (Legacy) ................. SUCCESS [ 1.924 > s] > [INFO] Ambari Metrics Alert Service ....................... SUCCESS [ 1.909 > s] > [INFO] Ambari Metrics Collector ........................... SUCCESS [05:43 > min] > [INFO] Ambari Metrics Monitor ............................. SUCCESS [ 5.014 > s] > [INFO] Ambari Metrics Grafana ............................. SUCCESS [ 6.102 > s] > [INFO] Ambari Metrics Host Aggregator ..................... SUCCESS [ 0.236 > s] > [INFO] Ambari Metrics Assembly ............................ SUCCESS [ 19.852 > s] > [INFO] ambari-metrics-spark ............................... SUCCESS [ 12.180 > s] > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 06:55 min > [INFO] Finished at: 2017-06-09T11:40:33-07:00 > [INFO] Final Memory: 134M/1333M > [INFO] > ------------------------------------------------------------------------ > > > Thanks, > > Aravindan Vijayan > >
