Re: Review Request 48030: AMBARI-16946 Storm Metrics Sink has high chance to discard some datapoints
> On 6 7, 2016, 6:07 오전, Sid Wagle wrote: > > Changes look good, only thing to consider is the changes to the metric > > name. Cluster Aggregation will not occur at topology level since appId = > > topologyName for metrics with the same metric name. Is the metric name to > > fine grained? Only thing to consider is whether we need task metrics to be > > aggregated across topology? If yes, taskId cannot be part of the metric > > name. > > > > Could you also add Aravindan Vijayan to the reviewers? Thanks. > > Sid Wagle wrote: > Rephrase: Cluster Aggregation will *now* occur at topology level > > Jungtaek Lim wrote: > > Only thing to consider is whether we need task metrics to be aggregated > across topology? If yes, taskId cannot be part of the metric name. > > It depends on users, but most cases I don't think user will aggregate > metrics across topologies. > > And like what I was saying, technically there're no way to aggregate > metrics on sink side since parallelism of sink can be set to more than 1 (I > mean, we could have multiple aggregation points which breaks aggregation.) > So we need to push task level metrics into AMS, and task id should be > included as metric name for making it unique. > > Based on that, we need `series aggregation` to aggregate task level > metrics by higher level. (I'm trying to address this via AMBARI-17027) > > The only thing which affects aggregation is appId. > > - When we set appId to 'component name' (current), same component (Spout, > Bolt) name across topologies can be queried together. > - When we set appId to 'topology name', same metric name (for Storm's > view) from different components in topology can be queried together. > - When we set appId to 'Storm', all metrics can be queried together. > (metric name should also include topology name as well) > > I'm not familiar with structure of metrics so I'm not sure how they > affects performance while storing / querying. So I'd like to hear opinions on > reviewers. > > For reference, below is how storm-graphite composes prefix of metric > name. It uses topology name, component name, worker host, worker port, task > id. > > https://github.com/verisign/storm-graphite/blob/master/src/main/java/com/verisign/storm/metrics/GraphiteMetricsConsumer.java#L278 > > When querying metrics from Graphite users can query with wildcards & > series function to aggregate metrics into one and Grafana can show that. > That's what I want to address to AMS. > > Aravindan Vijayan wrote: > Jungtaek, there is also another level of classification called > "instancedId" Every appId can have multiple instanceIds. You should find that > in the TimelineMetric object. Perhaps, we can use that here. > > Sriharsha Chintalapani wrote: > @Jungtaek what Sid was saying is we won't be able to see the aggregated > metrics for a single topology if we use taskId. Lets say if I want to see how > many tuples are processed in last hour for that topology that aggregation > won't be possible using taskId. @Aravindan Yeah, actually I was trying to use instanceId for the first time, but TimelineMetricCache (TimelineMetricHolder) only checks metric name as unique key while putting so that's what I described. Furthermore, I would want to aggregate task level metrics into component level metrics but there's no wildcard support on instanceId. So there seems to much effort to use instanceId for taskId, and having taskId as metric name is easier way to achieve aggregation for now. But I'm open to use instanceId, and support wildcard & aggregation on this. Please let me know we would want to use instanceId. @Sid @Sriharsha @Sriharsha I guess what Sid was saying is opposite to what you're saying. Let's pretend cluster aggregation is occurred. A. current appId is component name and metric name is just metric name so there's no distinction between topologies. (topology name is not placed anywhere now) So cluster aggregation will aggregate metrics across topologies, and we can't query last 1 hours of Topology T1, Bolt B1, Metric M1. There're no way to query within Topology T1 regardless of aggregation. B. after patch appId is topology name and metric name is component name + task id + metric name so metrics are classified as topology name. So cluster aggregation will aggregate metrics for each topology, and we can't query last 1 hours of all topologies, Bolt B1, Metric M1. There're no way to query across topologies regardless of aggregation. So if we want to have full flexible queries, topology name also should be placed to metric name as same as component name and task id. @Sid Pleas
Re: Review Request 48065: AMBARI-16949 Metrics Collector API shows NPE if we use wildcard (%25 for '%') for metric name
> On 6 7, 2016, 3:07 오후, Aravindan Vijayan wrote: > > Has this been checked in? If you need help doing that, I can do that for > > you. No it's not checked in yet. Please check this in. Thanks! - Jungtaek --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48065/#review136468 --- On 6 3, 2016, 12:43 오전, Jungtaek Lim wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48065/ > --- > > (Updated 6 3, 2016, 12:43 오전) > > > Review request for Ambari, Sriharsha Chintalapani and Sid Wagle. > > > Bugs: AMBARI-16949 > https://issues.apache.org/jira/browse/AMBARI-16949 > > > Repository: ambari > > > Description > --- > > When we request '/ws/v1/timeline/metrics' with metric name which contains %25 > (escaped '%'), response for the API is json describing there's NPE. > And NPE is logged for ambari-metrics-collector log file. > > ``` > 2016-05-30 09:15:05,061 WARN > org.apache.hadoop.yarn.webapp.GenericExceptionHandler: INTERNAL_SERVER_ERROR > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor.appendAggregateMetricFromResultSet(PhoenixHBaseAccessor.java:810) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor.getAggregateMetricRecords(PhoenixHBaseAccessor.java:772) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore.getTimelineMetrics(HBaseTimelineMetricStore.java:178) > at > org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices.getTimelineMetrics(TimelineWebServices.java:372) > at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) > ``` > > Reason of NPE: > > Metrics are properly fetched with wildcard. But when applying functions to > result set, actual metric name is not exist from map of metric name to list > of function. > > > Diffs > - > > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessor.java > 47962cb > > Diff: https://reviews.apache.org/r/48065/diff/ > > > Testing > --- > > Test failed from local but it's occurred from another modules, which is not > related to current modification. > > ``` > Results : > > Failed tests: > PrivilegeEventCreatorTest.putTest:107 expected:<...), Roles( > Permission[2: > Users: testuser2 > Permission1: > Users: testuser > Groups: testgroup])> but was:<...), Roles( > Permission[1: > Users: testuser > Groups: testgroup > Permission2: > Users: testuser2])> > RepositoryVersionEventCreatorTest.postTest:70 expected:<...ating system: > redhat[6 > Repository ID(2), Repository name(MyRepo6), Base url(http://example6.com) > Operating system: redhat7 > Repository ID(1), Repository name(MyRepo), Base url(http://example].com) > )> but was:<...ating system: redhat[7 > Repository ID(1), Repository name(MyRepo), Base url(http://example.com) > Operating system: redhat6 > Repository ID(2), Repository name(MyRepo6), Base url(http://example6].com) > )> > RepositoryVersionEventCreatorTest.putTest:100 expected:<...ating system: > redhat[6 > Repository ID(2), Repository name(MyRepo6), Base url(http://example6.com) > Operating system: redhat7 > Repository ID(1), Repository name(MyRepo), Base url(http://example].com) > )> but was:<...ating system: redhat[7 > Repository ID(1), Repository name(MyRepo), Base url(http://example.com) > Operating system: redhat6 > Repository ID(2), Repository name(MyRepo6), Base url(http://example6].com) > )> > ViewPrivilegeEventCreatorTest.putTest:85 expected:<...tatus(200 OK), > Type([MyView), Version(MyView), Name(MyView), Permissions( > Permission2: > Users: testuser2 > Permission1: > Users: testuser > Groups: testgroup])> but was:<...tatus(200 OK), Type([null), Version(null), > Name(null), Permissions( > Permission1: > Users: testuser > Groups: testgroup > Permission2: > Users: testuser2])> > > ComponentResourceProviderTest.testGetResourcesAsAdministrator:190->testGetResources:296 > expected:<[tru]e> but was:<
Review Request 48395: AMBARI-17027: Metrics Collector API: Introduce basic series aggregation functions
s] [INFO] Ambari Metrics Collector ... SUCCESS [04:12 min] [INFO] Ambari Metrics Monitor . SUCCESS [ 3.587 s] [INFO] Ambari Metrics Grafana . FAILURE [01:15 min] [INFO] Ambari Metrics Assembly SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 06:02 min [INFO] Finished at: 2016-06-08T14:48:58+09:00 [INFO] Final Memory: 79M/1149M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (default) on project ambari-metrics-grafana: An Ant BuildException has occured: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake [ERROR] around Ant part ...https://grafanarel.s3.amazonaws.com/builds/grafana-2.6.0.linux-x64.tar.gz; dest="/Users/jlim/WorkArea/JavaProjects/ambari/ambari-metrics/ambari-metrics-grafana/target/grafana/grafana.tgz"/>... @ 5:222 in /Users/jlim/WorkArea/JavaProjects/ambari/ambari-metrics/ambari-metrics-grafana/target/antrun/build-Download Ambari Grafana.xml: SSL peer shut down incorrectly ``` > mvn test -rf :ambari-metrics-grafana ``` [INFO] [INFO] Reactor Summary: [INFO] [INFO] Ambari Metrics Grafana . SUCCESS [ 38.966 s] [INFO] Ambari Metrics Assembly SUCCESS [ 3.855 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 43.037 s [INFO] Finished at: 2016-06-08T14:50:33+09:00 [INFO] Final Memory: 24M/300M [INFO] ---- ``` Thanks, Jungtaek Lim
Re: Review Request 48395: AMBARI-17027: Metrics Collector API: Introduce basic series aggregation functions
> On 6 10, 2016, 11:56 오후, Aravindan Vijayan wrote: > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java, > > line 257 > > <https://reviews.apache.org/r/48395/diff/1/?file=1410878#file1410878line257> > > > > Just throwing out a question here. > > > > Does it make any difference to do SUM(M1,M2) before or after applying > > post processing (rate/diff)? > > > > Ideally, these operations should be commutative (Order should not > > matter). > > > > Rate of Sum and Sum or Rate should be the same. If it is timestamp aligned, it is commutative for SUM. Let's say value of (M1, t1) to x1 and value of (M2, t1) to y1. t1 x1 y1 t2 x2 y2 t3 x3 y3 t4 x4 y4 If we calculate sum first, and rate, t1 = 0.0 t2 = ((x2 + y2) - (x1 + y1)) / (t2 - t1) = (x2 + y2 - x1 - y1) / (t2 - t1) t3 = ((x3 + y3) - (x2 + y2)) / (t3 - t2) = (x3 + y3 - x2 - y2) / (t3 - t2) t4 = ((x4 + y4) - (x3 + y3)) / (t4 - t3) = (x4 + y4 - x3 - y3) / (t4 - t3) If we calculate rate first, and sum, t1 = 0.0 t2 = ((x2 - x1) + (y2 - y1)) / (t2 - t1) = (x2 + y2 - x1 - y1) / (t2 - t1) t3 = ((x3 - x2) + (y3 - y2)) / (t3 - t2) = (x3 + y3 - x2 - y2) / (t3 - t2) t4 = ((x4 - x3) + (y4 - y3)) / (t4 - t3) = (x4 + y4 - x3 - y3) / (t4 - t3) So they're same. But if it is not timestamp aligned, it will be messed up. For example, t1 x1 y1 t2 y2 t3 x3 t4 x4 y4 If we calculate sum first, and rate, t1 = 0.0 t2 = (y2 - (x1 + y1)) / (t2 - t1) = (y2 - x1 - y1) / (t2 - t1) t3 = (x3 - y2) / (t3 - t2) = (x3 - y2) / (t3 - t2) t4 = (x4 + y4 - x3) / (t4 - t3) If we calculate rate first, and sum, t1 = 0.0 t2 = N/A + (y2 - y1) / (t2 - t1) = (y2 - y1) / (t2 - t1) t3 = (x3 - x1) / (t3 - t1) + N/A = (x3 - x1) / (t3 - t1) t4 = ((x4 - x3) / (t4 - t3)) + ((y4 - t2) / (t4 - t2)) So they're not same. I guess 'interpolation' seems important for this side. - Jungtaek --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48395/#review137086 ------- On 6 10, 2016, 12:50 오전, Jungtaek Lim wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48395/ > --- > > (Updated 6 10, 2016, 12:50 오전) > > > Review request for Ambari, Aravindan Vijayan, Dmytro Sen, Prajwal Rao, > Sriharsha Chintalapani, Sid Wagle, and Yusaku Sako. > > > Bugs: AMBARI-17027 > https://issues.apache.org/jira/browse/AMBARI-17027 > > > Repository: ambari > > > Description > --- > > AMS doesn't provide tag so metric is identified by appId, metric name, > hostname, instanceId. In this situation metric name is normally consist of > origin metric name and tag values, like graphite, but unlike Graphite, AMS > also doesn't provide series aggregation functions so aggregation should be > done from caller side. > > It would be great if Ambari Metrics Collector provides series aggregation > functions, like sumSeries / > averageSeries / minSeries / maxSeries on Graphite. > > Query outputs: > https://gist.github.com/HeartSaVioR/f4f28b5b8b7bf2e5477e59d7fd56090f > > Attached Grafana screenshots to AMBARI-17027. Please refer > https://issues.apache.org/jira/browse/AMBARI-17027 for details. > > > Diffs > - > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/datasource.js 7390aa8 > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/partials/query.editor.html > b034c03 > ambari-metrics/ambari-metrics-grafana/ambari-metrics/queryCtrl.js 2eb3613 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java > 1b2d02f > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStore.java > e37bc4d > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStoreWatcher.java > 7d49070 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/AbstractTimelineMetricsSeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/funct
Re: Review Request 48395: AMBARI-17027: Metrics Collector API: Introduce basic series aggregation functions
> On 6 10, 2016, 11:56 오후, Aravindan Vijayan wrote: > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/AbstractTimelineMetricsSeriesAggregateFunction.java, > > line 73 > > <https://reviews.apache.org/r/48395/diff/1/?file=1410881#file1410881line73> > > > > When the data is not timestamp aligned, this will lead to "difficult to > > comprehend" graphs. > > > > For example, let's take Sum of Metrics M1,M2,M3 where > > M1 - t1 - 5.0 > > M1 - t2 - 5.0 > > M2 - t2 - 5.0 > > M3 - t3 - 5.0 > > > > The graph will show up like > > (t1, 5.0) -> (t2,10.0) -> (t3,5.0) > > > > That is why we do some "interpolation" while we aggregate data. Yeah, I agree that output could be odd when data is not timestamp aligned. I didn't think about this side while working, and also I'm not expert on this. How AMS interpolate points when aggregating? Does it apply linear interpolation? - Jungtaek --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48395/#review137086 --- On 6 10, 2016, 12:50 오전, Jungtaek Lim wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48395/ > --- > > (Updated 6 10, 2016, 12:50 오전) > > > Review request for Ambari, Aravindan Vijayan, Dmytro Sen, Prajwal Rao, > Sriharsha Chintalapani, Sid Wagle, and Yusaku Sako. > > > Bugs: AMBARI-17027 > https://issues.apache.org/jira/browse/AMBARI-17027 > > > Repository: ambari > > > Description > --- > > AMS doesn't provide tag so metric is identified by appId, metric name, > hostname, instanceId. In this situation metric name is normally consist of > origin metric name and tag values, like graphite, but unlike Graphite, AMS > also doesn't provide series aggregation functions so aggregation should be > done from caller side. > > It would be great if Ambari Metrics Collector provides series aggregation > functions, like sumSeries / > averageSeries / minSeries / maxSeries on Graphite. > > Query outputs: > https://gist.github.com/HeartSaVioR/f4f28b5b8b7bf2e5477e59d7fd56090f > > Attached Grafana screenshots to AMBARI-17027. Please refer > https://issues.apache.org/jira/browse/AMBARI-17027 for details. > > > Diffs > - > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/datasource.js 7390aa8 > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/partials/query.editor.html > b034c03 > ambari-metrics/ambari-metrics-grafana/ambari-metrics/queryCtrl.js 2eb3613 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java > 1b2d02f > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStore.java > e37bc4d > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStoreWatcher.java > 7d49070 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/AbstractTimelineMetricsSeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/SeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunctionFactory.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAvgAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetric
Re: Review Request 48395: AMBARI-17027: Metrics Collector API: Introduce basic series aggregation functions
> On 6 9, 2016, 5:42 오후, Aravindan Vijayan wrote: > > Please add Dmitry Sen as well. Thanks > > Jungtaek Lim wrote: > Could you tell what's Dmitry's ID? Auto-complete can't find Dmitry Sen. > > Aravindan Vijayan wrote: > dsen I just found it. His name is registered as 'Dmytro Sen' so I couldn't find. - Jungtaek --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48395/#review136855 --- On 6 10, 2016, 12:50 오전, Jungtaek Lim wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48395/ > --- > > (Updated 6 10, 2016, 12:50 오전) > > > Review request for Ambari, Aravindan Vijayan, Dmytro Sen, Prajwal Rao, > Sriharsha Chintalapani, Sid Wagle, and Yusaku Sako. > > > Bugs: AMBARI-17027 > https://issues.apache.org/jira/browse/AMBARI-17027 > > > Repository: ambari > > > Description > --- > > AMS doesn't provide tag so metric is identified by appId, metric name, > hostname, instanceId. In this situation metric name is normally consist of > origin metric name and tag values, like graphite, but unlike Graphite, AMS > also doesn't provide series aggregation functions so aggregation should be > done from caller side. > > It would be great if Ambari Metrics Collector provides series aggregation > functions, like sumSeries / > averageSeries / minSeries / maxSeries on Graphite. > > Query outputs: > https://gist.github.com/HeartSaVioR/f4f28b5b8b7bf2e5477e59d7fd56090f > > Attached Grafana screenshots to AMBARI-17027. Please refer > https://issues.apache.org/jira/browse/AMBARI-17027 for details. > > > Diffs > - > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/datasource.js 7390aa8 > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/partials/query.editor.html > b034c03 > ambari-metrics/ambari-metrics-grafana/ambari-metrics/queryCtrl.js 2eb3613 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java > 1b2d02f > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStore.java > e37bc4d > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStoreWatcher.java > 7d49070 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/AbstractTimelineMetricsSeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/SeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunctionFactory.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAvgAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesMaxAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesMinAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesSumAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TimelineWebServices.java > ee3a097 > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TestTimelineMetricSto
Re: Review Request 48395: AMBARI-17027: Metrics Collector API: Introduce basic series aggregation functions
> On 6 9, 2016, 5:42 오후, Aravindan Vijayan wrote: > > Please add Dmitry Sen as well. Thanks Could you tell what's Dmitry's ID? Auto-complete can't find Dmitry Sen. - Jungtaek --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48395/#review136855 --- On 6 9, 2016, 4:46 오후, Jungtaek Lim wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48395/ > --- > > (Updated 6 9, 2016, 4:46 오후) > > > Review request for Ambari, Aravindan Vijayan, Prajwal Rao, Sriharsha > Chintalapani, Sid Wagle, and Yusaku Sako. > > > Bugs: AMBARI-17027 > https://issues.apache.org/jira/browse/AMBARI-17027 > > > Repository: ambari > > > Description > --- > > AMS doesn't provide tag so metric is identified by appId, metric name, > hostname, instanceId. In this situation metric name is normally consist of > origin metric name and tag values, like graphite, but unlike Graphite, AMS > also doesn't provide series aggregation functions so aggregation should be > done from caller side. > > It would be great if Ambari Metrics Collector provides series aggregation > functions, like sumSeries / > averageSeries / minSeries / maxSeries on Graphite. > > Query outputs: > https://gist.github.com/HeartSaVioR/f4f28b5b8b7bf2e5477e59d7fd56090f > > Attached Grafana screenshots to AMBARI-17027. Please refer > https://issues.apache.org/jira/browse/AMBARI-17027 for details. > > > Diffs > - > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/datasource.js 7390aa8 > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/partials/query.editor.html > b034c03 > ambari-metrics/ambari-metrics-grafana/ambari-metrics/queryCtrl.js 2eb3613 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java > 1b2d02f > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStore.java > e37bc4d > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStoreWatcher.java > 7d49070 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/AbstractTimelineMetricsSeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/SeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunctionFactory.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAvgAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesMaxAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesMinAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesSumAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TimelineWebServices.java > ee3a097 > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TestTimelineMetricStore.java > cfd1f58 > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/Ti
Re: Review Request 48030: AMBARI-16946 Storm Metrics Sink has high chance to discard some datapoints
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48030/ --- (Updated 6 9, 2016, 1:58 오전) Review request for Ambari, Aravindan Vijayan, Sriharsha Chintalapani, and Sid Wagle. Changes --- Addressing AMBARI-17133: Replace underscore character from Storm metrics sink. Since AMBARI-17133 should be built on top of AMBARI-16946 but it's tiny size I would like to let AMBARI-16946 containing AMBARI-17133. Bugs: AMBARI-16946 https://issues.apache.org/jira/browse/AMBARI-16946 Repository: ambari Description --- There's a mismatch between TimelineMetricsCache and Storm metrics unit, while TimelineMetricsCache considers "metric name + timestamp" to be unique but Storm is not. For example, assume that bolt B has task T1, T2 and B has registered metrics M1. It's possible for metrics sink to receive (T1, M1) and (T2, M1) with same timestamp TS1 (in TaskInfo, not current time), and received later will be discarded from TimelineMetricsCache. If we want to have unique metric point of Storm, we should use "topology name + component name + task id + metric name" to metric name so that "metric name + timestamp" will be unique. There're other issues I would like to address, too. - Currently, hostname is written to hostname of the machine which runs metrics sink. Since TaskInfo has hostname of the machine which runs task, we're better to use this. - Unit of timestamp of TaskInfo is second, while Storm Metrics Sink uses this as millisecond, resulting in timestamp flaw, and malfunction of cache eviction. It should be multiplied by 1000. - 'component name' is not unique across the cluster, so it's not fit for app id. 'topology name' is unique so proper value of app id is topology name. Consideration: Hostname for determining 'write shard' is set to hostname of the machine which runs metrics sink. Since I don't know read also be sharded, I'm not sure it's safe to use TaskInfo's hostname to hostname of TimelineMetric. Please review carefully regarding this. Diffs (updated) - ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java 02f5598 ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java 8171a4d Diff: https://reviews.apache.org/r/48030/diff/ Testing --- I tested this only ambari-metrics module since changeset is not related on other modules. [INFO] [INFO] Reactor Summary: [INFO] [INFO] ambari-metrics . SUCCESS [ 0.896 s] [INFO] Ambari Metrics Common .. SUCCESS [ 13.386 s] [INFO] Ambari Metrics Hadoop Sink . SUCCESS [ 5.085 s] [INFO] Ambari Metrics Flume Sink .. SUCCESS [ 6.827 s] [INFO] Ambari Metrics Kafka Sink .. SUCCESS [ 4.190 s] [INFO] Ambari Metrics Storm Sink .. SUCCESS [ 1.384 s] [INFO] Ambari Metrics Collector ... SUCCESS [04:06 min] [INFO] Ambari Metrics Monitor . SUCCESS [ 3.556 s] [INFO] Ambari Metrics Grafana . SUCCESS [01:03 min] [INFO] Ambari Metrics Assembly SUCCESS [ 3.567 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 05:48 min [INFO] Finished at: 2016-05-30T16:46:07+09:00 [INFO] Final Memory: 78M/1038M [INFO] In fact, I tried to run `mvn test` from ambari root directory but build is failing from ambari-web. ``` > fsevents@0.2.1 install > /Users/jlim/WorkArea/JavaProjects/ambari/ambari-web/node_modules/chokidar/node_modules/fsevents > node-gyp rebuild ... npm WARN install:fsevents fsevents@0.2.1 install: `node-gyp rebuild` npm WARN install:fsevents Exit status 1 ``` No luck on `npm install`, too. Thanks, Jungtaek Lim
Re: Review Request 48030: AMBARI-16946 Storm Metrics Sink has high chance to discard some datapoints
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48030/ --- (Updated 5 30, 2016, 8:21 오전) Review request for Ambari, Sriharsha Chintalapani and Sid Wagle. Bugs: AMBARI-16946 https://issues.apache.org/jira/browse/AMBARI-16946 Repository: ambari Description --- There's a mismatch between TimelineMetricsCache and Storm metrics unit, while TimelineMetricsCache considers "metric name + timestamp" to be unique but Storm is not. For example, assume that bolt B has task T1, T2 and B has registered metrics M1. It's possible for metrics sink to receive (T1, M1) and (T2, M1) with same timestamp TS1 (in TaskInfo, not current time), and received later will be discarded from TimelineMetricsCache. If we want to have unique metric point of Storm, we should use "topology name + component name + task id + metric name" to metric name so that "metric name + timestamp" will be unique. There're other issues I would like to address, too. - Currently, hostname is written to hostname of the machine which runs metrics sink. Since TaskInfo has hostname of the machine which runs task, we're better to use this. - Unit of timestamp of TaskInfo is second, while Storm Metrics Sink uses this as millisecond, resulting in timestamp flaw, and malfunction of cache eviction. It should be multiplied by 1000. - 'component name' is not unique across the cluster, so it's not fit for app id. 'topology name' is unique so proper value of app id is topology name. Consideration: Hostname for determining 'write shard' is set to hostname of the machine which runs metrics sink. Since I don't know read also be sharded, I'm not sure it's safe to use TaskInfo's hostname to hostname of TimelineMetric. Please review carefully regarding this. Diffs - ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java 02f5598 ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java 8171a4d Diff: https://reviews.apache.org/r/48030/diff/ Testing --- I tested this only ambari-metrics module since changeset is not related on other modules. [INFO] [INFO] Reactor Summary: [INFO] [INFO] ambari-metrics . SUCCESS [ 0.896 s] [INFO] Ambari Metrics Common .. SUCCESS [ 13.386 s] [INFO] Ambari Metrics Hadoop Sink . SUCCESS [ 5.085 s] [INFO] Ambari Metrics Flume Sink .. SUCCESS [ 6.827 s] [INFO] Ambari Metrics Kafka Sink .. SUCCESS [ 4.190 s] [INFO] Ambari Metrics Storm Sink .. SUCCESS [ 1.384 s] [INFO] Ambari Metrics Collector ... SUCCESS [04:06 min] [INFO] Ambari Metrics Monitor . SUCCESS [ 3.556 s] [INFO] Ambari Metrics Grafana . SUCCESS [01:03 min] [INFO] Ambari Metrics Assembly SUCCESS [ 3.567 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 05:48 min [INFO] Finished at: 2016-05-30T16:46:07+09:00 [INFO] Final Memory: 78M/1038M [INFO] In fact, I tried to run `mvn test` from ambari root directory but build is failing from ambari-web. ``` > fsevents@0.2.1 install > /Users/jlim/WorkArea/JavaProjects/ambari/ambari-web/node_modules/chokidar/node_modules/fsevents > node-gyp rebuild ... npm WARN install:fsevents fsevents@0.2.1 install: `node-gyp rebuild` npm WARN install:fsevents Exit status 1 ``` No luck on `npm install`, too. Thanks, Jungtaek Lim
Review Request 48065: AMBARI-16949 Metrics Collector API shows NPE if we use wildcard (%25 for '%') for metric name
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48065/
---
Review request for Ambari.
Bugs: AMBARI-16949
https://issues.apache.org/jira/browse/AMBARI-16949
Repository: ambari
Description
---
When we request '/ws/v1/timeline/metrics' with metric name which contains %25
(escaped '%'), response for the API is json describing there's NPE.
And NPE is logged for ambari-metrics-collector log file.
```
2016-05-30 09:15:05,061 WARN
org.apache.hadoop.yarn.webapp.GenericExceptionHandler: INTERNAL_SERVER_ERROR
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor.appendAggregateMetricFromResultSet(PhoenixHBaseAccessor.java:810)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor.getAggregateMetricRecords(PhoenixHBaseAccessor.java:772)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.HBaseTimelineMetricStore.getTimelineMetrics(HBaseTimelineMetricStore.java:178)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.TimelineWebServices.getTimelineMetrics(TimelineWebServices.java:372)
at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
```
Reason of NPE:
Metrics are properly fetched with wildcard. But when applying functions to
result set, actual metric name is not exist from map of metric name to list of
function.
Diffs
-
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessor.java
47962cb
Diff: https://reviews.apache.org/r/48065/diff/
Testing
---
Test failed from local but it's occurred from another modules, which is not
related to current modification.
```
Results :
Failed tests:
PrivilegeEventCreatorTest.putTest:107 expected:<...), Roles(
Permission[2:
Users: testuser2
Permission1:
Users: testuser
Groups: testgroup])> but was:<...), Roles(
Permission[1:
Users: testuser
Groups: testgroup
Permission2:
Users: testuser2])>
RepositoryVersionEventCreatorTest.postTest:70 expected:<...ating system:
redhat[6
Repository ID(2), Repository name(MyRepo6), Base url(http://example6.com)
Operating system: redhat7
Repository ID(1), Repository name(MyRepo), Base url(http://example].com)
)> but was:<...ating system: redhat[7
Repository ID(1), Repository name(MyRepo), Base url(http://example.com)
Operating system: redhat6
Repository ID(2), Repository name(MyRepo6), Base url(http://example6].com)
)>
RepositoryVersionEventCreatorTest.putTest:100 expected:<...ating system:
redhat[6
Repository ID(2), Repository name(MyRepo6), Base url(http://example6.com)
Operating system: redhat7
Repository ID(1), Repository name(MyRepo), Base url(http://example].com)
)> but was:<...ating system: redhat[7
Repository ID(1), Repository name(MyRepo), Base url(http://example.com)
Operating system: redhat6
Repository ID(2), Repository name(MyRepo6), Base url(http://example6].com)
)>
ViewPrivilegeEventCreatorTest.putTest:85 expected:<...tatus(200 OK),
Type([MyView), Version(MyView), Name(MyView), Permissions(
Permission2:
Users: testuser2
Permission1:
Users: testuser
Groups: testgroup])> but was:<...tatus(200 OK), Type([null), Version(null),
Name(null), Permissions(
Permission1:
Users: testuser
Groups: testgroup
Permission2:
Users: testuser2])>
ComponentResourceProviderTest.testGetResourcesAsAdministrator:190->testGetResources:296
expected:<[tru]e> but was:<[fals]e>
ComponentResourceProviderTest.testGetResourcesAsClusterAdministrator:195->testGetResources:296
expected:<[tru]e> but was:<[fals]e>
ComponentResourceProviderTest.testGetResourcesAsServiceAdministrator:200->testGetResources:296
expected:<[tru]e> but was:<[fals]e>
AmbariLdapDataPopulatorTest.testSynchronizeExistingLdapGroups_removeDuringIteration:333
Expectation failure on verify:
AmbariLdapDataPopulatorTestInstance.getLdapGroupByMemberAttr("group2"):
expected: 1, actual: 0
UpgradeCatalog222Test.testInitializeStromAndKafkaWidgets:1107
Unexpected method call
AmbariManagementController.initializeWidgetsAndLayouts(EasyMock for interface
org.apache.ambari.server.state.Cluster, EasyMock for interface
org.apache.ambari.server.state.Service):
AmbariManagementController.getClusters(): expected: at least 0, actual: 1
AmbariManagementController.initializeWidgetsAndLayouts(EasyMock for
interface org.apache.ambari.server.state.Cluster, EasyMock for interface
org.apache.ambari.server.state.Service): expected: 1, actual: 0
Tests in error:
KerberosCheckerTest.testCheckFailed » Unexpected exception,
expected
Re: Review Request 48395: AMBARI-17027: Metrics Collector API: Introduce basic series aggregation functions
> On 6 21, 2016, 4:40 오후, Prajwal Rao wrote: > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/partials/query.editor.html, > > line 112 > > <https://reviews.apache.org/r/48395/diff/2/?file=1425985#file1425985line112> > > > > Visually, I think it makes more sense to add this after "precision" as > > it seems to break the first row. Addressed from third patch. > On 6 21, 2016, 4:40 오후, Prajwal Rao wrote: > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/queryCtrl.js, line 39 > > <https://reviews.apache.org/r/48395/diff/2/?file=1425986#file1425986line39> > > > > You will need to accommodate for seriesAggregator not being set at all > > (undefined), for example when dashboards are imported without the > > seriesAggregator in them, it will throw up an error. > > > > You can rememdy this by doing a function similar to $scope.trasnform. Thanks for finding missed spot. Addressed from third patch. - Jungtaek --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48395/#review138870 --- On 6 22, 2016, 2:09 오전, Jungtaek Lim wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48395/ > --- > > (Updated 6 22, 2016, 2:09 오전) > > > Review request for Ambari, Aravindan Vijayan, Dmytro Sen, Prajwal Rao, > Sriharsha Chintalapani, Sid Wagle, and Yusaku Sako. > > > Bugs: AMBARI-17027 > https://issues.apache.org/jira/browse/AMBARI-17027 > > > Repository: ambari > > > Description > --- > > AMS doesn't provide tag so metric is identified by appId, metric name, > hostname, instanceId. In this situation metric name is normally consist of > origin metric name and tag values, like graphite, but unlike Graphite, AMS > also doesn't provide series aggregation functions so aggregation should be > done from caller side. > > It would be great if Ambari Metrics Collector provides series aggregation > functions, like sumSeries / > averageSeries / minSeries / maxSeries on Graphite. > > Query outputs: > https://gist.github.com/HeartSaVioR/f4f28b5b8b7bf2e5477e59d7fd56090f > > Attached Grafana screenshots to AMBARI-17027. Please refer > https://issues.apache.org/jira/browse/AMBARI-17027 for details. > > > Diffs > - > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/datasource.js b825774 > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/partials/query.editor.html > b034c03 > ambari-metrics/ambari-metrics-grafana/ambari-metrics/queryCtrl.js 2eb3613 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java > 1b2d02f > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStore.java > e37bc4d > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStoreWatcher.java > 7d49070 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/AbstractTimelineMetricsSeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/SeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunctionFactory.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAvgAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesMaxAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/jav
Re: Review Request 48395: AMBARI-17027: Metrics Collector API: Introduce basic series aggregation functions
afka Sink .. SUCCESS [ 4.064 s] [INFO] Ambari Metrics Storm Sink .. SUCCESS [ 1.289 s] [INFO] Ambari Metrics Collector ... SUCCESS [04:12 min] [INFO] Ambari Metrics Monitor . SUCCESS [ 3.587 s] [INFO] Ambari Metrics Grafana . FAILURE [01:15 min] [INFO] Ambari Metrics Assembly SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 06:02 min [INFO] Finished at: 2016-06-08T14:48:58+09:00 [INFO] Final Memory: 79M/1149M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (default) on project ambari-metrics-grafana: An Ant BuildException has occured: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake [ERROR] around Ant part ...https://grafanarel.s3.amazonaws.com/builds/grafana-2.6.0.linux-x64.tar.gz; dest="/Users/jlim/WorkArea/JavaProjects/ambari/ambari-metrics/ambari-metrics-grafana/target/grafana/grafana.tgz"/>... @ 5:222 in /Users/jlim/WorkArea/JavaProjects/ambari/ambari-metrics/ambari-metrics-grafana/target/antrun/build-Download Ambari Grafana.xml: SSL peer shut down incorrectly ``` > mvn test -rf :ambari-metrics-grafana ``` [INFO] [INFO] Reactor Summary: [INFO] [INFO] Ambari Metrics Grafana . SUCCESS [ 38.966 s] [INFO] Ambari Metrics Assembly SUCCESS [ 3.855 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 43.037 s [INFO] Finished at: 2016-06-08T14:50:33+09:00 [INFO] Final Memory: 24M/300M [INFO] ---- ``` Thanks, Jungtaek Lim
Re: Review Request 48395: AMBARI-17027: Metrics Collector API: Introduce basic series aggregation functions
> On 6 15, 2016, 6:16 오후, Prajwal Rao wrote: > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/queryCtrl.js, line 40 > > <https://reviews.apache.org/r/48395/diff/1/?file=1410877#file1410877line40> > > > > There seems to be no way to disable Series Aggregation. We need to have > > a "none" option, and "sum" shouldn't be set as the default option. > > Jungtaek Lim wrote: > There's check button on the UI (target.shouldAggregateSeries) and series > aggregation only works when button is checked. Please let me know there's a > bug on checking enable/disable series aggregation. Thanks! > > Jungtaek Lim wrote: > Prajwal clarified this issue: the point is UI inconsistency between other > things. In Ambari 2.4 they have "none" option on dropdown instead of having > checkbox to disable. I'll address it. > > Prajwal Rao wrote: > When "Aggregate Series" in unchecked, it should not show "sum" as an > option. It should show blank then, or should support "none" as an option. > Very confusing when it is unchecked but still shows "sum" as the selected > option. http://i.imgur.com/LeeC1k3.png Addressed. Please check diff between rev 1 and 2. Thanks! - Jungtaek --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48395/#review137790 --- On 6 21, 2016, 3:06 오전, Jungtaek Lim wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48395/ > --- > > (Updated 6 21, 2016, 3:06 오전) > > > Review request for Ambari, Aravindan Vijayan, Dmytro Sen, Prajwal Rao, > Sriharsha Chintalapani, Sid Wagle, and Yusaku Sako. > > > Bugs: AMBARI-17027 > https://issues.apache.org/jira/browse/AMBARI-17027 > > > Repository: ambari > > > Description > --- > > AMS doesn't provide tag so metric is identified by appId, metric name, > hostname, instanceId. In this situation metric name is normally consist of > origin metric name and tag values, like graphite, but unlike Graphite, AMS > also doesn't provide series aggregation functions so aggregation should be > done from caller side. > > It would be great if Ambari Metrics Collector provides series aggregation > functions, like sumSeries / > averageSeries / minSeries / maxSeries on Graphite. > > Query outputs: > https://gist.github.com/HeartSaVioR/f4f28b5b8b7bf2e5477e59d7fd56090f > > Attached Grafana screenshots to AMBARI-17027. Please refer > https://issues.apache.org/jira/browse/AMBARI-17027 for details. > > > Diffs > - > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/datasource.js b825774 > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/partials/query.editor.html > b034c03 > ambari-metrics/ambari-metrics-grafana/ambari-metrics/queryCtrl.js 2eb3613 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java > 1b2d02f > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStore.java > e37bc4d > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStoreWatcher.java > 7d49070 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/AbstractTimelineMetricsSeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/SeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunctionFactory.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAvgAggregateFunction.java > PR
Re: Review Request 48395: AMBARI-17027: Metrics Collector API: Introduce basic series aggregation functions
afka Sink .. SUCCESS [ 4.064 s] [INFO] Ambari Metrics Storm Sink .. SUCCESS [ 1.289 s] [INFO] Ambari Metrics Collector ... SUCCESS [04:12 min] [INFO] Ambari Metrics Monitor . SUCCESS [ 3.587 s] [INFO] Ambari Metrics Grafana . FAILURE [01:15 min] [INFO] Ambari Metrics Assembly SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 06:02 min [INFO] Finished at: 2016-06-08T14:48:58+09:00 [INFO] Final Memory: 79M/1149M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (default) on project ambari-metrics-grafana: An Ant BuildException has occured: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake [ERROR] around Ant part ...https://grafanarel.s3.amazonaws.com/builds/grafana-2.6.0.linux-x64.tar.gz; dest="/Users/jlim/WorkArea/JavaProjects/ambari/ambari-metrics/ambari-metrics-grafana/target/grafana/grafana.tgz"/>... @ 5:222 in /Users/jlim/WorkArea/JavaProjects/ambari/ambari-metrics/ambari-metrics-grafana/target/antrun/build-Download Ambari Grafana.xml: SSL peer shut down incorrectly ``` > mvn test -rf :ambari-metrics-grafana ``` [INFO] [INFO] Reactor Summary: [INFO] [INFO] Ambari Metrics Grafana . SUCCESS [ 38.966 s] [INFO] Ambari Metrics Assembly SUCCESS [ 3.855 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 43.037 s [INFO] Finished at: 2016-06-08T14:50:33+09:00 [INFO] Final Memory: 24M/300M [INFO] ---- ``` Thanks, Jungtaek Lim
Re: Review Request 48395: AMBARI-17027: Metrics Collector API: Introduce basic series aggregation functions
> On 6 15, 2016, 6:16 오후, Prajwal Rao wrote: > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/queryCtrl.js, line 40 > > <https://reviews.apache.org/r/48395/diff/1/?file=1410877#file1410877line40> > > > > There seems to be no way to disable Series Aggregation. We need to have > > a "none" option, and "sum" shouldn't be set as the default option. > > Jungtaek Lim wrote: > There's check button on the UI (target.shouldAggregateSeries) and series > aggregation only works when button is checked. Please let me know there's a > bug on checking enable/disable series aggregation. Thanks! Prajwal clarified this issue: the point is UI inconsistency between other things. In Ambari 2.4 they have "none" option on dropdown instead of having checkbox to disable. I'll address it. - Jungtaek --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48395/#review137790 ------- On 6 10, 2016, 12:50 오전, Jungtaek Lim wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48395/ > --- > > (Updated 6 10, 2016, 12:50 오전) > > > Review request for Ambari, Aravindan Vijayan, Dmytro Sen, Prajwal Rao, > Sriharsha Chintalapani, Sid Wagle, and Yusaku Sako. > > > Bugs: AMBARI-17027 > https://issues.apache.org/jira/browse/AMBARI-17027 > > > Repository: ambari > > > Description > --- > > AMS doesn't provide tag so metric is identified by appId, metric name, > hostname, instanceId. In this situation metric name is normally consist of > origin metric name and tag values, like graphite, but unlike Graphite, AMS > also doesn't provide series aggregation functions so aggregation should be > done from caller side. > > It would be great if Ambari Metrics Collector provides series aggregation > functions, like sumSeries / > averageSeries / minSeries / maxSeries on Graphite. > > Query outputs: > https://gist.github.com/HeartSaVioR/f4f28b5b8b7bf2e5477e59d7fd56090f > > Attached Grafana screenshots to AMBARI-17027. Please refer > https://issues.apache.org/jira/browse/AMBARI-17027 for details. > > > Diffs > - > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/datasource.js 7390aa8 > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/partials/query.editor.html > b034c03 > ambari-metrics/ambari-metrics-grafana/ambari-metrics/queryCtrl.js 2eb3613 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java > 1b2d02f > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStore.java > e37bc4d > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStoreWatcher.java > 7d49070 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/AbstractTimelineMetricsSeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/SeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunctionFactory.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAvgAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesMaxAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesMinAggregateFunction.java > PRE-CREATION > > ambari-metr
Re: Review Request 48395: AMBARI-17027: Metrics Collector API: Introduce basic series aggregation functions
> On 6 15, 2016, 6:16 오후, Prajwal Rao wrote: > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/queryCtrl.js, line 40 > > <https://reviews.apache.org/r/48395/diff/1/?file=1410877#file1410877line40> > > > > There seems to be no way to disable Series Aggregation. We need to have > > a "none" option, and "sum" shouldn't be set as the default option. There's check button on the UI (target.shouldAggregateSeries) and series aggregation only works when button is checked. Please let me know there's a bug on checking enable/disable series aggregation. Thanks! - Jungtaek --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48395/#review137790 ------- On 6 10, 2016, 12:50 오전, Jungtaek Lim wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48395/ > --- > > (Updated 6 10, 2016, 12:50 오전) > > > Review request for Ambari, Aravindan Vijayan, Dmytro Sen, Prajwal Rao, > Sriharsha Chintalapani, Sid Wagle, and Yusaku Sako. > > > Bugs: AMBARI-17027 > https://issues.apache.org/jira/browse/AMBARI-17027 > > > Repository: ambari > > > Description > --- > > AMS doesn't provide tag so metric is identified by appId, metric name, > hostname, instanceId. In this situation metric name is normally consist of > origin metric name and tag values, like graphite, but unlike Graphite, AMS > also doesn't provide series aggregation functions so aggregation should be > done from caller side. > > It would be great if Ambari Metrics Collector provides series aggregation > functions, like sumSeries / > averageSeries / minSeries / maxSeries on Graphite. > > Query outputs: > https://gist.github.com/HeartSaVioR/f4f28b5b8b7bf2e5477e59d7fd56090f > > Attached Grafana screenshots to AMBARI-17027. Please refer > https://issues.apache.org/jira/browse/AMBARI-17027 for details. > > > Diffs > - > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/datasource.js 7390aa8 > > ambari-metrics/ambari-metrics-grafana/ambari-metrics/partials/query.editor.html > b034c03 > ambari-metrics/ambari-metrics-grafana/ambari-metrics/queryCtrl.js 2eb3613 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java > 1b2d02f > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStore.java > e37bc4d > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStoreWatcher.java > 7d49070 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/AbstractTimelineMetricsSeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/SeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunctionFactory.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAvgAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesMaxAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesMinAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesSumAggregateFunction.java > PRE-CREATION > > ambari-metrics/ambari-metrics-t
Review Request 49173: AMBARI-17416: AMS-Grafana: show 'alias' when there's no datapoint available
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49173/ --- Review request for Ambari, Prajwal Rao, Sriharsha Chintalapani, and Yusaku Sako. Bugs: AMBARI-17416 https://issues.apache.org/jira/browse/AMBARI-17416 Repository: ambari Description --- When there's no datapoint available, AMS Grafana plugin just ignores alias and shows metric name instead. It seems to be not consistent in point of view, and not good UX I think. It'd be great AMS Grafana shows alias whenever there're datapoints available or not. Diffs - ambari-metrics/ambari-metrics-grafana/ambari-metrics/datasource.js 6a4f01b Diff: https://reviews.apache.org/r/49173/diff/ Testing --- Tested manually. Actually this is tiny fix so we can imagine how it will make change. Thanks, Jungtaek Lim
Re: Review Request 49249: AMBARI-17445: Storm metrics sink: expand metrics name to contain additional informations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49249/ --- (Updated 6 27, 2016, 11:02 오후) Review request for Ambari, Aravindan Vijayan, Prajwal Rao, Sriharsha Chintalapani, and Sid Wagle. Changes --- get rid of unused import Bugs: AMBARI-17445 https://issues.apache.org/jira/browse/AMBARI-17445 Repository: ambari Description --- With AMBARI-16946 we expanded metric name to have component name, task id, and key when type of metric value is Map. Topology name was placed to appId. While talking with Prajwal Rao and sriharsha chintalapani, we found that we should move out all necessary informations like topology name, worker host, worker port to metric name so that they can be templatized. This also supercedes AMBARI-17249 since all metrics will include worker host and worker port to metric name. Diffs (updated) - ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java eb572b3 ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java c4b54b4 Diff: https://reviews.apache.org/r/49249/diff/ Testing --- Tested locally. Attached screenshot to origin issue. https://issues.apache.org/jira/secure/attachment/12813582/AMBARI-17445-changes.png Thanks, Jungtaek Lim
Re: Review Request 49173: AMBARI-17416: AMS-Grafana: show 'alias' when there's no datapoint available
> On 6 24, 2016, 4:54 오후, Prajwal Rao wrote:
> > ambari-metrics/ambari-metrics-grafana/ambari-metrics/datasource.js, lines
> > 128-133
> > <https://reviews.apache.org/r/49173/diff/1/?file=1428948#file1428948line128>
> >
> > All graphs on HBase-Users have legend hidden.
> >
> > We do need to handle this on aggregate level dashboards and graphs, and
> > I already have a separate fix for that.
Prajwal Rao
I'm sorry but I don't get your feedback.
I just would like to show alias instead of raw metric name since it could be
too long and not beauty to show when we introduce series aggregation ('%' will
be used to metric name). But I may miss some spots in patch since I understand
this javascript partially.
If you already have a patch to cover this issue, please let me know. I'd be
happy to close the review. You can also close issue AMBARI-17416 as duplicated
or take over.
- Jungtaek
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49173/#review139384
-------
On 6 24, 2016, 1:42 오전, Jungtaek Lim wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49173/
> ---
>
> (Updated 6 24, 2016, 1:42 오전)
>
>
> Review request for Ambari, Prajwal Rao, Sriharsha Chintalapani, and Yusaku
> Sako.
>
>
> Bugs: AMBARI-17416
> https://issues.apache.org/jira/browse/AMBARI-17416
>
>
> Repository: ambari
>
>
> Description
> ---
>
> When there's no datapoint available, AMS Grafana plugin just ignores alias
> and shows metric name instead. It seems to be not consistent in point of
> view, and not good UX I think.
>
> It'd be great AMS Grafana shows alias whenever there're datapoints available
> or not.
>
>
> Diffs
> -
>
> ambari-metrics/ambari-metrics-grafana/ambari-metrics/datasource.js 6a4f01b
>
> Diff: https://reviews.apache.org/r/49173/diff/
>
>
> Testing
> ---
>
> Tested manually. Actually this is tiny fix so we can imagine how it will make
> change.
>
>
> Thanks,
>
> Jungtaek Lim
>
>
Review Request 49249: AMBARI-17445: Storm metrics sink: expand metrics name to contain additional informations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49249/ --- Review request for Ambari, Aravindan Vijayan, Prajwal Rao, Sriharsha Chintalapani, and Sid Wagle. Bugs: AMBARI-17445 https://issues.apache.org/jira/browse/AMBARI-17445 Repository: ambari Description --- With AMBARI-16946 we expanded metric name to have component name, task id, and key when type of metric value is Map. Topology name was placed to appId. While talking with Prajwal Rao and sriharsha chintalapani, we found that we should move out all necessary informations like topology name, worker host, worker port to metric name so that they can be templatized. This also supercedes AMBARI-17249 since all metrics will include worker host and worker port to metric name. Diffs - ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java eb572b3 ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java c4b54b4 Diff: https://reviews.apache.org/r/49249/diff/ Testing --- Tested locally. Attached screenshot to origin issue. https://issues.apache.org/jira/secure/attachment/12813582/AMBARI-17445-changes.png Thanks, Jungtaek Lim
Review Request 48840: AMBARI-17249 Storm metrics sink should include worker host and port to metric name when metrics are coming from SystemBolt
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48840/ --- Review request for Ambari, Aravindan Vijayan, Sriharsha Chintalapani, and Sid Wagle. Bugs: AMBARI-17249 https://issues.apache.org/jira/browse/AMBARI-17249 Repository: ambari Description --- Background: SystemBolt is reserved component for each worker which exposes worker level metrics like Memory, GC, and uptime, and so on. All SystemBolts have -1 as task id, so identifying SystemBolt components to task id as same as normal components will fail. Since each worker is identified by (worker host, port) pair, we can add hostname and port to metric name to make metric name unique. Since it's only for SystemBolt, instead of adding hostname and port to all of metrics, we can add these to specific metrics which are coming from SystemBolt. Diffs - ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java eb572b3 ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java c4b54b4 Diff: https://reviews.apache.org/r/48840/diff/ Testing --- mvn test succeed applied and tested manually (via 2.4 branch) Thanks, Jungtaek Lim
Re: Review Request 50479: AMBARI-17909 AMS Storm Sink: apply change of Storm metrics improvement - worker level aggregation
> On 7 27, 2016, 2:31 오전, Jungtaek Lim wrote: > > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/storm-site.xml, > > line 54 > > <https://reviews.apache.org/r/50479/diff/1/?file=1454635#file1454635line54> > > > > To tell the truth I don't understand how this tag works. > > Could you elaborate how on-ambari-upgrade works, and also could you > > check all the properties in this pom? > > Other properties (even outside of the patch in this pom) seems to be in > > the same line. > > Sriharsha Chintalapani wrote: > basically that property determines if we want to add this setting for new > upgrades. we should make that true on rest of settings as well. Thanks for explanation. I also think we need to add all the properties when upgrading. Addressed. - Jungtaek --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50479/#review143655 ------- On 7 27, 2016, 2:52 오전, Jungtaek Lim wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50479/ > --- > > (Updated 7 27, 2016, 2:52 오전) > > > Review request for Ambari, Aravindan Vijayan, Sriharsha Chintalapani, and Sid > Wagle. > > > Bugs: AMBARI-17909 > https://issues.apache.org/jira/browse/AMBARI-17909 > > > Repository: ambari > > > Description > --- > > Abstraction: > This issue is for following up changes on STORM-2006: Storm metrics feature > improvement: support per-worker level metrics aggregation. > > Details: > With recent patches for AMS and AMS Storm sink, AMS stores task level metrics > from Storm, and relevant dashboards can be configured via Grafana. > But we found that it incurs too many kinds of metrics and also too many data > points published to AMS because even a topology can have lots of tasks. > In order to reduce this, I addressed STORM-2006, but it also needs AMS Storm > sink and relevant Storm configurations to be changed. > > Note: > We're moving some properties so STORM-2006 must be go on together in order to > make metrics filter working properly. But AMS Storm sink itself is backward > compatible. > I don't modify legacy AMS Storm sink since STORM-2006 will not be ported back > to 0.10.x or lower. > > > Diffs > - > > > ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java > 3a4289b > > ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java > fadb00c > > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/storm-site.xml > 280fc42 > > Diff: https://reviews.apache.org/r/50479/diff/ > > > Testing > --- > > - Build succeed (build passed with 2.4 branch without Ignore annotation) > - Tested manually with STORM-2006 pre-applied cluster > > > Thanks, > > Jungtaek Lim > >
Re: Review Request 50479: AMBARI-17909 AMS Storm Sink: apply change of Storm metrics improvement - worker level aggregation
> On 7 27, 2016, 2:10 오전, Sriharsha Chintalapani wrote: > > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/storm-site.xml, > > line 54 > > <https://reviews.apache.org/r/50479/diff/1/?file=1454635#file1454635line54> > > > > we should make this as true on-upgrade no? > > Jungtaek Lim wrote: > To tell the truth I don't understand how this tag works. > Could you elaborate how on-ambari-upgrade works, and also could you check > all the properties in this pom? > Other properties (even outside of the patch in this pom) seems to be in > the same line. Submitted new patch. Please check again and make issue 'fixed' if it's ok. - Jungtaek --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50479/#review143652 ------- On 7 27, 2016, 2:52 오전, Jungtaek Lim wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50479/ > --- > > (Updated 7 27, 2016, 2:52 오전) > > > Review request for Ambari, Aravindan Vijayan, Sriharsha Chintalapani, and Sid > Wagle. > > > Bugs: AMBARI-17909 > https://issues.apache.org/jira/browse/AMBARI-17909 > > > Repository: ambari > > > Description > --- > > Abstraction: > This issue is for following up changes on STORM-2006: Storm metrics feature > improvement: support per-worker level metrics aggregation. > > Details: > With recent patches for AMS and AMS Storm sink, AMS stores task level metrics > from Storm, and relevant dashboards can be configured via Grafana. > But we found that it incurs too many kinds of metrics and also too many data > points published to AMS because even a topology can have lots of tasks. > In order to reduce this, I addressed STORM-2006, but it also needs AMS Storm > sink and relevant Storm configurations to be changed. > > Note: > We're moving some properties so STORM-2006 must be go on together in order to > make metrics filter working properly. But AMS Storm sink itself is backward > compatible. > I don't modify legacy AMS Storm sink since STORM-2006 will not be ported back > to 0.10.x or lower. > > > Diffs > - > > > ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java > 3a4289b > > ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java > fadb00c > > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/storm-site.xml > 280fc42 > > Diff: https://reviews.apache.org/r/50479/diff/ > > > Testing > --- > > - Build succeed (build passed with 2.4 branch without Ignore annotation) > - Tested manually with STORM-2006 pre-applied cluster > > > Thanks, > > Jungtaek Lim > >
Review Request 50479: AMBARI-17909 AMS Storm Sink: apply change of Storm metrics improvement - worker level aggregation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50479/ --- Review request for Ambari, Aravindan Vijayan, Sriharsha Chintalapani, and Sid Wagle. Bugs: AMBARI-17909 https://issues.apache.org/jira/browse/AMBARI-17909 Repository: ambari Description --- Abstraction: This issue is for following up changes on STORM-2006: Storm metrics feature improvement: support per-worker level metrics aggregation. Details: With recent patches for AMS and AMS Storm sink, AMS stores task level metrics from Storm, and relevant dashboards can be configured via Grafana. But we found that it incurs too many kinds of metrics and also too many data points published to AMS because even a topology can have lots of tasks. In order to reduce this, I addressed STORM-2006, but it also needs AMS Storm sink and relevant Storm configurations to be changed. Note: We're moving some properties so STORM-2006 must be go on together in order to make metrics filter working properly. But AMS Storm sink itself is backward compatible. I don't modify legacy AMS Storm sink since STORM-2006 will not be ported back to 0.10.x or lower. Diffs - ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java 3a4289b ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java fadb00c ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/storm-site.xml 280fc42 Diff: https://reviews.apache.org/r/50479/diff/ Testing --- - Build succeed (build passed with 2.4 branch without Ignore annotation) - Tested manually with STORM-2006 pre-applied cluster Thanks, Jungtaek Lim
Re: Review Request 50479: AMBARI-17909 AMS Storm Sink: apply change of Storm metrics improvement - worker level aggregation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50479/ --- (Updated 7 27, 2016, 2:52 오전) Review request for Ambari, Aravindan Vijayan, Sriharsha Chintalapani, and Sid Wagle. Changes --- Change on-ambari-upgrade values to true Bugs: AMBARI-17909 https://issues.apache.org/jira/browse/AMBARI-17909 Repository: ambari Description --- Abstraction: This issue is for following up changes on STORM-2006: Storm metrics feature improvement: support per-worker level metrics aggregation. Details: With recent patches for AMS and AMS Storm sink, AMS stores task level metrics from Storm, and relevant dashboards can be configured via Grafana. But we found that it incurs too many kinds of metrics and also too many data points published to AMS because even a topology can have lots of tasks. In order to reduce this, I addressed STORM-2006, but it also needs AMS Storm sink and relevant Storm configurations to be changed. Note: We're moving some properties so STORM-2006 must be go on together in order to make metrics filter working properly. But AMS Storm sink itself is backward compatible. I don't modify legacy AMS Storm sink since STORM-2006 will not be ported back to 0.10.x or lower. Diffs (updated) - ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java 3a4289b ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java fadb00c ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/storm-site.xml 280fc42 Diff: https://reviews.apache.org/r/50479/diff/ Testing --- - Build succeed (build passed with 2.4 branch without Ignore annotation) - Tested manually with STORM-2006 pre-applied cluster Thanks, Jungtaek Lim
Re: Review Request 50479: AMBARI-17909 AMS Storm Sink: apply change of Storm metrics improvement - worker level aggregation
> On 7 27, 2016, 2:10 오전, Sriharsha Chintalapani wrote: > > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/storm-site.xml, > > line 54 > > <https://reviews.apache.org/r/50479/diff/1/?file=1454635#file1454635line54> > > > > we should make this as true on-upgrade no? To tell the truth I don't understand how this tag works. Could you elaborate how on-ambari-upgrade works, and also could you check all the properties in this pom? Other properties (even outside of the patch in this pom) seems to be in the same line. - Jungtaek --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50479/#review143652 ------- On 7 27, 2016, 1:58 오전, Jungtaek Lim wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50479/ > --- > > (Updated 7 27, 2016, 1:58 오전) > > > Review request for Ambari, Aravindan Vijayan, Sriharsha Chintalapani, and Sid > Wagle. > > > Bugs: AMBARI-17909 > https://issues.apache.org/jira/browse/AMBARI-17909 > > > Repository: ambari > > > Description > --- > > Abstraction: > This issue is for following up changes on STORM-2006: Storm metrics feature > improvement: support per-worker level metrics aggregation. > > Details: > With recent patches for AMS and AMS Storm sink, AMS stores task level metrics > from Storm, and relevant dashboards can be configured via Grafana. > But we found that it incurs too many kinds of metrics and also too many data > points published to AMS because even a topology can have lots of tasks. > In order to reduce this, I addressed STORM-2006, but it also needs AMS Storm > sink and relevant Storm configurations to be changed. > > Note: > We're moving some properties so STORM-2006 must be go on together in order to > make metrics filter working properly. But AMS Storm sink itself is backward > compatible. > I don't modify legacy AMS Storm sink since STORM-2006 will not be ported back > to 0.10.x or lower. > > > Diffs > - > > > ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java > 3a4289b > > ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java > fadb00c > > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/storm-site.xml > 280fc42 > > Diff: https://reviews.apache.org/r/50479/diff/ > > > Testing > --- > > - Build succeed (build passed with 2.4 branch without Ignore annotation) > - Tested manually with STORM-2006 pre-applied cluster > > > Thanks, > > Jungtaek Lim > >
Review Request 50180: AMBARI-17784 AMS Storm Sink: remove redundant information from kafka offset metrics on storm-kafka (including AMBARI-17725)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50180/ --- Review request for Ambari, Aravindan Vijayan, Sriharsha Chintalapani, and Sid Wagle. Bugs: AMBARI-17784 https://issues.apache.org/jira/browse/AMBARI-17784 Repository: ambari Description --- This is follow-up issue for AMBARI-17725. After AMBARI-17725, we can distinguish cluster metrics and topology metrics, which makes us possible to topology name and component name as variables for templatized dashboard. But kafkaOffset metrics still have kafka topic and partition information to middle of metric name which make us hard to pick. Since worker host and worker port, and task id is not needed for kafkaOffset metrics, we can get rid of those things from only kafkaOffset metrics. > topology level * Before applying: topology.kafka-topology.kafka-spout..6700.14.kafkaOffset.myKafkaTopic/totalSpoutLag * After applying: topology.kafka-topology.kafka-topic.myKafkaTopic.totalSpoutLag > partition level * Before applying: topology.kafka-topology.kafka-spout..6700.14.kafkaOffset.myKafkaTopic/partition-5/earliestTimeOffset * After applying: topology.kafka-topology.kafka-topic.myKafkaTopic.partition-5.earliestTimeOffset Diffs - ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java 9e6cc98 ambari-metrics/ambari-metrics-storm-sink-legacy/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java 271c11f ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java dcc3192 ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java c76197b Diff: https://reviews.apache.org/r/50180/diff/ Testing --- * Unit test succeed (@ignore is added for trunk. it succeed for branch 2.4) * Tested manually Thanks, Jungtaek Lim
Re: Review Request 50180: AMBARI-17784 AMS Storm Sink: remove redundant information from kafka offset metrics on storm-kafka (including AMBARI-17725)
> On 7 19, 2016, 7:29 오후, Aravindan Vijayan wrote: > > Jungtaek, Is the metricName generation logic same for both the legacy and > > current version of Storm? > > Sriharsha Chintalapani wrote: > Aravindan, Given that all these metrics coming in HDP-2.5 storm why are > we worried about legacy version. We are not even storing topology level > metrics in previous version of storm and I am sure we are not even storing > the cluster level. > > Aravindan Vijayan wrote: > Sriharsha, the reason I asked this is because the patch has the same > changes in both legacy and current Storm sinks. Just confirming that this > will not cause any issue in clusters with Ambari-2.4.0 and HDP 2.4. Yes they should be same since there's no change on Storm side for topology level metrics. Btw, Reporter (cluster level metrics) should be different between legacy and current. - Jungtaek --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50180/#review142812 ------- On 7 19, 2016, 6:41 오전, Jungtaek Lim wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50180/ > --- > > (Updated 7 19, 2016, 6:41 오전) > > > Review request for Ambari, Aravindan Vijayan, Sriharsha Chintalapani, and Sid > Wagle. > > > Bugs: AMBARI-17784 > https://issues.apache.org/jira/browse/AMBARI-17784 > > > Repository: ambari > > > Description > --- > > This is follow-up issue for AMBARI-17725. > > After AMBARI-17725, we can distinguish cluster metrics and topology metrics, > which makes us possible to topology name and component name as variables for > templatized dashboard. > > But kafkaOffset metrics still have kafka topic and partition information to > middle of metric name which make us hard to pick. Since worker host and > worker port, and task id is not needed for kafkaOffset metrics, we can get > rid of those things from only kafkaOffset metrics. > > > topology level > > * Before applying: topology.kafka-topology.kafka-spout. host>.6700.14.kafkaOffset.myKafkaTopic/totalSpoutLag > * After applying: > topology.kafka-topology.kafka-topic.myKafkaTopic.totalSpoutLag > > > partition level > > * Before applying: topology.kafka-topology.kafka-spout. host>.6700.14.kafkaOffset.myKafkaTopic/partition-5/earliestTimeOffset > * After applying: > topology.kafka-topology.kafka-topic.myKafkaTopic.partition-5.earliestTimeOffset > > > Diffs > - > > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java > 9e6cc98 > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java > 271c11f > > ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java > dcc3192 > > ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java > c76197b > > Diff: https://reviews.apache.org/r/50180/diff/ > > > Testing > --- > > * Unit test succeed (@ignore is added for trunk. it succeed for branch 2.4) > * Tested manually > > > Thanks, > > Jungtaek Lim > >
Review Request 50063: AMBARI-17725: AMS Storm Sink: Storm topology level metrics should have prefix to distinguish cluster level metrics
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50063/
---
Review request for Ambari, Aravindan Vijayan, Sriharsha Chintalapani, and Sid
Wagle.
Bugs: AMBARI-17725
https://issues.apache.org/jira/browse/AMBARI-17725
Repository: ambari
Description
---
Since Ambari 2.4.0 we have two kinds of Storm metrics into AMS which one is
cluster level and another one is topology level.
Currently we don't add prefix on them which makes them hard to distinguish,
which should be needed to create topology level dashboard in Grafana.
To resolve this we should add the prefix to Storm topology level metrics.
This patch adds 'topology.' as prefix of topology level metrics so that it can
be distinguished.
Also shows warning message to user (in Storm worker) to notice period ('.') in
topology name is not recommended for AMS.
Diffs
-
ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java
9e6cc98
ambari-metrics/ambari-metrics-storm-sink-legacy/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java
271c11f
ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java
dcc3192
ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java
c76197b
Diff: https://reviews.apache.org/r/50063/diff/
Testing
---
Build OK via `mvn clean install` for both of storm-sink and storm-sink-legacy.
Tested manually.
Thanks,
Jungtaek Lim
Re: Review Request 49249: AMBARI-17445: Storm metrics sink: expand metrics name to contain additional informations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49249/ --- (Updated 7 1, 2016, 3:57 오전) Review request for Ambari, Aravindan Vijayan, Prajwal Rao, Sriharsha Chintalapani, and Sid Wagle. Changes --- Rebased to current trunk & restore debug log message Bugs: AMBARI-17445 https://issues.apache.org/jira/browse/AMBARI-17445 Repository: ambari Description --- With AMBARI-16946 we expanded metric name to have component name, task id, and key when type of metric value is Map. Topology name was placed to appId. While talking with Prajwal Rao and sriharsha chintalapani, we found that we should move out all necessary informations like topology name, worker host, worker port to metric name so that they can be templatized. This also supercedes AMBARI-17249 since all metrics will include worker host and worker port to metric name. Diffs (updated) - ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java 0ef09d6 ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java 4ea7396 Diff: https://reviews.apache.org/r/49249/diff/ Testing --- Tested locally. Attached screenshot to origin issue. https://issues.apache.org/jira/secure/attachment/12813582/AMBARI-17445-changes.png Thanks, Jungtaek Lim
Re: Review Request 49249: AMBARI-17445: Storm metrics sink: expand metrics name to contain additional informations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49249/ --- (Updated 7 4, 2016, 12:23 오후) Review request for Ambari, Aravindan Vijayan, Prajwal Rao, Sriharsha Chintalapani, and Sid Wagle. Changes --- Also applied changeset to legacy sink. (Rebased to recent trunk) Bugs: AMBARI-17445 https://issues.apache.org/jira/browse/AMBARI-17445 Repository: ambari Description --- With AMBARI-16946 we expanded metric name to have component name, task id, and key when type of metric value is Map. Topology name was placed to appId. While talking with Prajwal Rao and sriharsha chintalapani, we found that we should move out all necessary informations like topology name, worker host, worker port to metric name so that they can be templatized. This also supercedes AMBARI-17249 since all metrics will include worker host and worker port to metric name. Diffs (updated) - ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java 9266e5f ambari-metrics/ambari-metrics-storm-sink-legacy/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java 4ea7396 ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java d04bc15 ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java 3e9ed34 Diff: https://reviews.apache.org/r/49249/diff/ Testing --- Tested locally. Attached screenshot to origin issue. https://issues.apache.org/jira/secure/attachment/12813582/AMBARI-17445-changes.png Thanks, Jungtaek Lim
Re: Review Request 49514: Support Storm 1.0 in Ambari Metrics for Storm
> On 7 4, 2016, 2:09 오전, Jungtaek Lim wrote: > > Ship It! Looks good to me overall. I assumed 'ambari-metrics-storm-legacy' is just a copy of previous 'ambari-metrics-storm'. - Jungtaek --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49514/#review140599 --- On 7 3, 2016, 1:23 오후, Dmytro Sen wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49514/ > --- > > (Updated 7 3, 2016, 1:23 오후) > > > Review request for Ambari, Aravindan Vijayan, Jungtaek Lim, and Sid Wagle. > > > Bugs: AMBARI-17080 > https://issues.apache.org/jira/browse/AMBARI-17080 > > > Repository: ambari > > > Description > --- > > Support 2 storm sinks. > > > Diffs > - > > ambari-metrics/ambari-metrics-assembly/pom.xml 941c3aa > ambari-metrics/ambari-metrics-assembly/src/main/assembly/sink-windows.xml > e82d2d4 > ambari-metrics/ambari-metrics-assembly/src/main/assembly/sink.xml 4a3b7c5 > ambari-metrics/ambari-metrics-common/pom.xml 3e11e7f > ambari-metrics/ambari-metrics-storm-sink-legacy/pom.xml PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/assemblies/empty.xml > PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java > PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java > PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java > PRE-CREATION > ambari-metrics/ambari-metrics-storm-sink/pom.xml ee4d2c3 > > ambari-metrics/ambari-metrics-storm-sink/src/main/conf/storm-metrics2.properties.j2 > 4553224 > > ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/NumberUtil.java > PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java > 8f98563 > > ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java > 0ef09d6 > > ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java > 4ea7396 > ambari-metrics/pom.xml 726a823 > > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/storm-site.xml > 0d029e8 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/params_linux.py > fac6331 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/storm.py > 2d50767 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/ui_server.py > 6551067 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/templates/storm-metrics2.properties.j2 > ea8b1d0 > > ambari-server/src/main/resources/stacks/HDP/2.5/services/STORM/configuration/storm-site.xml > f3bbce8 > ambari-server/src/test/python/stacks/2.1/STORM/test_storm_nimbus.py 2fcb7e2 > ambari-server/src/test/python/stacks/2.1/STORM/test_storm_ui_server.py > f7a2686 > > Diff: https://reviews.apache.org/r/49514/diff/ > > > Testing > --- > > Unit testst passed > > > Thanks, > > Dmytro Sen > >
Re: Review Request 49514: Support Storm 1.0 in Ambari Metrics for Storm
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49514/#review140599 --- Ship it! Ship It! - Jungtaek Lim On 7 3, 2016, 1:23 오후, Dmytro Sen wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49514/ > --- > > (Updated 7 3, 2016, 1:23 오후) > > > Review request for Ambari, Aravindan Vijayan, Jungtaek Lim, and Sid Wagle. > > > Bugs: AMBARI-17080 > https://issues.apache.org/jira/browse/AMBARI-17080 > > > Repository: ambari > > > Description > --- > > Support 2 storm sinks. > > > Diffs > - > > ambari-metrics/ambari-metrics-assembly/pom.xml 941c3aa > ambari-metrics/ambari-metrics-assembly/src/main/assembly/sink-windows.xml > e82d2d4 > ambari-metrics/ambari-metrics-assembly/src/main/assembly/sink.xml 4a3b7c5 > ambari-metrics/ambari-metrics-common/pom.xml 3e11e7f > ambari-metrics/ambari-metrics-storm-sink-legacy/pom.xml PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/assemblies/empty.xml > PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java > PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java > PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java > PRE-CREATION > ambari-metrics/ambari-metrics-storm-sink/pom.xml ee4d2c3 > > ambari-metrics/ambari-metrics-storm-sink/src/main/conf/storm-metrics2.properties.j2 > 4553224 > > ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/NumberUtil.java > PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java > 8f98563 > > ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java > 0ef09d6 > > ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java > 4ea7396 > ambari-metrics/pom.xml 726a823 > > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/storm-site.xml > 0d029e8 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/params_linux.py > fac6331 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/storm.py > 2d50767 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/ui_server.py > 6551067 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/templates/storm-metrics2.properties.j2 > ea8b1d0 > > ambari-server/src/main/resources/stacks/HDP/2.5/services/STORM/configuration/storm-site.xml > f3bbce8 > ambari-server/src/test/python/stacks/2.1/STORM/test_storm_nimbus.py 2fcb7e2 > ambari-server/src/test/python/stacks/2.1/STORM/test_storm_ui_server.py > f7a2686 > > Diff: https://reviews.apache.org/r/49514/diff/ > > > Testing > --- > > Unit testst passed > > > Thanks, > > Dmytro Sen > >
Re: Review Request 49514: Support Storm 1.0 in Ambari Metrics for Storm
> On July 1, 2016, 5:09 p.m., Sid Wagle wrote: > > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/storm-site.xml, > > line 40 > > <https://reviews.apache.org/r/49514/diff/1/?file=1434997#file1434997line40> > > > > Does the parallelism = 1, mean 1 bolt getting all metric reports in a > > topology. Is it counter-intuitive for performance, something we should > > check with storm group. Its default value has been 1, so setting it explicitly means letting users to modify it easily. - Jungtaek --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49514/#review140384 --- On July 1, 2016, 3:32 p.m., Dmytro Sen wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49514/ > --- > > (Updated July 1, 2016, 3:32 p.m.) > > > Review request for Ambari, Aravindan Vijayan, Jungtaek Lim, and Sid Wagle. > > > Bugs: AMBARI-17080 > https://issues.apache.org/jira/browse/AMBARI-17080 > > > Repository: ambari > > > Description > --- > > Support 2 storm sinks. > > > Diffs > - > > ambari-metrics/ambari-metrics-assembly/pom.xml 5e2d819 > ambari-metrics/ambari-metrics-assembly/src/main/assembly/sink-windows.xml > e82d2d4 > ambari-metrics/ambari-metrics-assembly/src/main/assembly/sink.xml 4a3b7c5 > ambari-metrics/ambari-metrics-storm-sink-legacy/pom.xml PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/assemblies/empty.xml > PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java > PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java > PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java > PRE-CREATION > ambari-metrics/ambari-metrics-storm-sink/pom.xml 1591d39 > > ambari-metrics/ambari-metrics-storm-sink/src/main/conf/storm-metrics2.properties.j2 > 4553224 > > ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/NumberUtil.java > PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java > ab5f1e4 > > ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java > 6ab12e1 > > ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java > e582a95 > ambari-metrics/pom.xml 7221ab5 > > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/storm-site.xml > 0d029e8 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/params_linux.py > 073bb1c > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/storm.py > 2d50767 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/ui_server.py > 6551067 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/templates/storm-metrics2.properties.j2 > 9acf173 > > ambari-server/src/main/resources/stacks/HDP/2.5/services/STORM/configuration/storm-site.xml > f3bbce8 > > Diff: https://reviews.apache.org/r/49514/diff/ > > > Testing > --- > > Unit testst passed > > > Thanks, > > Dmytro Sen > >
Re: Review Request 49514: Support Storm 1.0 in Ambari Metrics for Storm
> On July 1, 2016, 3:54 p.m., Sid Wagle wrote: > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java, > > line 194 > > <https://reviews.apache.org/r/49514/diff/1/?file=1434988#file1434988line194> > > > > Good catch, however, why not look for "._" and replace that only if > > needed. > > Dmytro Sen wrote: > There is no change here. It's a copy of previous > StormTimelineMetricsSink.java It's originally for escaping '._' but only escaping that case is not intuitive for users. Escaping all _ will make users to predict escaped metric name easily. > On July 1, 2016, 3:54 p.m., Sid Wagle wrote: > > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/storm-site.xml, > > line 30 > > <https://reviews.apache.org/r/49514/diff/1/?file=1434997#file1434997line30> > > > > So both old and new properties are ok to be present at the same time? > > Dmytro Sen wrote: > yes yes since storm runs properly with unknown configs are presented. - Jungtaek --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49514/#review140370 --- On July 1, 2016, 3:32 p.m., Dmytro Sen wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49514/ > ------- > > (Updated July 1, 2016, 3:32 p.m.) > > > Review request for Ambari, Aravindan Vijayan, Jungtaek Lim, and Sid Wagle. > > > Bugs: AMBARI-17080 > https://issues.apache.org/jira/browse/AMBARI-17080 > > > Repository: ambari > > > Description > --- > > Support 2 storm sinks. > > > Diffs > - > > ambari-metrics/ambari-metrics-assembly/pom.xml 5e2d819 > ambari-metrics/ambari-metrics-assembly/src/main/assembly/sink-windows.xml > e82d2d4 > ambari-metrics/ambari-metrics-assembly/src/main/assembly/sink.xml 4a3b7c5 > ambari-metrics/ambari-metrics-storm-sink-legacy/pom.xml PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/assemblies/empty.xml > PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java > PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java > PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java > PRE-CREATION > ambari-metrics/ambari-metrics-storm-sink/pom.xml 1591d39 > > ambari-metrics/ambari-metrics-storm-sink/src/main/conf/storm-metrics2.properties.j2 > 4553224 > > ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/NumberUtil.java > PRE-CREATION > > ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java > ab5f1e4 > > ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java > 6ab12e1 > > ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java > e582a95 > ambari-metrics/pom.xml 7221ab5 > > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/storm-site.xml > 0d029e8 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/params_linux.py > 073bb1c > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/storm.py > 2d50767 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/ui_server.py > 6551067 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/templates/storm-metrics2.properties.j2 > 9acf173 > > ambari-server/src/main/resources/stacks/HDP/2.5/services/STORM/configuration/storm-site.xml > f3bbce8 > > Diff: https://reviews.apache.org/r/49514/diff/ > > > Testing > --- > > Unit testst passed > > > Thanks, > > Dmytro Sen > >
Re: Review Request 64513: Update the default storm log search configuration (use wildcards)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64513/#review193486 --- Ship it! Is it safe to put same configuration value from Storm logfeeder config? Then LGTM. - Jungtaek Lim On 12월 11, 2017, 11:47 오후, Oliver Szabo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64513/ > --- > > (Updated 12월 11, 2017, 11:47 오후) > > > Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Sid Wagle. > > > Bugs: AMBARI-22630 > https://issues.apache.org/jira/browse/AMBARI-22630 > > > Repository: ambari > > > Description > --- > > set the storm config defaults for logseach as other teams needs it. > > this patch is for branch-2.6, but if cannot be delivered in 2.6.1, that can > be used in a hotfix > > > Diffs > - > > > ambari-server/src/main/resources/common-services/STORM/0.9.1/configuration/storm-logsearch-conf.xml > d485017 > > > Diff: https://reviews.apache.org/r/64513/diff/1/ > > > Testing > --- > > not yet. > > > Thanks, > > Oliver Szabo > >
Review Request 64531: AMBARI-22633 MapDate provides the date incorrectly when Filter is cloned and used in multi-threads
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64531/ --- Review request for Ambari, Miklos Gergely, Oliver Szabo, and Sid Wagle. Bugs: AMBARI-22633 https://issues.apache.org/jira/browse/AMBARI-22633 Repository: ambari Description --- In AMBARI-22600, we cloned the Filter instance to assign the instance per child thread, which also clones the map of Mapper. Other Mapper implements only have String(s) type of fields hence thread-safe, but MapperDate has SimpleDateFormat type of fields which is known as non thread-safe, so cloning the Filter which has one or more MapperDate and using them concurrently makes multi-threads issue. Replacing SimpleDateFormat with FastDateFormat in commons-lang3 would resolve the issue. Diffs - ambari-logsearch/ambari-logsearch-logfeeder/pom.xml a06ae902fc ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/mapper/MapperDate.java 6a7fad7b70 Diff: https://reviews.apache.org/r/64531/diff/1/ Testing --- - existing unit tests - manual tests (manually deploying to cluster) Thanks, Jungtaek Lim
Review Request 64836: AMBARI-22696 Whitelist execute latency from Storm Ambari metrics (trunk)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64836/ --- Review request for Ambari, Arun Mahadevan and Aravindan Vijayan. Bugs: AMBARI-22696 https://issues.apache.org/jira/browse/AMBARI-22696 Repository: ambari Description --- We have an use case which leverages execute latency along with process latency. Given that we only register process latency as whitelist, we would need to add execute latency as whitelist as well. Diffs - ambari-server/src/main/resources/common-services/STORM/1.0.1.3.0/configuration/storm-site.xml 6b97fb6e41 ambari-server/src/main/resources/common-services/STORM/1.0.1.3.0/service_advisor.py 1d6bbe0f9f ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/storm-site.xml 558beaf8d9 ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 6f409cdfd1 ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py cc6bd9dca4 Diff: https://reviews.apache.org/r/64836/diff/1/ Testing --- Manually applied the change to configuration of test cluster, and confirmed it works. It doesn't mean I have tested it with upgrading, so please consider verifying it works well with various upgrade cases. Thanks, Jungtaek Lim
Review Request 64837: AMBARI-22696 Whitelist execute latency from Storm Ambari metrics (branch-2.6)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64837/ --- Review request for Ambari, Arun Mahadevan and Aravindan Vijayan. Bugs: AMBARI-22696 https://issues.apache.org/jira/browse/AMBARI-22696 Repository: ambari Description --- We have an use case which leverages execute latency along with process latency. Given that we only register process latency as whitelist, we would need to add execute latency as whitelist as well. Diffs - ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/storm-site.xml 558beaf8d9 ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 4187dc9b55 ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 8f27d79375 Diff: https://reviews.apache.org/r/64837/diff/1/ Testing --- Manually applied the change to configuration of test cluster, and confirmed it works. It doesn't mean I have tested it with upgrading, so please consider verifying it works well with various upgrade cases. Thanks, Jungtaek Lim
Re: Review Request 64837: AMBARI-22696 Whitelist execute latency from Storm Ambari metrics (branch-2.6)
> On 12월 28, 2017, 1:31 오전, Aravindan Vijayan wrote: > > These changes might be needed in > > ambari/ambari-server/src/main/resources/common-services/STORM/1.0.1.3.0/service_advisor.py > > as well. That file only exists on trunk branch. I've also published another review board for trunk branch. - Jungtaek --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64837/#review194559 --- On 12월 25, 2017, 4:38 오전, Jungtaek Lim wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64837/ > --- > > (Updated 12월 25, 2017, 4:38 오전) > > > Review request for Ambari, Arun Mahadevan and Aravindan Vijayan. > > > Bugs: AMBARI-22696 > https://issues.apache.org/jira/browse/AMBARI-22696 > > > Repository: ambari > > > Description > --- > > We have an use case which leverages execute latency along with process > latency. Given that we only register process latency as whitelist, we would > need to add execute latency as whitelist as well. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/storm-site.xml > 558beaf8d9 > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py > 4187dc9b55 > ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py > 8f27d79375 > > > Diff: https://reviews.apache.org/r/64837/diff/1/ > > > Testing > --- > > Manually applied the change to configuration of test cluster, and confirmed > it works. > > It doesn't mean I have tested it with upgrading, so please consider verifying > it works well with various upgrade cases. > > > Thanks, > > Jungtaek Lim > >
