Re: Review Request 52842: AMBARI-18593 : Provide ability to use downsampling function on certain metrics like client side topN

[Sid Wagle]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52842/#review153452
---


Ship it!




Ship It!

- Sid Wagle


On Oct. 20, 2016, 8:41 p.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52842/
> ---
> 
> (Updated Oct. 20, 2016, 8:41 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Sen, Sumit Mohanty, 
> and Sid Wagle.
> 
> 
> Bugs: AMBARI-18593
> https://issues.apache.org/jira/browse/AMBARI-18593
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> HDFS exposes top user activity broken down by operations in jmx (nntop).
> These metrics should be captured in AMS and exposed in Grafana's HDFS 
> dashboards.
> 
> Downsampling should likely be a function like MIN, MAX, AVG, SUM of 
> underlying timeseries specified from the client.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java
>  ba7807b 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/CustomDownSampler.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerUtils.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TopNDownSampler.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricClusterAggregator.java
>  c056d79 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricHostAggregator.java
>  118c695 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
>  177e444 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/TopNCondition.java
>  f7060e0 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerTest.java
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
>  b6e5da9 
> 
> Diff: https://reviews.apache.org/r/52842/diff/
> 
> 
> Testing
> ---
> 
> Manually tested.
> Unit tests added.
> mvn clean test on ambari-metrics pending.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 52842: AMBARI-18593 : Provide ability to use downsampling function on certain metrics like client side topN

[Aravindan Vijayan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52842/
---

(Updated Oct. 20, 2016, 8:41 p.m.)


Review request for Ambari, Alejandro Fernandez, Dmytro Sen, Sumit Mohanty, and 
Sid Wagle.


Changes
---

Addressed review comments.


Bugs: AMBARI-18593
https://issues.apache.org/jira/browse/AMBARI-18593


Repository: ambari


Description
---

HDFS exposes top user activity broken down by operations in jmx (nntop).
These metrics should be captured in AMS and exposed in Grafana's HDFS 
dashboards.

Downsampling should likely be a function like MIN, MAX, AVG, SUM of underlying 
timeseries specified from the client.


Diffs (updated)
-

  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java
 ba7807b 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/CustomDownSampler.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerUtils.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TopNDownSampler.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricClusterAggregator.java
 c056d79 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricHostAggregator.java
 118c695 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
 177e444 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/TopNCondition.java
 f7060e0 
  
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerTest.java
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
 b6e5da9 

Diff: https://reviews.apache.org/r/52842/diff/


Testing
---

Manually tested.
Unit tests added.
mvn clean test on ambari-metrics pending.


Thanks,

Aravindan Vijayan

Re: Review Request 52842: AMBARI-18593 : Provide ability to use downsampling function on certain metrics like client side topN

[Aravindan Vijayan]

> On Oct. 20, 2016, 2:01 a.m., Sid Wagle wrote:
> > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TopNDownSampler.java,
> >  line 96
> > 
> >
> > The only problem I see here is we loose context of what topN funtion 
> > was applied if we copy same value everywhere. What about only for the 
> > column = function and rest as NULL?

I am ok with setting "NULL" values for columns that are not computed through 
the downsampling. But, I feel that the downsampled metric value should be set 
in the AVG column, which is returned by default when the metric is requested. 
Even though we do a Top N by Max value for the downsampling function, the 
computed value will be in the default value column, and not the MAX column. 

The user's query will be like "Get me the Top 3 op-user combination for every 5 
min slice the last 6 hours". The query need not explicitly request "MAX" of the 
metric, since that is basically what the topn downsampling configs dictated.


- Aravindan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52842/#review153330
---


On Oct. 18, 2016, 6:35 p.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52842/
> ---
> 
> (Updated Oct. 18, 2016, 6:35 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Sen, Sumit Mohanty, 
> and Sid Wagle.
> 
> 
> Bugs: AMBARI-18593
> https://issues.apache.org/jira/browse/AMBARI-18593
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> HDFS exposes top user activity broken down by operations in jmx (nntop).
> These metrics should be captured in AMS and exposed in Grafana's HDFS 
> dashboards.
> 
> Downsampling should likely be a function like MIN, MAX, AVG, SUM of 
> underlying timeseries specified from the client.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java
>  ba7807b 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/CustomDownSampler.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerUtils.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TopNDownSampler.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricClusterAggregator.java
>  c056d79 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricHostAggregator.java
>  118c695 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
>  177e444 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/TopNCondition.java
>  f7060e0 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerTest.java
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
>  b6e5da9 
> 
> Diff: https://reviews.apache.org/r/52842/diff/
> 
> 
> Testing
> ---
> 
> Manually tested.
> Unit tests added.
> mvn clean test on ambari-metrics pending.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 52842: AMBARI-18593 : Provide ability to use downsampling function on certain metrics like client side topN

[Sid Wagle]

> On Oct. 20, 2016, 2:01 a.m., Sid Wagle wrote:
> > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml,
> >  line 686
> > 
> >
> > 1. Why %.user=% vs op=% ?
> > 2. I was expecting op=% vs op=_%, is that special syntx?
> 
> Aravindan Vijayan wrote:
> 1. We don't want to include a metric like 
> dfs.NNTopUserOpCounts.windowMs=6.op=getFileInfo.user=yarn and exclude a 
> summed up metric like dfs.NNTopUserOpCounts.windowMs=6.op=* in our Top N 
> calculation. 
> 2. op=% allows op=* , whereas op=__* does not. The small yet reasonable 
> hitch is that an op name should have at least 2 characters in length.
> 
> Aravindan Vijayan wrote:
> In the first point, I meant "We want to include...", not "We don't want 
> to include".

Got it !


- Sid


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52842/#review153330
---


On Oct. 18, 2016, 6:35 p.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52842/
> ---
> 
> (Updated Oct. 18, 2016, 6:35 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Sen, Sumit Mohanty, 
> and Sid Wagle.
> 
> 
> Bugs: AMBARI-18593
> https://issues.apache.org/jira/browse/AMBARI-18593
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> HDFS exposes top user activity broken down by operations in jmx (nntop).
> These metrics should be captured in AMS and exposed in Grafana's HDFS 
> dashboards.
> 
> Downsampling should likely be a function like MIN, MAX, AVG, SUM of 
> underlying timeseries specified from the client.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java
>  ba7807b 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/CustomDownSampler.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerUtils.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TopNDownSampler.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricClusterAggregator.java
>  c056d79 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricHostAggregator.java
>  118c695 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
>  177e444 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/TopNCondition.java
>  f7060e0 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerTest.java
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
>  b6e5da9 
> 
> Diff: https://reviews.apache.org/r/52842/diff/
> 
> 
> Testing
> ---
> 
> Manually tested.
> Unit tests added.
> mvn clean test on ambari-metrics pending.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 52842: AMBARI-18593 : Provide ability to use downsampling function on certain metrics like client side topN

[Aravindan Vijayan]

> On Oct. 20, 2016, 2:01 a.m., Sid Wagle wrote:
> > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml,
> >  line 686
> > 
> >
> > 1. Why %.user=% vs op=% ?
> > 2. I was expecting op=% vs op=_%, is that special syntx?
> 
> Aravindan Vijayan wrote:
> 1. We don't want to include a metric like 
> dfs.NNTopUserOpCounts.windowMs=6.op=getFileInfo.user=yarn and exclude a 
> summed up metric like dfs.NNTopUserOpCounts.windowMs=6.op=* in our Top N 
> calculation. 
> 2. op=% allows op=* , whereas op=__* does not. The small yet reasonable 
> hitch is that an op name should have at least 2 characters in length.

In the first point, I meant "We want to include...", not "We don't want to 
include".


- Aravindan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52842/#review153330
---


On Oct. 18, 2016, 6:35 p.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52842/
> ---
> 
> (Updated Oct. 18, 2016, 6:35 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Sen, Sumit Mohanty, 
> and Sid Wagle.
> 
> 
> Bugs: AMBARI-18593
> https://issues.apache.org/jira/browse/AMBARI-18593
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> HDFS exposes top user activity broken down by operations in jmx (nntop).
> These metrics should be captured in AMS and exposed in Grafana's HDFS 
> dashboards.
> 
> Downsampling should likely be a function like MIN, MAX, AVG, SUM of 
> underlying timeseries specified from the client.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java
>  ba7807b 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/CustomDownSampler.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerUtils.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TopNDownSampler.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricClusterAggregator.java
>  c056d79 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricHostAggregator.java
>  118c695 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
>  177e444 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/TopNCondition.java
>  f7060e0 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerTest.java
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
>  b6e5da9 
> 
> Diff: https://reviews.apache.org/r/52842/diff/
> 
> 
> Testing
> ---
> 
> Manually tested.
> Unit tests added.
> mvn clean test on ambari-metrics pending.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 52842: AMBARI-18593 : Provide ability to use downsampling function on certain metrics like client side topN

[Aravindan Vijayan]

> On Oct. 20, 2016, 2:01 a.m., Sid Wagle wrote:
> > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml,
> >  line 686
> > 
> >
> > 1. Why %.user=% vs op=% ?
> > 2. I was expecting op=% vs op=_%, is that special syntx?

1. We don't want to include a metric like 
dfs.NNTopUserOpCounts.windowMs=6.op=getFileInfo.user=yarn and exclude a 
summed up metric like dfs.NNTopUserOpCounts.windowMs=6.op=* in our Top N 
calculation. 
2. op=% allows op=* , whereas op=__* does not. The small yet reasonable hitch 
is that an op name should have at least 2 characters in length.


- Aravindan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52842/#review153330
---


On Oct. 18, 2016, 6:35 p.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52842/
> ---
> 
> (Updated Oct. 18, 2016, 6:35 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Sen, Sumit Mohanty, 
> and Sid Wagle.
> 
> 
> Bugs: AMBARI-18593
> https://issues.apache.org/jira/browse/AMBARI-18593
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> HDFS exposes top user activity broken down by operations in jmx (nntop).
> These metrics should be captured in AMS and exposed in Grafana's HDFS 
> dashboards.
> 
> Downsampling should likely be a function like MIN, MAX, AVG, SUM of 
> underlying timeseries specified from the client.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java
>  ba7807b 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/CustomDownSampler.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerUtils.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TopNDownSampler.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricClusterAggregator.java
>  c056d79 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricHostAggregator.java
>  118c695 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
>  177e444 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/TopNCondition.java
>  f7060e0 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerTest.java
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
>  b6e5da9 
> 
> Diff: https://reviews.apache.org/r/52842/diff/
> 
> 
> Testing
> ---
> 
> Manually tested.
> Unit tests added.
> mvn clean test on ambari-metrics pending.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 52842: AMBARI-18593 : Provide ability to use downsampling function on certain metrics like client side topN

[Sid Wagle]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52842/#review153330
---




ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/CustomDownSampler.java
 (line 29)


Add come javadoc.



ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerUtils.java
 (line 52)


formatting.



ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TopNDownSampler.java
 (line 96)


The only problem I see here is we loose context of what topN funtion was 
applied if we copy same value everywhere. What about only for the column = 
function and rest as NULL?



ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
 (line 686)


1. Why %.user=% vs op=% ?
2. I was expecting op=% vs op=_%, is that special syntx?


- Sid Wagle


On Oct. 18, 2016, 6:35 p.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52842/
> ---
> 
> (Updated Oct. 18, 2016, 6:35 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Sen, Sumit Mohanty, 
> and Sid Wagle.
> 
> 
> Bugs: AMBARI-18593
> https://issues.apache.org/jira/browse/AMBARI-18593
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> HDFS exposes top user activity broken down by operations in jmx (nntop).
> These metrics should be captured in AMS and exposed in Grafana's HDFS 
> dashboards.
> 
> Downsampling should likely be a function like MIN, MAX, AVG, SUM of 
> underlying timeseries specified from the client.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java
>  ba7807b 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/CustomDownSampler.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerUtils.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TopNDownSampler.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricClusterAggregator.java
>  c056d79 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricHostAggregator.java
>  118c695 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
>  177e444 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/TopNCondition.java
>  f7060e0 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerTest.java
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
>  b6e5da9 
> 
> Diff: https://reviews.apache.org/r/52842/diff/
> 
> 
> Testing
> ---
> 
> Manually tested.
> Unit tests added.
> mvn clean test on ambari-metrics pending.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 52842: AMBARI-18593 : Provide ability to use downsampling function on certain metrics like client side topN

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52842/#review153154
---


Ship it!




Ship It!

- Alejandro Fernandez


On Oct. 18, 2016, 6:35 p.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52842/
> ---
> 
> (Updated Oct. 18, 2016, 6:35 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Sen, Sumit Mohanty, 
> and Sid Wagle.
> 
> 
> Bugs: AMBARI-18593
> https://issues.apache.org/jira/browse/AMBARI-18593
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> HDFS exposes top user activity broken down by operations in jmx (nntop).
> These metrics should be captured in AMS and exposed in Grafana's HDFS 
> dashboards.
> 
> Downsampling should likely be a function like MIN, MAX, AVG, SUM of 
> underlying timeseries specified from the client.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java
>  ba7807b 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/CustomDownSampler.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerUtils.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TopNDownSampler.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricClusterAggregator.java
>  c056d79 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricHostAggregator.java
>  118c695 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
>  177e444 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/TopNCondition.java
>  f7060e0 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerTest.java
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
>  b6e5da9 
> 
> Diff: https://reviews.apache.org/r/52842/diff/
> 
> 
> Testing
> ---
> 
> Manually tested.
> Unit tests added.
> mvn clean test on ambari-metrics pending.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 52842: AMBARI-18593 : Provide ability to use downsampling function on certain metrics like client side topN

[Aravindan Vijayan]

> On Oct. 18, 2016, 6:01 p.m., Alejandro Fernandez wrote:
> > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerUtils.java,
> >  line 55
> > 
> >
> > May want to remove any leading/trailing whitespace after splitting.

Done.


> On Oct. 18, 2016, 6:01 p.m., Alejandro Fernandez wrote:
> > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerUtils.java,
> >  line 103
> > 
> >
> > Can "topn" be a constant, plus the other strings used in these files? 
> > Case insensitive equals?

Done. Will move to an enum if more downsampling types come up in the future.


> On Oct. 18, 2016, 6:01 p.m., Alejandro Fernandez wrote:
> > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/TopNCondition.java,
> >  line 112
> > 
> >
> > May still be good to  put " ASC" in the else clause to make it clear.

Done.


- Aravindan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52842/#review153117
---


On Oct. 18, 2016, 6:34 p.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52842/
> ---
> 
> (Updated Oct. 18, 2016, 6:34 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-18593
> https://issues.apache.org/jira/browse/AMBARI-18593
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> HDFS exposes top user activity broken down by operations in jmx (nntop).
> These metrics should be captured in AMS and exposed in Grafana's HDFS 
> dashboards.
> 
> Downsampling should likely be a function like MIN, MAX, AVG, SUM of 
> underlying timeseries specified from the client.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java
>  ba7807b 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/CustomDownSampler.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerUtils.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TopNDownSampler.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricClusterAggregator.java
>  c056d79 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricHostAggregator.java
>  118c695 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
>  177e444 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/TopNCondition.java
>  f7060e0 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerTest.java
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
>  b6e5da9 
> 
> Diff: https://reviews.apache.org/r/52842/diff/
> 
> 
> Testing
> ---
> 
> Manually tested.
> Unit tests added.
> mvn clean test on ambari-metrics pending.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 52842: AMBARI-18593 : Provide ability to use downsampling function on certain metrics like client side topN

[Aravindan Vijayan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52842/
---

(Updated Oct. 18, 2016, 6:34 p.m.)


Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.


Bugs: AMBARI-18593
https://issues.apache.org/jira/browse/AMBARI-18593


Repository: ambari


Description
---

HDFS exposes top user activity broken down by operations in jmx (nntop).
These metrics should be captured in AMS and exposed in Grafana's HDFS 
dashboards.

Downsampling should likely be a function like MIN, MAX, AVG, SUM of underlying 
timeseries specified from the client.


Diffs (updated)
-

  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java
 ba7807b 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/CustomDownSampler.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerUtils.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TopNDownSampler.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricClusterAggregator.java
 c056d79 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricHostAggregator.java
 118c695 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
 177e444 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/TopNCondition.java
 f7060e0 
  
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerTest.java
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
 b6e5da9 

Diff: https://reviews.apache.org/r/52842/diff/


Testing
---

Manually tested.
Unit tests added.
mvn clean test on ambari-metrics pending.


Thanks,

Aravindan Vijayan

Re: Review Request 52842: AMBARI-18593 : Provide ability to use downsampling function on certain metrics like client side topN

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52842/#review153117
---




ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java
 (line 389)


The log statements will already contain the timestamp, no need for new 
Date().



ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerUtils.java
 (line 55)


May want to remove any leading/trailing whitespace after splitting.



ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerUtils.java
 (line 103)


Can "topn" be a constant, plus the other strings used in these files? Case 
insensitive equals?



ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/TopNCondition.java
 (line 112)


May still be good to  put " ASC" in the else clause to make it clear.


- Alejandro Fernandez


On Oct. 13, 2016, 8:40 p.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52842/
> ---
> 
> (Updated Oct. 13, 2016, 8:40 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-18593
> https://issues.apache.org/jira/browse/AMBARI-18593
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> HDFS exposes top user activity broken down by operations in jmx (nntop).
> These metrics should be captured in AMS and exposed in Grafana's HDFS 
> dashboards.
> 
> Downsampling should likely be a function like MIN, MAX, AVG, SUM of 
> underlying timeseries specified from the client.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java
>  ba7807b 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/CustomDownSampler.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerUtils.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TopNDownSampler.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricClusterAggregator.java
>  c056d79 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricHostAggregator.java
>  118c695 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
>  177e444 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/TopNCondition.java
>  f7060e0 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerTest.java
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
>  c2aea8c 
> 
> Diff: https://reviews.apache.org/r/52842/diff/
> 
> 
> Testing
> ---
> 
> Manually tested.
> Unit tests added.
> mvn clean test on ambari-metrics pending.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Review Request 52842: AMBARI-18593 : Provide ability to use downsampling function on certain metrics like client side topN

[Aravindan Vijayan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52842/
---

Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.


Bugs: AMBARI-18593
https://issues.apache.org/jira/browse/AMBARI-18593


Repository: ambari


Description
---

HDFS exposes top user activity broken down by operations in jmx (nntop).
These metrics should be captured in AMS and exposed in Grafana's HDFS 
dashboards.

Downsampling should likely be a function like MIN, MAX, AVG, SUM of underlying 
timeseries specified from the client.


Diffs
-

  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java
 ba7807b 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/CustomDownSampler.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerUtils.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TopNDownSampler.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricClusterAggregator.java
 c056d79 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricHostAggregator.java
 118c695 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
 177e444 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/TopNCondition.java
 f7060e0 
  
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/DownSamplerTest.java
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
 c2aea8c 

Diff: https://reviews.apache.org/r/52842/diff/


Testing
---

Manually tested.
Unit tests added.
mvn clean test on ambari-metrics pending.


Thanks,

Aravindan Vijayan