Re: Review Request 45518: AMBARI-15638 : AMS Sum Calculation Incorrect

2016-03-31 Thread Sumit Mohanty 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45518/#review126495
---


Ship it!




Ship It!

- Sumit Mohanty


On March 31, 2016, 9:35 p.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45518/
> ---
> 
> (Updated March 31, 2016, 9:35 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-15638
> https://issues.apache.org/jira/browse/AMBARI-15638
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Sum Calculation is incorrect when the time range is more than 2 hrs.
> This issue affects all metrics that are queried with the "sum" aggregator 
> (with or without hostname being specified) for more than a 2hr time-range.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AggregatorUtils.java
>  ce79b6f 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricClusterAggregatorSecond.java
>  9354bc3 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricReadHelper.java
>  846ae92 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
>  e67a5b8 
> 
> Diff: https://reviews.apache.org/r/45518/diff/
> 
> 
> Testing
> ---
> 
> Manually tested. Unit tests pass.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 45518: AMBARI-15638 : AMS Sum Calculation Incorrect

2016-03-31 Thread Sumit Mohanty 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45518/#review126494
---




ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AggregatorUtils.java
 (line 46)


It may not always be correct to ignore 0 as a datapoint.


- Sumit Mohanty


On March 31, 2016, 9:35 p.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45518/
> ---
> 
> (Updated March 31, 2016, 9:35 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-15638
> https://issues.apache.org/jira/browse/AMBARI-15638
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Sum Calculation is incorrect when the time range is more than 2 hrs.
> This issue affects all metrics that are queried with the "sum" aggregator 
> (with or without hostname being specified) for more than a 2hr time-range.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AggregatorUtils.java
>  ce79b6f 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricClusterAggregatorSecond.java
>  9354bc3 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricReadHelper.java
>  846ae92 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
>  e67a5b8 
> 
> Diff: https://reviews.apache.org/r/45518/diff/
> 
> 
> Testing
> ---
> 
> Manually tested. Unit tests pass.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 45518: AMBARI-15638 : AMS Sum Calculation Incorrect

2016-03-31 Thread Aravindan Vijayan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45518/
---

(Updated March 31, 2016, 9:35 p.m.)


Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.


Changes
---

A couple of more important changes.

1. Change aggregator UPSERT statement to select > startTime and <= endTime 
instead of >=startTime and < endTime, since we do a right align of aggregated 
data.
2. Only Non Zero values should contribute to Counts during the insertion in 
METRIC_RECORD as well as time slicing for METRIC_AGGREGATE.


Bugs: AMBARI-15638
https://issues.apache.org/jira/browse/AMBARI-15638


Repository: ambari


Description
---

Sum Calculation is incorrect when the time range is more than 2 hrs.
This issue affects all metrics that are queried with the "sum" aggregator (with 
or without hostname being specified) for more than a 2hr time-range.


Diffs (updated)
-

  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AggregatorUtils.java
 ce79b6f 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricClusterAggregatorSecond.java
 9354bc3 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricReadHelper.java
 846ae92 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
 e67a5b8 

Diff: https://reviews.apache.org/r/45518/diff/


Testing
---

Manually tested. Unit tests pass.


Thanks,

Aravindan Vijayan

Re: Review Request 45518: AMBARI-15638 : AMS Sum Calculation Incorrect

2016-03-31 Thread Dmytro Sen 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45518/#review126321
---


Ship it!




Ship It!

- Dmytro Sen


On Март 31, 2016, 1:36 д.п., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45518/
> ---
> 
> (Updated Март 31, 2016, 1:36 д.п.)
> 
> 
> Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-15638
> https://issues.apache.org/jira/browse/AMBARI-15638
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Sum Calculation is incorrect when the time range is more than 2 hrs.
> This issue affects all metrics that are queried with the "sum" aggregator 
> (with or without hostname being specified) for more than a 2hr time-range.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricReadHelper.java
>  846ae92 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
>  e67a5b8 
> 
> Diff: https://reviews.apache.org/r/45518/diff/
> 
> 
> Testing
> ---
> 
> Manually tested. Unit tests pass.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 45518: AMBARI-15638 : AMS Sum Calculation Incorrect

2016-03-31 Thread Aravindan Vijayan 


> On March 31, 2016, 10:36 a.m., Dmytro Sen wrote:
> > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricReadHelper.java,
> >  line 75
> > 
> >
> > sum divided by count is an average value, but not sum?

DS, that was an intentional change in this patch.

While performing aggregation from METRIC_AGGREGATE to METRIC_AGGREGATE_MINUTE, 
we simply sum all the METRIC_SUM column values in the METRIC_AGGREGATE table, 
which has no real meaning to the user for most metrics.

Let's consider a cluster where there are HBase regionservers on 2 hosts with 1 
and 2 regions respectively. For example, when the user requests for something 
like "Sum of the regionserver count metric across hosts for the last 3 hours", 
we pick it up from the METRIC_AGGREGATE_MINUTE table.  Hence, we will get a 
value of 30 (3 multiplied by 10 30-second slices in the 5 minute window got 
from METRIC_AGGREGATE). Rather we would want the same "3", which is the sum of 
the regions in the cluster. So, there are 2 levels of aggregation - one across 
timeseries (hosts) and other across hosts (downsampling). We have decided to go 
with AVG for the downsampling method by default, and plan to make it 
configurable in the future.

Similarly, for METRIC_RECORD table, we have changed it in the read path.


- Aravindan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45518/#review126290
---


On March 31, 2016, 1:36 a.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45518/
> ---
> 
> (Updated March 31, 2016, 1:36 a.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-15638
> https://issues.apache.org/jira/browse/AMBARI-15638
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Sum Calculation is incorrect when the time range is more than 2 hrs.
> This issue affects all metrics that are queried with the "sum" aggregator 
> (with or without hostname being specified) for more than a 2hr time-range.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricReadHelper.java
>  846ae92 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
>  e67a5b8 
> 
> Diff: https://reviews.apache.org/r/45518/diff/
> 
> 
> Testing
> ---
> 
> Manually tested. Unit tests pass.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 45518: AMBARI-15638 : AMS Sum Calculation Incorrect

2016-03-31 Thread Dmytro Sen 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45518/#review126290
---




ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricReadHelper.java
 (line 75)


sum divided by count is an average value, but not sum?


- Dmytro Sen


On Март 31, 2016, 1:36 д.п., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45518/
> ---
> 
> (Updated Март 31, 2016, 1:36 д.п.)
> 
> 
> Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-15638
> https://issues.apache.org/jira/browse/AMBARI-15638
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Sum Calculation is incorrect when the time range is more than 2 hrs.
> This issue affects all metrics that are queried with the "sum" aggregator 
> (with or without hostname being specified) for more than a 2hr time-range.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricReadHelper.java
>  846ae92 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
>  e67a5b8 
> 
> Diff: https://reviews.apache.org/r/45518/diff/
> 
> 
> Testing
> ---
> 
> Manually tested. Unit tests pass.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Review Request 45518: AMBARI-15638 : AMS Sum Calculation Incorrect

2016-03-30 Thread Aravindan Vijayan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45518/
---

Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.


Bugs: AMBARI-15638
https://issues.apache.org/jira/browse/AMBARI-15638


Repository: ambari


Description
---

Sum Calculation is incorrect when the time range is more than 2 hrs.
This issue affects all metrics that are queried with the "sum" aggregator (with 
or without hostname being specified) for more than a 2hr time-range.


Diffs
-

  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricReadHelper.java
 846ae92 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
 e67a5b8 

Diff: https://reviews.apache.org/r/45518/diff/


Testing
---

Manually tested. Unit tests pass.


Thanks,

Aravindan Vijayan