rdingankar opened a new pull request, #5486:
URL: https://github.com/apache/hadoop/pull/5486
…ic value is better
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
Currently quantiles are used for latencies, where lower numeric value is
better.
Hence p90 gives us a value val(p90) such that 90% of our sample set has a
value better (lower) than val(p90)
However for metrics such as calculating transfer rates (eg :
[HDFS-16917](https://issues.apache.org/jira/browse/HDFS-16917) ) higher numeric
value is better. Thus for such metrics the current quantiles dont work.
For these metrics in order for p90 to give a value val(p90) where 90% of the
sample set is better (higher) than val(p90) we need to inverse the selection by
choosing a value at the (100 - 90)th location instead of the usual 90th
position.
Note: There will be error guarantees for percentiles as our quantile
implementation following [Cormode, Korn, Muthukrishnan, and Srivastava
algorithm](http://dimacs.rutgers.edu/~graham/pubs/papers/bquant-icde.pdf)
### How was this patch tested?
Results from UT testInverseQuantiles
Starting run 0
For quantile 0.500000 Expected 50000 with error 5000, estimated 50502
For quantile 0.750000 Expected 25000 with error 7500, estimated 27103
For quantile 0.900000 Expected 9999 with error 9000, estimated 12125
For quantile 0.950000 Expected 5000 with error 9500, estimated 8024
For quantile 0.990000 Expected 1000 with error 9900, estimated 5971
Starting run 1
For quantile 0.500000 Expected 50000 with error 5000, estimated 50796
For quantile 0.750000 Expected 25000 with error 7500, estimated 26340
For quantile 0.900000 Expected 9999 with error 9000, estimated 11428
For quantile 0.950000 Expected 5000 with error 9500, estimated 7980
For quantile 0.990000 Expected 1000 with error 9900, estimated 5421
Starting run 2
For quantile 0.500000 Expected 50000 with error 5000, estimated 50743
For quantile 0.750000 Expected 25000 with error 7500, estimated 26328
For quantile 0.900000 Expected 9999 with error 9000, estimated 12007
For quantile 0.950000 Expected 5000 with error 9500, estimated 7287
For quantile 0.990000 Expected 1000 with error 9900, estimated 5492
Starting run 3
For quantile 0.500000 Expected 50000 with error 5000, estimated 51313
For quantile 0.750000 Expected 25000 with error 7500, estimated 26258
For quantile 0.900000 Expected 9999 with error 9000, estimated 12024
For quantile 0.950000 Expected 5000 with error 9500, estimated 8160
For quantile 0.990000 Expected 1000 with error 9900, estimated 5570
Starting run 4
For quantile 0.500000 Expected 50000 with error 5000, estimated 50258
For quantile 0.750000 Expected 25000 with error 7500, estimated 25645
For quantile 0.900000 Expected 9999 with error 9000, estimated 11501
For quantile 0.950000 Expected 5000 with error 9500, estimated 7376
For quantile 0.990000 Expected 1000 with error 9900, estimated 5174
Starting run 5
For quantile 0.500000 Expected 50000 with error 5000, estimated 50718
For quantile 0.750000 Expected 25000 with error 7500, estimated 26589
For quantile 0.900000 Expected 9999 with error 9000, estimated 12808
For quantile 0.950000 Expected 5000 with error 9500, estimated 7920
For quantile 0.990000 Expected 1000 with error 9900, estimated 5635
Starting run 6
For quantile 0.500000 Expected 50000 with error 5000, estimated 50203
For quantile 0.750000 Expected 25000 with error 7500, estimated 25996
For quantile 0.900000 Expected 9999 with error 9000, estimated 12232
For quantile 0.950000 Expected 5000 with error 9500, estimated 7880
For quantile 0.990000 Expected 1000 with error 9900, estimated 5875
Starting run 7
For quantile 0.500000 Expected 50000 with error 5000, estimated 50423
For quantile 0.750000 Expected 25000 with error 7500, estimated 25862
For quantile 0.900000 Expected 9999 with error 9000, estimated 10842
For quantile 0.950000 Expected 5000 with error 9500, estimated 7702
For quantile 0.990000 Expected 1000 with error 9900, estimated 5371
Starting run 8
For quantile 0.500000 Expected 50000 with error 5000, estimated 50380
For quantile 0.750000 Expected 25000 with error 7500, estimated 26115
For quantile 0.900000 Expected 9999 with error 9000, estimated 12687
For quantile 0.950000 Expected 5000 with error 9500, estimated 8189
For quantile 0.990000 Expected 1000 with error 9900, estimated 6029
Starting run 9
For quantile 0.500000 Expected 50000 with error 5000, estimated 50548
For quantile 0.750000 Expected 25000 with error 7500, estimated 26427
For quantile 0.900000 Expected 9999 with error 9000, estimated 11489
For quantile 0.950000 Expected 5000 with error 9500, estimated 7639
For quantile 0.990000 Expected 1000 with error 9900, estimated 5315
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
