[
https://issues.apache.org/jira/browse/HDFS-16949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701368#comment-17701368
]
ASF GitHub Bot commented on HDFS-16949:
---------------------------------------
mkuchenbecker commented on code in PR #5486:
URL: https://github.com/apache/hadoop/pull/5486#discussion_r1139322336
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableQuantiles.java:
##########
@@ -52,6 +52,7 @@ public class MutableQuantiles extends MutableMetric {
new Quantile(0.75, 0.025), new Quantile(0.90, 0.010),
new Quantile(0.95, 0.005), new Quantile(0.99, 0.001) };
+ protected boolean inverseQuantiles = false;
Review Comment:
`private final boolean`
You will also need to set it in the constructor.
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/util/SampleQuantiles.java:
##########
@@ -243,7 +245,12 @@ synchronized public Map<Quantile, Long> snapshot() {
Map<Quantile, Long> values = new TreeMap<Quantile, Long>();
for (int i = 0; i < quantiles.length; i++) {
- values.put(quantiles[i], query(quantiles[i].quantile));
+ /* eg : effectiveQuantile for 0.99 with inverseQuantiles will be 0.01.
+ For inverse quantiles higher numeric value is better and hence we want
+ to query from the opposite end of the sorted sample
+ */
+ double effectiveQuantile = inverseQuantiles ? 1 - quantiles[i].quantile
: quantiles[i].quantile;
Review Comment:
Shouldn't this come after the next line?
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/util/SampleQuantiles.java:
##########
@@ -243,7 +245,12 @@ synchronized public Map<Quantile, Long> snapshot() {
Map<Quantile, Long> values = new TreeMap<Quantile, Long>();
for (int i = 0; i < quantiles.length; i++) {
- values.put(quantiles[i], query(quantiles[i].quantile));
+ /* eg : effectiveQuantile for 0.99 with inverseQuantiles will be 0.01.
Review Comment:
Leaning into OOO: I might make an inversequantile class that overrides this
function to keep it simple rather than overloading quantile with multiple
definitions.
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/util/SampleQuantiles.java:
##########
@@ -243,7 +245,12 @@ synchronized public Map<Quantile, Long> snapshot() {
Map<Quantile, Long> values = new TreeMap<Quantile, Long>();
for (int i = 0; i < quantiles.length; i++) {
- values.put(quantiles[i], query(quantiles[i].quantile));
+ /* eg : effectiveQuantile for 0.99 with inverseQuantiles will be 0.01.
+ For inverse quantiles higher numeric value is better and hence we want
+ to query from the opposite end of the sorted sample
+ */
+ double effectiveQuantile = inverseQuantiles ? 1 - quantiles[i].quantile
: quantiles[i].quantile;
+ values.put(quantiles[i], query(effectiveQuantile));
Review Comment:
Wouldn't reversing list order traversal give you the same thing wihtout
altering the math?
> Update ReadTransferRate to ReadLatencyPerGB for effective percentile metrics
> ----------------------------------------------------------------------------
>
> Key: HDFS-16949
> URL: https://issues.apache.org/jira/browse/HDFS-16949
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Reporter: Ravindra Dingankar
> Assignee: Ravindra Dingankar
> Priority: Minor
> Labels: pull-request-available
> Fix For: 3.3.0, 3.4.0
>
>
> HDFS-16917 added ReadTransferRate quantiles to calculate the rate which data
> is read per unit of time.
> With percentiles the values are sorted in ascending order and hence for the
> transfer rate p90 gives us the value where 90 percent rates are lower
> (worse), p99 gives us the value where 99 percent values are lower (worse).
> Note that value(p90) < p(99) thus p99 is a better transfer rate as compared
> to p90.
> However as the percentile increases the value should become worse in order to
> know how good our system is.
> Hence instead of calculating the data read transfer rate, we should calculate
> it's inverse. We will instead calculate the time taken for a GB of data to be
> read. ( seconds / GB )
> After this the p90 value will give us 90 percentage of total values where the
> time taken is less than value(p90), similarly for p99 and others.
> Also p(90) < p(99) and here p(99) will become a worse value (taking more time
> each byte) as compared to p(90)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
