[
https://issues.apache.org/jira/browse/HDFS-16949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravindra Dingankar updated HDFS-16949:
--------------------------------------
Description:
HDFS-16917 added ReadTransferRate quantiles to calculate the rate which data is
read per unit of time.
With percentiles the values are sorted in ascending order and hence for the
transfer rate p90 gives us the value where 90 percent rates are lower (worse),
p99 gives us the value where 99 percent values are lower (worse).
Note that value(p90) < p(99) thus p99 is a better transfer rate as compared to
p90.
However as the percentile increases the value should become worse in order to
know how good our system is.
Hence instead of calculating the data read transfer rate, we should calculate
it's inverse. We will instead calculate the time taken for a GB of data to be
read. ( seconds / GB )
After this the p90 value will give us 90 percentage of total values where the
time taken is less than value(p90), similarly for p99 and others.
Also p(90) < p(99) and here p(99) will become a worse value (taking more time
each byte) as compared to p(90)
was:
HDFS-16917 added ReadTransferRate quantiles to calculate the rate which data is
read per unit of time.
With percentiles the values are sorted in ascending order and hence for the
transfer rate p90 gives us the value where 90 percent rates are lower (worse),
p99 gives us the value where 99 percent values are lower (worse).
Note that value(p90) < p(99) thus p99 is a better transfer rate as compared to
p90.
However as the percentile increases the value should become worse in order to
know how good our system is.
Hence instead of calculating the data read transfer rate, we should calculate
it's inverse. We will instead calculate the time taken for a byte of data to be
read. ( seconds / byte )
After this the p90 value will give us 90 percentage of total values where the
time taken is less than value(p90), similarly for p99 and others.
Also p(90) < p(99) and here p(99) will become a worse value (taking more time
each byte) as compared to p(90)
> Update ReadTransferRate to ReadLatencyPerByte for effective percentile metrics
> ------------------------------------------------------------------------------
>
> Key: HDFS-16949
> URL: https://issues.apache.org/jira/browse/HDFS-16949
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Reporter: Ravindra Dingankar
> Assignee: Ravindra Dingankar
> Priority: Minor
> Fix For: 3.3.0, 3.4.0
>
>
> HDFS-16917 added ReadTransferRate quantiles to calculate the rate which data
> is read per unit of time.
> With percentiles the values are sorted in ascending order and hence for the
> transfer rate p90 gives us the value where 90 percent rates are lower
> (worse), p99 gives us the value where 99 percent values are lower (worse).
> Note that value(p90) < p(99) thus p99 is a better transfer rate as compared
> to p90.
> However as the percentile increases the value should become worse in order to
> know how good our system is.
> Hence instead of calculating the data read transfer rate, we should calculate
> it's inverse. We will instead calculate the time taken for a GB of data to be
> read. ( seconds / GB )
> After this the p90 value will give us 90 percentage of total values where the
> time taken is less than value(p90), similarly for p99 and others.
> Also p(90) < p(99) and here p(99) will become a worse value (taking more time
> each byte) as compared to p(90)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
