[
https://issues.apache.org/jira/browse/HBASE-27224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Beaudreault updated HBASE-27224:
--------------------------------------
Status: Open (was: Patch Available)
MutableRangeHistogram did not work. It did improve the min/max as expected, but
the ranges I added using histogram.getCountAtOrBelow (like we do in jmx) were
completely inaccurate. Copying from my now-closed PR:
{quote}I realized that MutableRangeHistogram's buckets actually are very
inaccurate on the first call to `histogram.snapshot()`. The initial bins are
configured for very large ranges, and as `snapshot()` is called over time those
are resized to fit the actual data based on the distribution at that time. The
HistogramImpl.getQuantiles method does some complicated math to estimate the
quantiles despite incorrect bins, but the `getCountAtOrBelow` does not. I
thought about trying to account for that, but it seems overly complicated for
something that is used pretty much everywhere.
Instead I'm going to revert to using codahale metrics, fix to use
UniformDistribution, and add some supplemental range tracking just for the
HFilePrettyPrinter.
{quote}
Going to try to get another PR up later today.
> HFile tool statistic sampling produces misleading results
> ---------------------------------------------------------
>
> Key: HBASE-27224
> URL: https://issues.apache.org/jira/browse/HBASE-27224
> Project: HBase
> Issue Type: Improvement
> Reporter: Bryan Beaudreault
> Assignee: Bryan Beaudreault
> Priority: Major
> Labels: patch-available
>
> HFile tool uses codahale metrics for collecting statistics about key/values
> in an HFile. We recently had a case where the statistics printed out that the
> max row size was only 25k. This was confusing because I was seeing bucket
> cache allocation failures for blocks as large as 1.5mb.
> Digging in, I was able to find the large row using the "-p" argument (which
> was obviously very verbose). Once I found the row, I saw the vlen was listed
> as ~1.5mb which made much more sense.
> First thing I notice here is that default codahale metrics histogram is using
> ExponentiallyDecayingReservoir. This probably makes sense for a long-lived
> histogram, but the HFile tool is run at a point in time. It might be best to
> use UniformReservoir instead.
> Secondly, we do not need sampling for min/max. Let's supplement the histogram
> with our own calculation which is guaranteed to be accurate for the entirety
> of the file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)