Lewis John McGibbney created NUTCH-3247:
-------------------------------------------
Summary: Fix UpdateHostDbReducer storing the median in every
percentile
Key: NUTCH-3247
URL: https://issues.apache.org/jira/browse/NUTCH-3247
Project: Nutch
Issue Type: Sub-task
Reporter: Lewis John McGibbney
{\{UpdateHostDbReducer.reduce}} builds a \{{TDigest}} per configured numeric
metadata field and writes one \{{HostDatum}} metadata entry per configured
percentile. \{{hostdb.percentiles}} defaults to \{{50,75,95,99}}
(\{{conf/nutch-default.xml}}). The write loop uses the median for every entry:
{code:java}
for (int i = 0; i < percentiles.length; i++) {
hostDatum.getMetaData().put(
new Text("pct" + Long.toString(percentiles[i]) + "." + entry.getKey()),
new FloatWritable((float) entry.getValue().quantile(0.5)));
}
{code}
{\{pct75.*}}, \{{pct95.*}}, and \{{pct99.*}} are copies of \{{pct50.*}}. The
digest itself is built correctly; \{{tdigest.add(metadataValue)}} is called per
value. The quantile argument is the bug. This is host metadata, not a Hadoop
counter, and it is not a host-score distribution. The score is stored on the
datum separately, a few lines above this loop.
h2. Proposal
Pass the configured percentile. \{{quantile}} takes a fraction from 0 to 1, and
the configuration is a percent, so the call is \{{quantile(percentiles[i] /
100.0)}}.
h2. Acceptance
* With \{{hostdb.percentiles}} at the default, a fixture whose values are not
uniform leaves \{{pct50}}, \{{pct75}}, \{{pct95}}, and \{{pct99}} different
from each other, and \{{pct50}} equal to \{{quantile(0.5)}}.
* A single-value fixture leaves every percentile equal to that value.
* A unit test covers both fixtures.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
