Samuel Verstraete created SOLR-18598:
----------------------------------------
Summary: Make the precision of distributed hll() and numBuckets
configurable
Key: SOLR-18598
URL: https://issues.apache.org/jira/browse/SOLR-18598
Project: Solr
Issue Type: Improvement
Components: Facet Module
Affects Versions: 10.0
Reporter: Samuel Verstraete
In a distributed request, shards return \{{hll()}} and \{{numBuckets}} as
hyper-log-log sketches with fixed parameters (\{{HLLAgg.HLLFactory}}: log2m 13,
regwidth 6, automatic explicit threshold). The count is exact up to 768 values
and an estimate above that, with a standard error of about 1.15%. On our data
(3 shards, 151 facets above 768 values) the difference with a single core goes
up to 2.9%. There is no way to change these parameters.
h3. Proposal
Two system properties, read with \{{EnvUtils}}, with today's values as defaults
so nothing changes unless they are set:
* \{{solr.facet.hll.log2m}} (default 13): the number of registers as a power of
2. Each step up divides the error by √2 and doubles the sketch (6KB at 13, 48KB
at 16).
* \{{solr.facet.hll.expthresh}} (default -1): up to 2^(expthresh - 1) values, a
sketch keeps their hashes and counts them exactly, at 8 bytes per value per
shard. The maximum, 18, counts up to 131,072 values exactly. -1 is today's
automatic threshold.
Measured with the HLL class, 3 shards (size per shard / extra merge time per
facet / error):
||Setting||5,000 values||100,000 values||
|default|6KB / - / 0.6%|6KB / - / 0.8%|
|log2m 16|16KB / - / exact|48KB / +1.7ms / 0.3%|
|expthresh 18|16KB / - / exact|320KB / +8.5ms / exact|
h3. Caveat
All nodes of a cluster need the same log2m: \{{HLL.union}} does not check that
two sketches are compatible (there is a TODO for it), so sketches with
different log2m merge into a wrong count or an error. A change of log2m
therefore needs a full restart, not a rolling one. A change of expthresh alone
is safe.
h3. Use case
We compare the facets of a sharded collection with those of a single core in an
automated test. With \{{solr.facet.hll.expthresh=18}}, all distinct counts of
our 3-shard test collection (up to about 97,000 values) match the single core
exactly.
A PR with the properties, a unit test and a note in the JSON facet API page of
the reference guide follows.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
