[
https://issues.apache.org/jira/browse/SOLR-18598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SOLR-18598:
----------------------------------
Labels: pull-request-available (was: )
> Make the precision of distributed hll() and numBuckets configurable
> -------------------------------------------------------------------
>
> Key: SOLR-18598
> URL: https://issues.apache.org/jira/browse/SOLR-18598
> Project: Solr
> Issue Type: Improvement
> Components: Facet Module
> Affects Versions: 10.0
> Reporter: Samuel Verstraete
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In a distributed request, shards return \{{hll()}} and \{{numBuckets}} as
> hyper-log-log sketches with fixed parameters (\{{HLLAgg.HLLFactory}}: log2m
> 13, regwidth 6, automatic explicit threshold). The count is exact up to 768
> values and an estimate above that, with a standard error of about 1.15%. On
> our data (3 shards, 151 facets above 768 values) the difference with a single
> core goes up to 2.9%. There is no way to change these parameters.
> h3. Proposal
> Two system properties, read with \{{EnvUtils}}, with today's values as
> defaults so nothing changes unless they are set:
> * \{{solr.facet.hll.log2m}} (default 13): the number of registers as a power
> of 2. Each step up divides the error by √2 and doubles the sketch (6KB at 13,
> 48KB at 16).
> * \{{solr.facet.hll.expthresh}} (default -1): up to 2^(expthresh - 1) values,
> a sketch keeps their hashes and counts them exactly, at 8 bytes per value per
> shard. The maximum, 18, counts up to 131,072 values exactly. -1 is today's
> automatic threshold.
> Measured with the HLL class, 3 shards (size per shard / extra merge time per
> facet / error):
> ||Setting||5,000 values||100,000 values||
> |default|6KB / - / 0.6%|6KB / - / 0.8%|
> |log2m 16|16KB / - / exact|48KB / +1.7ms / 0.3%|
> |expthresh 18|16KB / - / exact|320KB / +8.5ms / exact|
> h3. Caveat
> All nodes of a cluster need the same log2m: \{{HLL.union}} does not check
> that two sketches are compatible (there is a TODO for it), so sketches with
> different log2m merge into a wrong count or an error. A change of log2m
> therefore needs a full restart, not a rolling one. A change of expthresh
> alone is safe.
> h3. Use case
> We compare the facets of a sharded collection with those of a single core in
> an automated test. With \{{solr.facet.hll.expthresh=18}}, all distinct counts
> of our 3-shard test collection (up to about 97,000 values) match the single
> core exactly.
> A PR with the properties, a unit test and a note in the JSON facet API page
> of the reference guide follows.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
