elangelo opened a new pull request, #5251: URL: https://github.com/apache/solr/pull/5251
https://issues.apache.org/jira/browse/SOLR-18598 # Description Shards return `hll()` and `numBuckets` as hyper-log-log sketches with fixed parameters (`HLLAgg.HLLFactory`: log2m 13, regwidth 6, automatic explicit threshold): exact up to 768 values, a standard error of about 1.15% above. This PR lets a cluster trade bigger shard responses for a more exact count. # Solution `HLLAgg.HLLFactory` reads two system properties through `EnvUtils`, with today's values as defaults, so nothing changes unless they are set: - `solr.facet.hll.log2m` (default 13): the number of registers as a power of 2. - `solr.facet.hll.expthresh` (default -1): up to 2^(expthresh - 1) values are counted exactly from their hashes; 18, the maximum of the HLL implementation, counts up to 131,072 values exactly. The coordinator takes the parameters from the shards' serialized sketches, so only the shards need the properties. All nodes need the same log2m, since `HLL.union` does not check compatibility; the reference guide says so. The JSON facet API page of the reference guide describes both properties next to the aggregation functions, and the `numBuckets` row says that it is an estimate in a distributed request. This change was written with the help of an AI coding assistant (Claude Code) and reviewed by me. # Tests `HLLAggTest` checks that the defaults count up to 768 values exactly, that `solr.facet.hll.expthresh=15` counts up to 16,384 values exactly (also after a serialize and union), and that `solr.facet.hll.log2m=16` raises the exact range to 6,144 values and keeps a 100,000-value estimate within 2%. `TestJsonFacets` passes, `./gradlew check -x test` passes, and the reference guide builds with its links checked. On a 3-shard collection with our full test data set (facets up to about 97,000 values), `-Dsolr.facet.hll.expthresh=18` makes every distinct count match a single core, against differences of up to 2.9% with the defaults. # Checklist - [x] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability. - [x] I have created a Jira issue and added the issue ID to my pull request title. - [x ] I have given Solr maintainers access to contribute to my PR branch. - [x] I have developed this patch against the `main` branch. - [x] I have run `./gradlew check`. - [x] I have added tests for my changes. - [x] I have added documentation for the Reference Guide - [x] I have added a changelog entry for my change -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
