Samuel Verstraete created SOLR-18597:
----------------------------------------
Summary: JSON facet numBuckets with a prefix counts all values of
the matching docs in a distributed request
Key: SOLR-18597
URL: https://issues.apache.org/jira/browse/SOLR-18597
Project: Solr
Issue Type: Bug
Components: Facet Module
Affects Versions: 10.0
Reporter: Samuel Verstraete
In a distributed request, a terms facet with \{{prefix}} and
\{{numBuckets:true}} on a multi-valued field counts values that do not start
with the prefix. On a single core, the same request counts only the values with
the prefix.
h3. Steps to reproduce
A collection with 2 or more shards and a multi-valued string field
\{{multi_ss}}, with these docs:
{code}
{"id":"1", "multi_ss":["a","b"]}
{"id":"2", "multi_ss":["b"]}
{"id":"3", "multi_ss":["a"]}
{"id":"4", "multi_ss":["b","a"]}
{code}
{code}
json.facet=\{f1:{type:terms, field:multi_ss, prefix:a, numBuckets:true}}
{code}
Expected (and returned on a single core): \{{numBuckets: 1}}. Returned on the
distributed request: \{{numBuckets: 2}}.
The effect is much larger on fields with many values per doc. We index
hierarchies as path values, one per level ("0|root", "1|root|child", ...), and
facet on one level with \{{prefix:"0|"}}. On a 3-shard collection, numBuckets
for a level that has 61 values returns 350,131, about the number of nodes of
all levels. On a 2-shard collection with 10 roots, 100 nodes on level 1 and
1,000 on level 2, stock 9.11.0 returns 1,113 for both \{{prefix:"0|"}} and
\{{prefix:"1|"}}.
h3. Cause
On a shard, \{{FacetFieldProcessor.calculateNumBuckets}} applies the prefix to
the domain: it keeps the docs that have a value with the prefix. It then
collects an HLL over all values of the field in those docs (\{{HLLAgg}} /
\{{UniqueSlotAcc.getShardHLL}}). On a multi-valued field, those docs also have
values without the prefix, and these are counted too. A single-valued field is
not affected, since a doc that matches the prefix has no other value.
h3. Proposed fix
When the facet has a prefix, let the shard HLL include only the values that
start with it. \{{getShardHLL}} already looks up every collected value to hash
it, so this adds one byte comparison per value. The prefix query on the domain
stays, as it keeps the docs to collect small.
This keeps the semantics of SOLR-10552: distributed numBuckets is still an
estimate that ignores mincount.
A PR with the fix and a test in \{{TestJsonFacets.doStats}} (which runs
single-core and distributed, with every facet method) follows. We have run the
fix on 9.11.0 on a 3-shard collection with our full test data set: all tree
level counts match a single core.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
