[
https://issues.apache.org/jira/browse/SOLR-18597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SOLR-18597:
----------------------------------
Labels: pull-request-available (was: )
> JSON facet numBuckets with a prefix counts all values of the matching docs in
> a distributed request
> ---------------------------------------------------------------------------------------------------
>
> Key: SOLR-18597
> URL: https://issues.apache.org/jira/browse/SOLR-18597
> Project: Solr
> Issue Type: Bug
> Components: Facet Module
> Affects Versions: 10.0
> Reporter: Samuel Verstraete
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In a distributed request, a terms facet with \{{prefix}} and
> \{{numBuckets:true}} on a multi-valued field counts values that do not start
> with the prefix. On a single core, the same request counts only the values
> with the prefix.
> h3. Steps to reproduce
> A collection with 2 or more shards and a multi-valued string field
> \{{multi_ss}}, with these docs:
> {code}
> {"id":"1", "multi_ss":["a","b"]}
> {"id":"2", "multi_ss":["b"]}
> {"id":"3", "multi_ss":["a"]}
> {"id":"4", "multi_ss":["b","a"]}
> {code}
> {code}
> json.facet=\{f1:{type:terms, field:multi_ss, prefix:a, numBuckets:true}}
> {code}
> Expected (and returned on a single core): \{{numBuckets: 1}}. Returned on the
> distributed request: \{{numBuckets: 2}}.
> The effect is much larger on fields with many values per doc. We index
> hierarchies as path values, one per level ("0|root", "1|root|child", ...),
> and facet on one level with \{{prefix:"0|"}}. On a 3-shard collection,
> numBuckets for a level that has 61 values returns 350,131, about the number
> of nodes of all levels. On a 2-shard collection with 10 roots, 100 nodes on
> level 1 and 1,000 on level 2, stock 9.11.0 returns 1,113 for both
> \{{prefix:"0|"}} and \{{prefix:"1|"}}.
> h3. Cause
> On a shard, \{{FacetFieldProcessor.calculateNumBuckets}} applies the prefix
> to the domain: it keeps the docs that have a value with the prefix. It then
> collects an HLL over all values of the field in those docs (\{{HLLAgg}} /
> \{{UniqueSlotAcc.getShardHLL}}). On a multi-valued field, those docs also
> have values without the prefix, and these are counted too. A single-valued
> field is not affected, since a doc that matches the prefix has no other value.
> h3. Proposed fix
> When the facet has a prefix, let the shard HLL include only the values that
> start with it. \{{getShardHLL}} already looks up every collected value to
> hash it, so this adds one byte comparison per value. The prefix query on the
> domain stays, as it keeps the docs to collect small.
> This keeps the semantics of SOLR-10552: distributed numBuckets is still an
> estimate that ignores mincount.
> A PR with the fix and a test in \{{TestJsonFacets.doStats}} (which runs
> single-core and distributed, with every facet method) follows. We have run
> the fix on 9.11.0 on a 3-shard collection with our full test data set: all
> tree level counts match a single core.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
