Serge Huber created UNOMI-1020:
----------------------------------
Summary: Aggregate _filtered count is approximate above 10,000
matches on the optimized path
Key: UNOMI-1020
URL: https://issues.apache.org/jira/browse/UNOMI-1020
Project: Apache Unomi
Issue Type: Bug
Reporter: Serge Huber
Fix For: unomi-4.0.0
h3. Summary
On the optimized aggregate path ({{aggregateWithOptimizedQuery}}, or REST with
{{optimizedQuery=true}}), {{_filtered}} comes from the search hit count. That
count stops at 10,000 unless exact counting is enabled. Above 10,000, Unomi
replaces {{_filtered}} with the sum of the buckets plus {{_missing}}, which is
only right when every matching item falls in exactly one returned bucket:
* A terms aggregate cut short by the bucket limit undercounts.
* Overlapping ranges overcount.
* Ranges that do not cover every item undercount.
Goal reports split by an aggregate take their start and target counts from
{{_filtered}}, so they are affected for goals with more than 10,000 matching
sessions. Date splits are exact in practice, because every session lands in one
date bucket.
The non-optimized path is exact since UNOMI-1014, because it enables exact
counting.
h3. Ways to explore
* Enable exact counting on the optimized path. Aggregations already visit every
matching document, so counting them exactly should cost little.
* Or keep the bucket sum only where it is provably exact (date histogram plus
{{_missing}}) and count exactly otherwise.
h3. How to test
On both engines, with more than 10,000 matching items, {{_filtered}} must equal
{{queryCount}} for a terms aggregate whose bucket limit is smaller than the
number of distinct values.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
