[
https://issues.apache.org/jira/browse/UNOMI-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Serge Huber updated UNOMI-1020:
-------------------------------
Description:
h3. In short
Above 10,000 matches, some reports show the wrong number of matches, usually
too low (for example goal reports split by category).
h3. Details
* On the optimized aggregate path, {{_filtered}} comes from the search hit
count, which stops at 10,000.
* Above that, Unomi uses the sum of the returned buckets instead, which is
wrong when the bucket limit drops values or ranges overlap.
h3. Fix and test
Count hits exactly on both aggregate paths. With more than 10,000 matches,
{{_filtered}} must equal {{queryCount}} on both engines.
was:
h3. In short
When more than 10,000 visitors or sessions match a query, some reports can show
the wrong number of matches, usually too low. Goal reports broken down by
category are one place where this can appear. Sites with fewer matches are not
affected.
h3. Summary
On the optimized aggregate path ({{aggregateWithOptimizedQuery}}, or REST with
{{optimizedQuery=true}}), {{_filtered}} comes from the search hit count. That
count stops at 10,000 unless exact counting is enabled. Above 10,000, Unomi
replaces {{_filtered}} with the sum of the buckets plus {{_missing}}, which is
only right when every matching item falls in exactly one returned bucket:
* A terms aggregate cut short by the bucket limit undercounts.
* Overlapping ranges overcount.
* Ranges that do not cover every item undercount.
Goal reports split by an aggregate take their start and target counts from
{{_filtered}}, so they are affected for goals with more than 10,000 matching
sessions. Date splits are exact in practice, because every session lands in one
date bucket.
The non-optimized path is exact since UNOMI-1014, because it enables exact
counting.
h3. Ways to explore
* Enable exact counting on the optimized path. Aggregations already visit every
matching document, so counting them exactly should cost little.
* Or keep the bucket sum only where it is provably exact (date histogram plus
{{_missing}}) and count exactly otherwise.
h3. How to test
On both engines, with more than 10,000 matching items, {{_filtered}} must equal
{{queryCount}} for a terms aggregate whose bucket limit is smaller than the
number of distinct values.
> Aggregate _filtered count is approximate above 10,000 matches on the
> optimized path
> -----------------------------------------------------------------------------------
>
> Key: UNOMI-1020
> URL: https://issues.apache.org/jira/browse/UNOMI-1020
> Project: Apache Unomi
> Issue Type: Bug
> Reporter: Serge Huber
> Priority: Major
> Fix For: unomi-4.0.0
>
>
> h3. In short
> Above 10,000 matches, some reports show the wrong number of matches, usually
> too low (for example goal reports split by category).
> h3. Details
> * On the optimized aggregate path, {{_filtered}} comes from the search hit
> count, which stops at 10,000.
> * Above that, Unomi uses the sum of the returned buckets instead, which is
> wrong when the bucket limit drops values or ranges overlap.
> h3. Fix and test
> Count hits exactly on both aggregate paths. With more than 10,000 matches,
> {{_filtered}} must equal {{queryCount}} on both engines.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
