[
https://issues.apache.org/jira/browse/UNOMI-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Serge Huber updated UNOMI-1020:
-------------------------------
Description:
h3. In short
When more than 10,000 visitors or sessions match a query, some reports can show
the wrong number of matches, usually too low. Goal reports broken down by
category are one place where this can appear. Sites with fewer matches are not
affected.
h3. Summary
On the optimized aggregate path ({{aggregateWithOptimizedQuery}}, or REST with
{{optimizedQuery=true}}), {{_filtered}} comes from the search hit count. That
count stops at 10,000 unless exact counting is enabled. Above 10,000, Unomi
replaces {{_filtered}} with the sum of the buckets plus {{_missing}}, which is
only right when every matching item falls in exactly one returned bucket:
* A terms aggregate cut short by the bucket limit undercounts.
* Overlapping ranges overcount.
* Ranges that do not cover every item undercount.
Goal reports split by an aggregate take their start and target counts from
{{_filtered}}, so they are affected for goals with more than 10,000 matching
sessions. Date splits are exact in practice, because every session lands in one
date bucket.
The non-optimized path is exact since UNOMI-1014, because it enables exact
counting.
h3. Ways to explore
* Enable exact counting on the optimized path. Aggregations already visit every
matching document, so counting them exactly should cost little.
* Or keep the bucket sum only where it is provably exact (date histogram plus
{{_missing}}) and count exactly otherwise.
h3. How to test
On both engines, with more than 10,000 matching items, {{_filtered}} must equal
{{queryCount}} for a terms aggregate whose bucket limit is smaller than the
number of distinct values.
was:
h3. Summary
On the optimized aggregate path ({{aggregateWithOptimizedQuery}}, or REST with
{{optimizedQuery=true}}), {{_filtered}} comes from the search hit count. That
count stops at 10,000 unless exact counting is enabled. Above 10,000, Unomi
replaces {{_filtered}} with the sum of the buckets plus {{_missing}}, which is
only right when every matching item falls in exactly one returned bucket:
* A terms aggregate cut short by the bucket limit undercounts.
* Overlapping ranges overcount.
* Ranges that do not cover every item undercount.
Goal reports split by an aggregate take their start and target counts from
{{_filtered}}, so they are affected for goals with more than 10,000 matching
sessions. Date splits are exact in practice, because every session lands in one
date bucket.
The non-optimized path is exact since UNOMI-1014, because it enables exact
counting.
h3. Ways to explore
* Enable exact counting on the optimized path. Aggregations already visit every
matching document, so counting them exactly should cost little.
* Or keep the bucket sum only where it is provably exact (date histogram plus
{{_missing}}) and count exactly otherwise.
h3. How to test
On both engines, with more than 10,000 matching items, {{_filtered}} must equal
{{queryCount}} for a terms aggregate whose bucket limit is smaller than the
number of distinct values.
> Aggregate _filtered count is approximate above 10,000 matches on the
> optimized path
> -----------------------------------------------------------------------------------
>
> Key: UNOMI-1020
> URL: https://issues.apache.org/jira/browse/UNOMI-1020
> Project: Apache Unomi
> Issue Type: Bug
> Reporter: Serge Huber
> Priority: Major
> Fix For: unomi-4.0.0
>
>
> h3. In short
> When more than 10,000 visitors or sessions match a query, some reports can
> show the wrong number of matches, usually too low. Goal reports broken down
> by category are one place where this can appear. Sites with fewer matches are
> not affected.
> h3. Summary
> On the optimized aggregate path ({{aggregateWithOptimizedQuery}}, or REST
> with {{optimizedQuery=true}}), {{_filtered}} comes from the search hit count.
> That count stops at 10,000 unless exact counting is enabled. Above 10,000,
> Unomi replaces {{_filtered}} with the sum of the buckets plus {{_missing}},
> which is only right when every matching item falls in exactly one returned
> bucket:
> * A terms aggregate cut short by the bucket limit undercounts.
> * Overlapping ranges overcount.
> * Ranges that do not cover every item undercount.
> Goal reports split by an aggregate take their start and target counts from
> {{_filtered}}, so they are affected for goals with more than 10,000 matching
> sessions. Date splits are exact in practice, because every session lands in
> one date bucket.
> The non-optimized path is exact since UNOMI-1014, because it enables exact
> counting.
> h3. Ways to explore
> * Enable exact counting on the optimized path. Aggregations already visit
> every matching document, so counting them exactly should cost little.
> * Or keep the bucket sum only where it is provably exact (date histogram plus
> {{_missing}}) and count exactly otherwise.
> h3. How to test
> On both engines, with more than 10,000 matching items, {{_filtered}} must
> equal {{queryCount}} for a terms aggregate whose bucket limit is smaller than
> the number of distinct values.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
