[
https://issues.apache.org/jira/browse/IMPALA-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Mate resolved IMPALA-9699.
--------------------------------
Resolution: Resolved
> Skip '-1' values when aggregating num_null incremental statistics
> -----------------------------------------------------------------
>
> Key: IMPALA-9699
> URL: https://issues.apache.org/jira/browse/IMPALA-9699
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Affects Versions: Impala 3.3.0
> Reporter: Tamas Mate
> Assignee: Tamas Mate
> Priority: Major
> Labels: backwards-compatibility
>
> IMPALA-7659 added the population of NULL counts while computing stats, later
> IMPALA-8566 fixed an accuracy issue caused by the initialization of
> statistics. The initial value was changed from '-1' to '0'. The fix also
> contained a slight change on how the values are being summarized. Earlier the
> negative values were excluded from the summary:
> {code:java}
> if (num_new_nulls >= 0) num_nulls += num_new_nulls;
> {code}
> while in the new implementation, as these values should not be negative, the
> condition was removed:
> {code:java}
> num_nulls += num_new_nulls;
> {code}
> This change does not cause any problem for stats created after this fix,
> however it can make table metadata unavailable between earlier and newer
> releases. The metadata can be invalid if a compute incremental stats is
> issued on a partition because the '-1' values can decrease the column level
> num_nulls under '-1'. Later a smaller than '-1' num_null will fail on a
> precondition check when CatalogD is trying to fetch the table metadata.
> The condition should not cause any problem and due to backward compatibility
> reasons we should put it back.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
