[
https://issues.apache.org/jira/browse/IMPALA-8566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099789#comment-17099789
]
ASF subversion and git services commented on IMPALA-8566:
---------------------------------------------------------
Commit 56c20ac4639cab3d6e5e0ae77ce395934a9294f6 in impala's branch
refs/heads/master from Tamas Mate
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=56c20ac ]
IMPALA-9699: Skip '-1' values when aggregating num_null statistics
This change partially reverts IMPALA-8566 to make Impala backward
compatible with the old incremental partition stats. IMPALA-7659 added
collecting the number of null value statistics and IMPALA-8566 changed
the initial value of the incremental partition statistics from '-1' to
'0', because with '-1' the estimates were off by 1.
The old statistics in a new release can make the table metadata
inaccessible when the column stats are recomputed from the incremental
partition stats, which can be triggered by a partition level
'COMPUTE INCREMENTAL STATS'. In this case the old '-1' values can be
aggregated to a '<-1' 'num_nulls' value that later can fail a
Precondition check during table load.
The new behavior ensures that if any incremental partition stat has a
value of '-1' for 'num_nulls', the aggregated stats will be '-1',
regardless of whether or not other partitions have valid values for
'num_nulls'. This will prevent the planner from utilizing incomplete
statistics and the users will be notified about the missing statistics
with the general warning in the profile:
'The following tables are missing relevant table and/or column
statistics.'
Testing:
- Added unit test to verify the accepted values and aggregation result
Change-Id: I3fdf48a6c88378145078e068e12ade48c460f956
Reviewed-on: http://gerrit.cloudera.org:8080/15835
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> COMPUTE INCREMENTAL STATS sets num_nulls off-by-one
> ---------------------------------------------------
>
> Key: IMPALA-8566
> URL: https://issues.apache.org/jira/browse/IMPALA-8566
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.2.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Major
> Fix For: Impala 3.3.0
>
>
> IMPALA-7659 added the population of NULL counts while computing stats, but
> this functionality isn't working properly for incremental stats. The query is
> produced correctly, but the null count set in the table is one lower than it
> should be. In the case that the table has no nulls, this ends up setting a
> '-1' count, which is interpreted as 'unknown'. In the case that there are
> nulls, we'll just be a little inaccurate.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
