[
https://issues.apache.org/jira/browse/IMPALA-10230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17213270#comment-17213270
]
Tim Armstrong commented on IMPALA-10230:
----------------------------------------
I can reproduce on master when manually modifying the incremental stats. I
tweaked the Metastore SQL a bit to work on my postgres database:
{noformat}
SELECT d."NAME",t."TBL_NAME",p.*,pp.*
FROM "PARTITIONS" p join
"TBLS" t on p."TBL_ID" = t."TBL_ID" join
"DBS" d on t."DB_ID" = d."DB_ID" join
"PARTITION_PARAMS" pp on p."PART_ID"=pp."PART_ID"
WHERE d."NAME"='default'
AND t."TBL_NAME"='test_column_stats';
update "PARTITION_PARAMS"
set
"PARAM_VALUE"='HBYCABsDjARpbnQxGAz/AK4AAAH/AP8ATwARFgAVABcAAAAAAAAQQBYCAARzdHIxGAj/AP8A/wD/ABEWARUAFwAAAAAAAAAAFgAABHN0cjIYDP8A/wD/AAAAAAH9ABEWABUQFwAAAAAAACBAFgIAAA=='
where "PARAM_KEY"='impala_intermediate_stats_chunk0'
and "PART_ID" in (
SELECT p."PART_ID"
FROM "PARTITIONS" p join
"TBLS" t on p."TBL_ID" = t."TBL_ID" join
"DBS" d on t."DB_ID" = d."DB_ID"
WHERE d."NAME"='default'
AND t."TBL_NAME"='test_column_stats');
{noformat}
{noformat}
[localhost.EXAMPLE.COM:21050] default> Query: compute incremental stats
test_column_stats partition(ds=20200107)
> ERROR: TableLoadingException: Failed to
load metadata for table: default.test_column_stats
> CAUSED BY: IllegalStateException:
ColumnStats{avgSize_=3.0, avgSerializedSize_=15.0, maxSize_=3, numDistinct_=1, n
umNulls_=-5}
{noformat}
I think you're right that we should no longer write the bad stats after
IMPALA-9699 is fixed, but it would be good if we could gracefully handle bad
stats.
> column stats num_nulls less than -1
> -----------------------------------
>
> Key: IMPALA-10230
> URL: https://issues.apache.org/jira/browse/IMPALA-10230
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 3.4.0
> Reporter: logan zheng
> Priority: Critical
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> when update impala 3.2.0(CDH6.3.2 ) to asf3.4.0 ,after when "increment stats
> default.test partition(xx=yyyy)":
> {noformat}
> ERROR: TableLoadingException: Failed to load metadata for table: default.test
> CAUSED BY: IllegalStateException: ColumnStats{avgSize_=13.0,
> avgSerializedSize_=25.0, maxSize_=19, numDistinct_=12, numNulls_=-2}{noformat}
> The table default.test already exists in impala 3.2.0, and has been running
> for a long time, and has also been added stats.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
