[
https://issues.apache.org/jira/browse/IMPALA-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118320#comment-18118320
]
ASF subversion and git services commented on IMPALA-15361:
----------------------------------------------------------
Commit fd26e466263e331359088099b946dd7f211f0d93 in impala's branch
refs/heads/master from Aleksandr Efimov
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=fd26e4662 ]
IMPALA-15361: Read Hive 4 timestamp stats
Hive 4 reconstructs persisted TIMESTAMP statistics in the dedicated
TimestampColumnStatsData union field based on the column type,
regardless of which engine wrote them. Impala also uses
LongColumnStatsData when it writes the statistics directly.
Read NDV and null counts from both representations and keep low and high
values unset. Handle Hive 4 field 8 in Hive3MetastoreShimBase so both
Apache and CDP Hive 3 shims use it without referencing the Hive 4 API.
Add unit coverage for both representations. The Hive 4 case is skipped
when the timestampStats API is unavailable.
Testing:
- Frontend build and ColumnStatsTest with Hive 4.0.1 (0 skipped)
- Frontend build and ColumnStatsTest with Apache Hive 3.1.3 (1 skipped)
- Frontend build and ColumnStatsTest with CDP Hive 3 (1 skipped)
Change-Id: I0b99acbe65665226938b2e4ac5191063d3925426
Assisted-by: gpt-5.6-sol (OpenAI Codex)
Reviewed-on: http://gerrit.cloudera.org:8080/24894
Reviewed-by: Csaba Ringhofer <[email protected]>
Tested-by: Csaba Ringhofer <[email protected]>
> TIMESTAMP column stats are written to a Hive 4 metastore and never read back
> ----------------------------------------------------------------------------
>
> Key: IMPALA-15361
> URL: https://issues.apache.org/jira/browse/IMPALA-15361
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Aleksandr Efimov
> Assignee: Aleksandr Efimov
> Priority: Major
>
> Impala 5.0.0-SNAPSHOT (build e8d42f753d) with an Apache Hive 4.0.0 metastore.
> After COMPUTE STATS, REFRESH rejects statistics for TIMESTAMP columns as
> incompatible and SHOW COLUMN STATS reports NDV as -1. In 78 minutes, catalogd
> logged 212 such warnings for 119 columns in 58 tables, all TIMESTAMP.
> The cause is confirmed. Hive 4 reconstructs statistics according to the
> column type and returns TIMESTAMP as timestampStats, field 8 of
> ColumnStatisticsData, regardless of which engine wrote them. Impala writes
> longStats, but ColumnStats.update() does not read the timestampStats
> representation returned by Hive 4.
> The failure was reproduced end to end with Hive 4.0.0. The same Hive behavior
> is present in 4.2.1, and the compatible reader path was checked with the Hive
> 4.0.1 and 4.2.1 APIs.
> The reader should accept both longStats and timestampStats while keeping
> compatibility with older Hive APIs. Low and high values remain unset.
> Filed under IMPALA-13647.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
