[
https://issues.apache.org/jira/browse/IMPALA-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Efimov updated IMPALA-15361:
--------------------------------------
Description:
Impala 5.0.0-SNAPSHOT (build e8d42f753d) with an Apache Hive 4.0.0 metastore.
After COMPUTE STATS, REFRESH rejects statistics for TIMESTAMP columns as
incompatible and SHOW COLUMN STATS reports NDV as -1. In 78 minutes, catalogd
logged 212 such warnings for 119 columns in 58 tables, all TIMESTAMP.
The cause is confirmed. Hive 4 reconstructs statistics according to the column
type and returns TIMESTAMP as timestampStats, field 8 of ColumnStatisticsData,
regardless of which engine wrote them. Impala writes longStats, but
ColumnStats.update() does not read the timestampStats representation returned
by Hive 4.
The failure was reproduced end to end with Hive 4.0.0. The same Hive behavior
is present in 4.2.1, and the compatible reader path was checked with the Hive
4.0.1 and 4.2.1 APIs.
The reader should accept both longStats and timestampStats while keeping
compatibility with older Hive APIs. Low and high values remain unset.
Filed under IMPALA-13647.
was:
Impala 5.0.0-SNAPSHOT (build e8d42f753d), Apache Hive 4.0.0 metastore.
{{COMPUTE STATS}} on a 10-column table left a row in {{TAB_COL_STATS}} for
every column, TIMESTAMP ones included, each with {{NUM_DISTINCTS}} set. On the
next {{REFRESH}}, catalogd logged {{Failed to load column stats ...
incompatible with column type TIMESTAMP}} for the four TIMESTAMP columns and
nothing for the other six, and {{SHOW COLUMN STATS}} has shown -1 for those
four ever since. Recomputing does not change it. In 78 minutes of catalogd log
there are 212 such warnings on 58 tables, every one TIMESTAMP.
That warning fires only when a {{ColumnStatisticsObj}} did arrive and
{{ColumnStats.update()}} returned false, so the metastore is answering with
something {{isSetLongStats()}} rejects -- while the write side sets
{{longStats}} at {{ColumnStats.java:896-897}}. The comment on the read branch
states the assumption Hive 4 appears to break:
{code}
case TIMESTAMP: // Hive and Impala use LongColumnStatsData for timestamps.
isCompatible = statsData.isSetLongStats(); // ColumnStats.java:610-611
{code}
What I have not read is the wire: which member of the union comes back.
{{timestampStats}} is the candidate, since {{TimestampColumnStatsData}} appears
nowhere in Impala's sources while DATE has its own arm -- but a missing read
arm and HMS 4 converting on write want different fixes, and I have not tried
HMS 3.x.
Meanwhile {{COMPUTE STATS}} reports success and planning falls back to defaults
on every TIMESTAMP predicate and join key. One trade worth naming before a fix
picks a side: writing {{timestampStats}} too would take stats away from older
readers on the same metastore.
Filed under IMPALA-13647.
> TIMESTAMP column stats are written to a Hive 4 metastore and never read back
> ----------------------------------------------------------------------------
>
> Key: IMPALA-15361
> URL: https://issues.apache.org/jira/browse/IMPALA-15361
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Aleksandr Efimov
> Priority: Major
>
> Impala 5.0.0-SNAPSHOT (build e8d42f753d) with an Apache Hive 4.0.0 metastore.
> After COMPUTE STATS, REFRESH rejects statistics for TIMESTAMP columns as
> incompatible and SHOW COLUMN STATS reports NDV as -1. In 78 minutes, catalogd
> logged 212 such warnings for 119 columns in 58 tables, all TIMESTAMP.
> The cause is confirmed. Hive 4 reconstructs statistics according to the
> column type and returns TIMESTAMP as timestampStats, field 8 of
> ColumnStatisticsData, regardless of which engine wrote them. Impala writes
> longStats, but ColumnStats.update() does not read the timestampStats
> representation returned by Hive 4.
> The failure was reproduced end to end with Hive 4.0.0. The same Hive behavior
> is present in 4.2.1, and the compatible reader path was checked with the Hive
> 4.0.1 and 4.2.1 APIs.
> The reader should accept both longStats and timestampStats while keeping
> compatibility with older Hive APIs. Low and high values remain unset.
> Filed under IMPALA-13647.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
