Zoltán Borók-Nagy created IMPALA-15462:
------------------------------------------
Summary: Iceberg: one column of a type Impala cannot read
(GEOMETRY, GEOGRAPHY, TIME, FIXED, TIMESTAMP_NS, UNKNOWN) makes the whole table
unloadable
Key: IMPALA-15462
URL: https://issues.apache.org/jira/browse/IMPALA-15462
Project: IMPALA
Issue Type: Improvement
Components: Catalog, Frontend
Reporter: Zoltán Borók-Nagy
h3. Problem
Impala converts the whole Iceberg schema when it loads a table. If any column
has an Iceberg type that {{IcebergSchemaConverter.toImpalaType()}} does not
map, the load fails. Nested struct fields count too. After that, every
statement on the table fails, even one that never reads the column: {{SELECT
count(*)}}, a SELECT of other columns, DESCRIBE, SHOW CREATE TABLE, SHOW
TABLE/COLUMN STATS, SHOW FILES, DESCRIBE HISTORY, metadata tables, COMPUTE
STATS and CREATE EXTERNAL TABLE. Only SHOW TABLES works. For HMS tables,
INVALIDATE METADATA and DROP TABLE also succeed. REFRESH reports success only
once, on a table that was loaded before the change (see below). After a failed
load, REFRESH returns the cached load error without reloading, even after
another engine has removed the column. Only INVALIDATE METADATA clears it.
ALTER TABLE ... DROP COLUMN fails too, so the column cannot be removed from
Impala. The error names the type but not the column.
On master the unmapped types are GEOMETRY, GEOGRAPHY, TIMESTAMP_NANO
(timestamp_ns and timestamptz_ns, both run), UNKNOWN (all v3), and TIME and
FIXED (valid since v1). I verified all six.
Why this matters:
* A schema change made by another engine breaks existing Impala queries. After
DuckDB added one GEOMETRY column, every Impala query on the table failed (for
HMS tables, after the next REFRESH).
* IMPALA-15162 will make GEOMETRY load, but not the other five types. Its patch
(Gerrit 24536 PS19, code reading only) also throws in {{toImpalaType()}} for a
geometry column whose CRS is not OGC:CRS84, so such tables will still fail to
load.
h3. Steps to reproduce
DuckDB 1.5.5 (iceberg + spatial extensions), attached to an Iceberg REST
catalog:
{code:sql}
CREATE TABLE ath.zz.geom (id BIGINT, name VARCHAR, g GEOMETRY) WITH
('format-version'='3');
INSERT INTO ath.zz.geom VALUES (1, 'a', ST_Point(1, 2)), (2, 'b', ST_Point(3,
4)), (3, 'c', NULL);
{code}
Impala, with the REST catalog configured through {{--catalog_config_dir}}:
{code:sql}
SELECT count(*) FROM zz.geom;
SELECT id FROM zz.geom;
DESCRIBE zz.geom;
{code}
HMS variant: publish the HadoopTables layout ({{metadata/v1.metadata.json}} +
{{version-hint.text}}), then run {{CREATE EXTERNAL TABLE ... STORED AS ICEBERG
LOCATION '...' TBLPROPERTIES('iceberg.catalog'='hadoop.tables')}}.
Schema-change variant: register and query a v3 table {{(id, name)}} (on a v2
table, adding a GEOMETRY column fails at commit; there, {{ADD COLUMN t TIME}}
breaks the table the same way). In DuckDB, run {{ALTER TABLE ... ADD COLUMN g
GEOMETRY}}. Republish the layout for the new metadata file, run REFRESH in
Impala, then query again.
Other types: empty tables created through the REST API with {{time}} or
{{fixed\[16\]}} (v2), or with {{timestamp_ns}}, {{timestamptz_ns}},
{{geography}}, {{unknown}} or {{struct<x: long, g: geometry>}} (v3).
h3. Expected
The table loads. Statements that do not reference the column work: count(*),
other columns, predicates on other columns. DESCRIBE lists the column and marks
its type as unsupported. A statement that references the column, including
{{SELECT *}}, fails during analysis with a message that names the column and
its Iceberg type. Writes to the table are rejected.
Impala already does this for HBase ({{FeHBaseTable.java:181-186}}: "We can load
the metadata but won't be able to select from it."). HMS tables with a type
Impala parses but does not support, such as DATETIME, also load
({{FeCatalogUtils.java:78-83}}, {{ScalarType.isSupported()}} :317-319;
documented in {{AnalyzerTest.java:276-279}}). Verified on master 3d4d47dc30
with an HMS table {{(id BIGINT, d DATETIME)}}: count(*), SELECT id and DESCRIBE
work. SELECT d fails in the analyzer ({{SlotRef.java:172-181}}) with
{{Unsupported type 'DATETIME' in 'd'.}} (5d4b095998fac9e2:d226f1f800000000).
INSERT fails ({{InsertStmt.java:502-508}}) with {{... because the column 'd'
has an unsupported type 'DATETIME'.}} (b6480a608e8312f4:54d1e2b800000000).
COMPUTE STATS skips the column ({{ComputeStatsStmt.java:967-971}}).
h3. Actual (master 3d4d47dc30, RELEASE, docker; REST catalog in hybrid mode)
REST catalog, {{SELECT count(*) FROM zz_jira_geometry.geom}} (query
df4539d7adb14864:270377a100000000), abridged:
{noformat}
AnalysisException: Failed to load metadata for table: 'zz_jira_geometry.geom'
CAUSED BY: TableLoadingException: Could not load table zz_jira_geometry.geom
from catalog
CAUSED BY: TException: Every MetaProvider failed with the following exceptions:
Iceberg REST (http://rest:8181): ...
Caused by: java.lang.IllegalStateException: Error loading Iceberg table
zz_jira_geometry.geom
Caused by: org.apache.impala.common.ImpalaRuntimeException: Iceberg type
'GEOMETRY' is not supported in Impala
at
org.apache.impala.util.IcebergSchemaConverter.toImpalaType(IcebergSchemaConverter.java:130)
at
...IcebergSchemaConverter.convertToHiveSchema(IcebergSchemaConverter.java:142)
at
...IcebergMetaProvider.createStorageDescriptor(IcebergMetaProvider.java:227)
Catalogd (URI TODO): ... InconsistentMetadataFetchException: Fetching TABLE
failed: DB_NOT_FOUND ...
{noformat}
* Same error: {{SELECT id}} a5499332860a7a52:6230cfd600000000, DESCRIBE
d74978d157aefac9:26251e0400000000, SHOW CREATE TABLE
e6466d4188b5f1c5:fd825dd400000000, {{geom.snapshots}}
164976bebaaca411:9e0a672f00000000; on a second copy SHOW TABLE STATS
5349cee4ce80dbb2, SHOW COLUMN STATS 3a400b6729412d5a, SHOW FILES
ae446eac6ea07b53, DESCRIBE HISTORY ce44b9534dc0f87f.
* {{geo.trip_v3}}: count(*) e640bebbf68ed2d8:2a94a4fb00000000, the {{p1-xy}}
query aa41c487e3416d7f:c9426ac200000000; {{geo.trip_v3_hilbert}}
0f40363e5763f70c:899514f200000000; {{geo.building_v3}}
af4cbfe9648386b4:6e28afb700000000. The {{p1-xy}} query on {{geo.trip_bridge}}
(same rows, WKB as {{binary}}) returns cnt=14, sum_key=459188959
(cd4392972c099afa:c241684800000000). Trino 483 and StarRocks 4.1.4 return the
same answer on {{geo.trip_v3}}.
* Other types: TIME 6149bceac19da5da:89a4aef500000000, FIXED
b946ceb98bef8118:8c7796df00000000, TIMESTAMP_NANO
ac4678f901cd284d:4ffba16300000000 (timestamp_ns) and
6945d06cc6a0529f:1bd34f9500000000 (timestamptz_ns), GEOGRAPHY
4e486e1d79f40c99:1c7ef78200000000, UNKNOWN 7345ff425846346d:b749633000000000.
The struct case (7c4eee4eaf32248e:823aada600000000) says only {{Iceberg type
'GEOMETRY'}}, with no column or field name.
* HMS, CREATE EXTERNAL TABLE (49482d742e20196c:e9ff080900000000):
{{ImpalaRuntimeException: Error making 'createTable' RPC to Hive Metastore:
CAUSED BY: TableLoadingException: Iceberg type 'GEOMETRY' is not supported in
Impala}}. This is misleading because no HMS RPC failed. The error comes from
{{IcebergCatalogOpExecutor.populateExternalTableCols()}}
({{IcebergCatalogOpExecutor.java:130-132}}) and is wrapped at
{{CatalogOpExecutor.java:4407-4408}}.
* HMS, schema change: after the ADD COLUMN, count(*) still returned 3 from the
cached metadata (d84da571554ad277:bbd56b7800000000); through the REST catalog
the same table already failed (aa4cbd9eeee43854:896e11c200000000). REFRESH then
reported success (c0466540b7caa985:bc25661400000000), because it invalidates
the table on an IcebergTableLoadingException
({{CatalogOpExecutor.java:7547-7553}}). The next count(*) failed
(974adf54cb4e39d8:fc02f34400000000; catalogd: {{IcebergTableLoadingException
... CAUSED BY: TableLoadingException: Iceberg type 'GEOMETRY' is not supported
in Impala}}). In a rerun, a second REFRESH failed with the same load error
(6e4756d5c2cdecbe:220fa5af00000000). After DuckDB inserted a row and dropped
the column (layout republished), REFRESH still returned the cached error in 1
ms without reloading ({{CatalogOpExecutor.java:8418-8425}};
794839ac448986ad:8569e91c00000000). INVALIDATE METADATA fixed it
(2b4bb8afe84c3111:e5458c9c00000000; count(*) 60423e89b7fb4a96:a120357800000000
returned 4). On a second copy, ALTER TABLE ... DROP COLUMN g (6f4ebcf4ca1051fa)
and COMPUTE STATS (5841545f1791c4ad) failed with the load error; INVALIDATE
METADATA (524746c0a8aedee8) and DROP TABLE (c74d1db0872eca91) succeeded.
h3. Analysis (master 3d4d47dc30)
* {{IcebergSchemaConverter.toImpalaType()}} throws in its {{default:}} branch
({{IcebergSchemaConverter.java:129-131}}). {{convertToHiveSchema()}} (:142) and
{{convertToImpalaSchema()}} (:159) call it for every column at load time: in
catalogd from {{IcebergTable.loadSchema()}} ({{IcebergTable.java:783-790}}, via
{{load()}} :533 -> {{loadTableMetadata()}} :558 -> {{loadSchemaFromIceberg()}}
:770), and for REST tables from {{IcebergMetaProvider.loadTable()}} (:199,
wrapped at :212-213) and {{LocalIcebergTable.java:109}}. Time travel
({{IcebergTimeTravelTable.java:124}}), metadata tables
({{IcebergMetadataTable.java:71}}), partition fields
({{FeIcebergTable.java:1237}}), equality-delete fields
({{IcebergEqualityDeleteTable.java:50}}), fields with default values
({{IcebergScanPlanner.java:1165}}), CTAS ({{IcebergCtasTarget.java:122}}) and
CREATE EXTERNAL TABLE ({{IcebergCatalogOpExecutor.java:130}}) call it too.
* Impala 4.5.2 has the same {{default:}} throw
({{IcebergSchemaConverter.java:116-117}}), and there UUID is unmapped too.
Verified on the apache/impala:4.5.2 docker images (HMS, hadoop.tables). CREATE
EXTERNAL TABLE of a v2 table with a TIME, FIXED or UUID column fails with
{{Error making 'createTable' RPC to Hive Metastore: CAUSED BY:
TableLoadingException: Iceberg type 'TIME' is not supported in Impala}}
(d64def7424a0718e:041fa42d00000000; FIXED 4f4c0d96def483cf:062d42a700000000,
UUID 64425d6bcd397a29:faa16a8000000000). A registered v2 table that got a TIME
column from DuckDB fails to load after REFRESH, with a 6-line error
(534319ac164af4e4:160748ca00000000), and its DROP COLUMN fails too
(754d967e597bd146:480bac8200000000). 4.5.2 cannot read v3 tables at all: its
Iceberg 1.3.1 rejects them with {{IllegalArgumentException: Cannot read
unsupported version 3}} (bc4a4a38b37f8093:d17eaa7e00000000).
* Suggested fix: when converting an existing Iceberg schema (table load, and
CREATE EXTERNAL TABLE, which copies it at
{{IcebergCatalogOpExecutor.java:130}}), but not user-given types in CREATE
TABLE or ALTER TABLE, map an unmapped Iceberg type to a placeholder instead of
throwing: {{Type.INVALID}}, as for HBase, or an unsupported ScalarType that
keeps the Iceberg type string for DESCRIBE and SHOW CREATE TABLE. Give it an
HMS type string such as {{binary}} (PS19 of IMPALA-15162 stores GEOMETRY as
binary in HMS). Keep the column in both {{convertToHiveSchema()}} and
{{convertToImpalaSchema()}}: {{LocalIcebergTable.validateColumns()}}
({{LocalIcebergTable.java:183-189}}) needs the same column count and names
(code reading). The reverse conversion (used by CREATE, ALTER and CTAS) must
reject the placeholder. Make the SlotRef error name the Iceberg type. Reject
DML that writes data files. Until then, the load error should name the column
(and struct path) and not claim an HMS RPC failure.
* Fix scope: use the placeholder also for time-travel schemas and metadata
tables. Time travel to a snapshot written with the column fails even after the
column is dropped (see Workaround). In Iceberg 1.11 the {{readable_metrics}} of
{{files}} and {{entries}} has lower_bound/upper_bound fields of the column's
own type, so {{IcebergMetadataTable.java:71}} would still throw (code reading).
Partition fields of every spec, old ones included, are converted at load
({{FeIcebergTable.java:1222}}, :1237), so an identity partition on a TIME,
FIXED or timestamp_ns column needs the placeholder too. Equality-delete fields
with such sources can stay unsupported at scan time. For HiveCatalog tables,
Iceberg 1.11's {{HiveSchemaUtil}} throws on every commit for TIMESTAMP_NANO,
GEOMETRY, GEOGRAPHY and UNKNOWN (code reading; PS19 patches GEOMETRY and
GEOGRAPHY), so writes and ALTER TABLE on such tables can still fail.
* Tests: Impala test data has no Iceberg table with these types (git grep; the
IMPALA-14738 commit deferred the V3-type tables). They can be made with REST
createTable, as in the repro.
h3. Workaround
None in Impala. ALTER TABLE ... DROP COLUMN fails with the same load error
(6f4ebcf4ca1051fa:ae17ca1100000000). Dropping the column with another engine
makes the table load again. After DuckDB {{ALTER TABLE ... DROP COLUMN g}}, the
REST table loaded at once (97452099defe564c:78798b7c00000000). The HMS table
needed INVALIDATE METADATA; REFRESH kept returning the cached load error
(794839ac448986ad:8569e91c00000000). Time travel to a snapshot written with the
column still fails with {{AnalysisException: Could not create iceberg schema.
CAUSED BY: ImpalaRuntimeException: Iceberg type 'GEOMETRY' is not supported in
Impala}} (e34ed00cf978fb01:71d3d1db00000000). Otherwise, keep a copy of the
table without the unsupported column. The benchmark keeps a v2 "bridge" copy
with WKB in {{binary}} plus x/y DOUBLE columns. Impala reads that copy, and all
36 bridge runs executed (34 ok, 2 timed out on {{sb8}}).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
