Zoltán Borók-Nagy created IMPALA-15446:
------------------------------------------
Summary: Iceberg: columns of Avro data files are resolved by name
instead of field id (wrong results after RENAME or DROP + ADD COLUMN)
Key: IMPALA-15446
URL: https://issues.apache.org/jira/browse/IMPALA-15446
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Zoltán Borók-Nagy
h3. Problem
The Iceberg spec says "Columns in Iceberg data files are selected by field id".
For Avro data files Impala matches columns by *name*, so after ordinary Iceberg
schema evolution it silently returns wrong values:
* after {{RENAME COLUMN a TO b}}, {{b}} is NULL for every row of the old files;
* after {{DROP COLUMN a}} followed by {{ADD COLUMN a}}, the new column {{a}}
returns the *dropped* column's old values instead of NULL;
* on master, in a V3 table the re-added column's {{initial-default}} is
ignored: after {{DROP COLUMN a; ADD COLUMNS (a BIGINT DEFAULT 7)}} Impala
returns the old values 10, 20 instead of 7 (a new column {{c BIGINT DEFAULT 5}}
correctly returns 5);
* renames that swap two column names swap their values;
* mixed-case column names (e.g. {{MyCol}} written by Spark) read as NULL for
every row even without any schema change (IMPALA-8335, see Notes).
There is no error or warning. (If the re-added column has an incompatible type,
e.g. INT -> STRING, the file is skipped with an "Unresolvable types" warning
instead; see IMPALA-xxxxx.) The Impala docs say "Impala currently does not
support schema evolution for tables with AVRO file format", but nothing
enforces it, neither at query time nor in Impala's own ALTER TABLE: {{CHANGE
COLUMN a b INT}}, or {{DROP COLUMN a}} + {{ADD COLUMNS (a INT)}}, succeeds on
an Iceberg table with Avro files and gives the same wrong results (verified on
master). A column added under a new name is resolved correctly, and master
supports and tests that for Avro (IMPALA-14589 initial-default values, test
table {{iceberg_v3_default_value_avro}}), so what breaks is specifically
renames and reuse of a column name with a new field id. Parquet data files are
resolved by field id and are correct in the same scenarios.
Impala OPTIMIZE writes the wrongly resolved values into new Parquet files under
the current field ids (INSERT ... SELECT and CTAS from such a table copy them
as well), so the damage becomes permanent and other engines see it too.
Verified on master: after OPTIMIZE, Spark reads a = 10, 20 for the re-added
column and b = NULL for the renamed one. (UPDATE/DELETE/MERGE do not run on
Avro files.)
h3. Steps to reproduce
Spark 4.1.3 + Iceberg 1.11.0 (Spark catalog of type {{hadoop}}):
{code:sql}
CREATE TABLE ren (id BIGINT, a INT) USING iceberg TBLPROPERTIES
('format-version'='2', 'write.format.default'='avro');
INSERT INTO ren VALUES (1, 10), (2, 20);
ALTER TABLE ren RENAME COLUMN a TO b;
CREATE TABLE readd (id BIGINT, a INT) USING iceberg TBLPROPERTIES
('format-version'='2', 'write.format.default'='avro');
INSERT INTO readd VALUES (1, 10), (2, 20);
ALTER TABLE readd DROP COLUMN a;
ALTER TABLE readd ADD COLUMN a INT;
{code}
Impala:
{code:sql}
CREATE EXTERNAL TABLE ren STORED AS ICEBERG LOCATION '<warehouse>/<ns>/ren'
TBLPROPERTIES ('iceberg.catalog'='hadoop.tables');
CREATE EXTERNAL TABLE readd STORED AS ICEBERG LOCATION '<warehouse>/<ns>/readd'
TBLPROPERTIES ('iceberg.catalog'='hadoop.tables');
SELECT id, b FROM ren ORDER BY id;
SELECT id, a FROM readd ORDER BY id;
{code}
h3. Expected result (Spark returns this)
{{ren}}: (1, 10), (2, 20). {{readd}}: (1, NULL), (2, NULL).
h3. Actual result
Impala 4.5.2 (build f6a5cf48, verified 2026-09-28): {{ren}}: *(1, NULL), (2,
NULL)* (query 6d4e9fd39b5a0c9e:42a5f65100000000). {{readd}}: *(1, 10), (2, 20)*
(query 47403e86810a1f2a:cf5784a300000000). No warnings in the query log. The
same statements with {{'write.format.default'='parquet'}} return the expected
rows.
Master 3d4d47dc30 (5.0.0-SNAPSHOT): identical rows, {{ren}} (query
394957dfe25e1491:3ad5468200000000) and {{readd}} (query
4441727bf63e5643:2d69244c00000000). Query Status OK, profile Errors empty,
BytesSkipped 0, default query options. It also reproduces with format-version 1
and when the table is registered before the rename and REFRESHed. Files written
after the schema change are read correctly. The Parquet and ORC copies return
the expected rows.
h3. Analysis
{{HdfsAvroScanner::ResolveSchemas()}} looks up each table column in the file's
writer schema by name:
{{avro_schema_record_field_get_index(file_record->schema, field_name)}}
({{be/src/exec/avro/hdfs-avro-scanner.cc:248-251}}, same on 4.5.2 and master).
On master, IMPALA-14589's initial-default path ({{NeedDataInFile()}},
{{:253-258}}) is taken only when this lookup finds no field of that name.
Iceberg writes the field id of every column into the Avro schema
({{"field-id"}} on each field), so the scanner could match by field id for
Iceberg tables, as the Parquet and ORC scanners do.
The table-side reader schema has no field ids. It is built from the HMS column
names by {{AvroSchemaConverter}} ({{IcebergTable.setAvroSchema()}},
{{IcebergTable.java:801}} -> {{HdfsTable.setAvroSchemaInternal()}},
{{HdfsTable.java:1895}}, master). The BE already has the Iceberg field ids in
the table descriptor ({{col_descs()[i].field_id()}}), which the Parquet
({{parquet-metadata-utils.cc:809-816}}) and ORC
({{orc-metadata-utils.cc:275-276}}) resolvers use.
Fix direction: for Iceberg tables, resolve Avro fields by the {{field-id}}
property of the writer schema (fall back to the name mapping for files without
ids). The bundled avro-c 1.7.4 has no API for custom field attributes, so the
ids must be taken from the raw {{avro.schema}} JSON in the header
({{hdfs-avro-scanner.cc:163-166}}) or from the {{iceberg.schema}} header entry
that Iceberg also writes. The codegen fast path is chosen by
{{avro_schema_equal(table, file)}} ({{hdfs-avro-scanner.cc:180-181}}), which
compares names and types only, so it has to become id-aware too; otherwise the
drop/re-add case can still take it. Until then, fail the scan (at planning
time, from the Iceberg schema history) when a column of an Avro-containing
table was renamed, or its name was reused with a different field id, instead of
returning wrong values. Plain ADD COLUMN is resolved correctly and must stay
allowed (IMPALA-14589). Impala's own {{CHANGE COLUMN}} rename and DROP + re-ADD
of the same name could also be rejected on such tables.
h3. Workaround
Rewrite the Avro data files as Parquet or ORC with an engine that resolves by
field id, e.g. Spark: {{ALTER TABLE ... SET TBLPROPERTIES
('write.format.default'='parquet')}}, then {{CALL
<cat>.system.rewrite_data_files(table => '...', options =>
map('rewrite-all','true'))}}. After a REFRESH, Impala returns the correct rows.
Do NOT use Impala OPTIMIZE, INSERT ... SELECT or CTAS on such tables: they
persist the wrong values.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
