Zoltán Borók-Nagy created IMPALA-15706:
------------------------------------------
Summary: Iceberg DELETE/UPDATE/MERGE on Avro data files fail with
an internal FILE__POSITION error
Key: IMPALA-15706
URL: https://issues.apache.org/jira/browse/IMPALA-15706
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 5.0.0
Reporter: Zoltán Borók-Nagy
DELETE, UPDATE and MERGE need the position of the rows they delete. Impala does
not support the {{FILE\_\_POSITION}} virtual column for Avro files. So when one
of these statements scans an Avro data file of an Iceberg table, the scanner
fails with an error about a virtual column that the user never referenced:
{noformat}
Virtual column FILE__POSITION is not supported for AVRO files.
{noformat}
The same happens when a query reads the row lineage column {{\_row\_id}} of a
V3 table, and when OPTIMIZE TABLE on a V3 table rewrites Avro data files,
because both use {{FILE\_\_POSITION}} internally.
Repro on master (create the table and the Avro data file in Hive, then run the
rest in Impala):
{code:sql}
-- Hive
create table t (i int) stored by iceberg
tblproperties('format-version'='2', 'write.format.default'='avro');
insert into t values (1), (2);
-- Impala
invalidate metadata t;
alter table t set tblproperties('write.format.default'='parquet');
delete from t where i = 1;
{code}
The error comes from HdfsScanner::ValidateSlotDescriptors() at execution time.
UPDATE fails the same way. MERGE currently hits IMPALA-15705 (NPE in planning)
first, and shows this error only with RUNTIME_FILTER_MODE=OFF.
*Expected*
Fail during planning with a message that names the statement and the file
format, e.g. "DELETE is not supported on Iceberg tables that have AVRO data
files", and do not mention {{FILE\_\_POSITION}} unless the query references it.
HdfsScanNode.checkUuidReadSupported() /
FeIcebergTable.validateUuidReadSupported() already do the same kind of check
for UUID columns, based on the formats of the scanned files.
*Suggested fix*
In the planner, when an Iceberg scan materializes {{FILE\_\_POSITION}}
implicitly (DELETE, UPDATE, MERGE, {{\_row\_id}}, OPTIMIZE) and the scanned
files include Avro, throw an AnalysisException with a statement-level message.
A longer-term alternative is to support {{FILE\_\_POSITION}} for Avro files by
counting the records of the preceding Avro blocks.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
