Zoltán Borók-Nagy created IMPALA-15637:
------------------------------------------
Summary: Pushed down Iceberg predicates are bound to the wrong
columns
Key: IMPALA-15637
URL: https://issues.apache.org/jira/browse/IMPALA-15637
Project: IMPALA
Issue Type: Bug
Components: Frontend
Reporter: Zoltán Borók-Nagy
Assignee: Zoltán Borók-Nagy
IcebergPredicateConverter refers to columns by name. Iceberg binds the names to
the current schema of the table, or to the schema of the snapshot for time
travel to an older snapshot. The name doesn't always refer to the same field as
the Impala column:
* The hidden columns of V3 tables, e.g. {{_file_row_id}} and
{{_file_last_updated_sequence_number}}, are not in the Iceberg schema, so
planning fails.
* Schema changes don't create snapshots. So time travel to the current
snapshot, e.g. {{FOR SYSTEM_TIME AS OF now()}}, reads it with the schema it was
created with, but Iceberg binds the predicates to the current schema. If two
columns swapped names since then, the predicate is bound to the other column,
and Iceberg skips files with matching rows. If a column was renamed or dropped,
planning fails.
{code:sql}
create table v3 (i int, p int) partitioned by spec (p) stored as iceberg
tblproperties ('format-version'='3');
insert into v3 values (1, 1);
select count(*) from v3 where p = 1 and _file_row_id is null;
-- ValidationException: Cannot find field '_file_row_id' in struct: ...
create table sw (a int, b int, p int) partitioned by spec (p) stored as iceberg;
insert into sw values (1, 10, 1);
alter table sw change column a tmp int;
alter table sw change column b a int;
alter table sw change column tmp b int;
select count(*) from sw for system_time as of now() where p = 1 and a = 1;
-- returns 0, expected 1
create table rn (a int, b int, p int) partitioned by spec (p) stored as iceberg;
insert into rn values (1, 10, 1);
alter table rn change column a c int;
select count(*) from rn for system_time as of now() where p = 1 and a = 1;
-- ValidationException: Cannot find field 'a' in struct: ...
create table dr (i int, ts timestamp, p int) partitioned by spec (p) stored as
iceberg;
insert into dr values (1, '2024-01-01 00:00:00', 1);
alter table dr drop column ts;
select count(*) from dr for system_time as of now()
where p = 1 and ts = '2024-01-01 00:00:00';
-- IllegalStateException
{code}
Found while working on IMPALA-15629. Master only pushes down predicates to
Iceberg if one of them is on a partition column, or on a column listed in the
{{impala.iceberg.push_down_hint}} table property, so the repros use these.
IMPALA-15629 would push down predicates on any column.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
