Zoltán Borók-Nagy created IMPALA-15639:
------------------------------------------
Summary: ALTER TABLE CONVERT TO ICEBERG records wrong column bounds
Key: IMPALA-15639
URL: https://issues.apache.org/jira/browse/IMPALA-15639
Project: IMPALA
Issue Type: Bug
Components: Frontend, Catalog
Reporter: Zoltán Borók-Nagy
Assignee: Zoltán Borók-Nagy
MigrateTableUtil gets the metrics of the data files from Iceberg's
TableMigrationUtil. That reads the min/max statistics in the Parquet footers as
if they had the Iceberg type of the column, which is wrong when the Parquet
type differs:
* INT64 timestamps with MILLIS or NANOS unit: their bounds are stored as if
they were microseconds.
* Columns whose type was changed in the Hive table before the migration, e.g.
DECIMAL(6,3) to DECIMAL(8,4).
Other mismatches are probably affected too, e.g. INT64 timestamps with
isAdjustedToUTC=true written by other engines.
Iceberg uses these bounds to skip data files, so queries that push down
predicates on these columns return wrong results:
{code:sql}
set parquet_timestamp_type=INT64_MILLIS;
create table m (id int, ts timestamp) partitioned by (p int) stored as parquet;
insert into m partition (p=1)
values (1, '2020-01-01 00:00:00'), (2, '2020-06-01 12:00:00');
alter table m convert to iceberg;
select count(*) from m where p = 1 and ts = '2020-01-01 00:00:00';
-- returns 0, expected 1 (same with INT64_NANOS)
create table d (id int, d decimal(6,3)) partitioned by (p int) stored as
parquet;
insert into d partition (p=1) values (1, 1.5);
alter table d change column d d decimal(8,4);
alter table d convert to iceberg;
select count(*) from d where p = 1 and d = 1.5;
-- returns 0, expected 1
{code}
Possible fix:
* CONVERT TO ICEBERG shouldn't record bounds for columns whose Parquet type
doesn't match the Iceberg type.
* Tables that were migrated earlier keep the wrong bounds. Impala could detect
the migrated data files by their location: they keep their original paths
outside of the table's data location ({{<table location>/data}} by default).
For tables with such files only predicates on partition columns would be pushed
down. This also covers tables migrated by other engines, and the restriction
goes away when the files are rewritten, e.g. by OPTIMIZE.
Found while working on IMPALA-15629. Master only pushes down predicates to
Iceberg if one of them is on a partition column, or on a column listed in the
{{impala.iceberg.push_down_hint}} table property, so the repros use these.
IMPALA-15629 would push down predicates on any column.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
