Zoltán Borók-Nagy created IMPALA-15638:
------------------------------------------
Summary: Queries fail on migrated Iceberg tables if string bounds
are not valid UTF-8
Key: IMPALA-15638
URL: https://issues.apache.org/jira/browse/IMPALA-15638
Project: IMPALA
Issue Type: Bug
Components: Frontend
Reporter: Zoltán Borók-Nagy
Assignee: Zoltán Borók-Nagy
Parquet files written without the UTF8 annotation store strings as plain
BINARY. When such files are migrated to Iceberg, their bounds are truncated to
16 bytes, possibly in the middle of a multi-byte character. When a predicate on
such a column is pushed down, Iceberg fails to decode the bound, and the query
fails:
{code:sql}
set parquet_annotate_strings_utf8=false;
create table m (s string) partitioned by (p int) stored as parquet;
insert into m partition (p=1) values ('中文中文中文');
alter table m convert to iceberg;
set parquet_annotate_strings_utf8=true;
select count(*) from m where p = 1 and s = 'x';
-- RuntimeIOException: Failed to decode value as UTF-8:
-- java.nio.HeapByteBuffer[pos=0 lim=16 cap=16]
{code}
Impala could plan the scan without pushing down the predicates in this case.
Found while working on IMPALA-15629. Master only pushes down predicates to
Iceberg if one of them is on a partition column, or on a column listed in the
{{impala.iceberg.push_down_hint}} table property, so the repros use these.
IMPALA-15629 would push down predicates on any column.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
