Zoltán Borók-Nagy created IMPALA-15635:
------------------------------------------
Summary: Iceberg predicate push down doesn't follow Impala's
semantics for NaN and -0.0
Key: IMPALA-15635
URL: https://issues.apache.org/jira/browse/IMPALA-15635
Project: IMPALA
Issue Type: Bug
Components: Frontend
Reporter: Zoltán Borók-Nagy
Assignee: Zoltán Borók-Nagy
Iceberg compares floating-point values by their total order, where -0.0 is less
than 0.0, and NaN is greater than any other value. In Impala -0.0 equals 0.0,
and comparisons with NaN are false. IcebergPredicateConverter converts
predicates on FLOAT and DOUBLE columns without taking this into account:
* A NaN identity partition matches {{d > 5}}. A -0.0 identity partition matches
{{d < 0}}, but not {{d = 0}} or {{d >= 0}}. The scanner doesn't evaluate
predicates on identity partition columns, so queries return wrong rows, and
ALTER TABLE DROP PARTITION drops partitions that don't match.
* Iceberg skips data files whose upper bound is -0.0 for {{d = 0}} and {{d >=
0}}.
* Impala writes no NaN value counts and leaves NaNs out of the bounds. So
Iceberg skips a file with the values 1 and NaN for {{d != 1}}, {{d NOT IN (1,
2)}} and {{NOT (d = 1)}}. It also rewrites {{NOT (d < 5)}} to {{d >= 5}}, which
is false for NaN.
* Narrowing a literal to FLOAT can give -0.0, e.g. {{CAST(-1e-46 AS FLOAT)}}.
{code:sql}
create table nan_p (i int, d double) partitioned by spec (d) stored as iceberg;
insert into nan_p values (1, cast('nan' as double)), (2, 7), (3, -3);
select i from nan_p where d > 5;
-- returns 1 and 2, expected 2
alter table nan_p drop partition (d > 5);
-- drops 2 partitions, expected 1 (d=7)
create table src (x double) stored as iceberg;
insert into src values (0), (1);
-- -x gives -0.0 and -1.0
create table zero_p (i int, d double) partitioned by spec (d) stored as iceberg;
insert into zero_p select cast(x as int), -x from src;
select i from zero_p where d = 0;
-- returns no rows, expected 0
select i from zero_p where d < 0;
-- returns 0 and 1, expected 1
alter table zero_p drop partition (d < 0);
-- drops 2 partitions, expected 1 (d=-1)
create table zero_t (d double) stored as iceberg
tblproperties ('impala.iceberg.push_down_hint'='d');
insert into zero_t select -x from src;
select count(*) from zero_t where d = 0; -- 0, expected 1
create table nan_t (d double, f float) stored as iceberg
tblproperties ('impala.iceberg.push_down_hint'='d,f');
insert into nan_t values (1, 1), (cast('nan' as double), cast('nan' as float));
select count(*) from nan_t where d != 1; -- 0, expected 1
select count(*) from nan_t where f not in (1, 2); -- 0, expected 1
select count(*) from nan_t where not (d < 5); -- 0, expected 1
{code}
Found while working on IMPALA-15629. Master only pushes down predicates to
Iceberg if one of them is on a partition column, or on a column listed in the
{{impala.iceberg.push_down_hint}} table property, so the repros use these.
IMPALA-15629 would push down predicates on any column.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
