[
https://issues.apache.org/jira/browse/IMPALA-15635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Borók-Nagy resolved IMPALA-15635.
----------------------------------------
Resolution: Fixed
> Iceberg predicate push down doesn't follow Impala's semantics for NaN and -0.0
> ------------------------------------------------------------------------------
>
> Key: IMPALA-15635
> URL: https://issues.apache.org/jira/browse/IMPALA-15635
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Critical
> Labels: correctness, impala-iceberg
> Fix For: Impala 5.0.0
>
>
> Iceberg compares floating-point values by their total order, where -0.0 is
> less than 0.0, and NaN is greater than any other value. In Impala -0.0 equals
> 0.0, and comparisons with NaN are false. IcebergPredicateConverter converts
> predicates on FLOAT and DOUBLE columns without taking this into account:
> * A NaN identity partition matches {{d > 5}}. A -0.0 identity partition
> matches {{d < 0}}, but not {{d = 0}} or {{d >= 0}}. The scanner doesn't
> evaluate predicates on identity partition columns, so queries return wrong
> rows, and ALTER TABLE DROP PARTITION drops partitions that don't match.
> * Iceberg skips data files whose upper bound is -0.0 for {{d = 0}} and {{d >=
> 0}}.
> * Impala writes no NaN value counts and leaves NaNs out of the bounds. So
> Iceberg skips a file with the values 1 and NaN for {{d != 1}}, {{d NOT IN (1,
> 2)}} and {{NOT (d = 1)}}. It also rewrites {{NOT (d < 5)}} to {{d >= 5}},
> which is false for NaN.
> * Narrowing a literal to FLOAT can give -0.0, e.g. {{CAST(-1e-46 AS FLOAT)}}.
> {code:sql}
> create table nan_p (i int, d double) partitioned by spec (d) stored as
> iceberg;
> insert into nan_p values (1, cast('nan' as double)), (2, 7), (3, -3);
> select i from nan_p where d > 5;
> -- returns 1 and 2, expected 2
> alter table nan_p drop partition (d > 5);
> -- drops 2 partitions, expected 1 (d=7)
> create table src (x double) stored as iceberg;
> insert into src values (0), (1);
> -- -x gives -0.0 and -1.0
> create table zero_p (i int, d double) partitioned by spec (d) stored as
> iceberg;
> insert into zero_p select cast(x as int), -x from src;
> select i from zero_p where d = 0;
> -- returns no rows, expected 0
> select i from zero_p where d < 0;
> -- returns 0 and 1, expected 1
> alter table zero_p drop partition (d < 0);
> -- drops 2 partitions, expected 1 (d=-1)
> create table zero_t (d double) stored as iceberg
> tblproperties ('impala.iceberg.push_down_hint'='d');
> insert into zero_t select -x from src;
> select count(*) from zero_t where d = 0; -- 0, expected 1
> create table nan_t (d double, f float) stored as iceberg
> tblproperties ('impala.iceberg.push_down_hint'='d,f');
> insert into nan_t values (1, 1), (cast('nan' as double), cast('nan' as
> float));
> select count(*) from nan_t where d != 1; -- 0, expected 1
> select count(*) from nan_t where f not in (1, 2); -- 0, expected 1
> select count(*) from nan_t where not (d < 5); -- 0, expected 1
> {code}
> Found while working on IMPALA-15629. Master only pushes down predicates to
> Iceberg if one of them is on a partition column, or on a column listed in the
> {{impala.iceberg.push_down_hint}} table property, so the repros use these.
> IMPALA-15629 would push down predicates on any column.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
