[
https://issues.apache.org/jira/browse/IMPALA-15634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arnab Karmakar resolved IMPALA-15634.
-------------------------------------
Resolution: Fixed
> Iceberg LIKE predicate push down returns wrong results
> ------------------------------------------------------
>
> Key: IMPALA-15634
> URL: https://issues.apache.org/jira/browse/IMPALA-15634
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Zoltán Borók-Nagy
> Assignee: Arnab Karmakar
> Priority: Critical
> Labels: correctness, impala-iceberg
> Fix For: Impala 5.0.0
>
>
> IcebergPredicateConverter has several problems with LIKE predicates:
> * A pattern like {{'ab_'}} is converted to {{startsWith('ab')}} and marked as
> fully converted. So the scanner doesn't evaluate it on identity partition
> columns, and SHOW FILES and ALTER TABLE DROP PARTITION use it as an exact
> filter.
> * NOT is applied to relaxed predicates: {{NOT (s LIKE 'a%b')}} becomes {{NOT
> startsWith(s, 'a')}}, which skips files with values like {{'axc'}}.
> * Backslashes are not interpreted the way the backend interprets them. The
> backend matches constant and simple prefix patterns verbatim, see
> {{LikePredicate::OptimizeConstantPatternMatch()}}. In its regex path, a
> backslash escapes any character, and a trailing lone backslash is dropped.
> The converter only unescapes a backslash that is followed by %, _ or another
> backslash.
> {code:sql}
> create table p (i int, s string) partitioned by spec (s) stored as iceberg;
> insert into p values (1, 'abcd'), (2, 'abc');
> select i from p where s like 'ab_';
> -- returns 1 and 2, expected 2
> alter table p drop partition (s like 'ab_');
> -- drops both partitions, expected only s='abc'
> create table t (s string) stored as iceberg
> tblproperties ('impala.iceberg.push_down_hint'='s');
> insert into t values ('axc');
> insert into t values ('abcd');
> insert into t values ('dab');
> insert into t values ('dcd');
> select count(*) from t where s not like 'a%b'; -- 2, expected 4
> select count(*) from t where s not like 'ab_'; -- 3, expected 4
> create table e (s string) stored as iceberg
> tblproperties ('impala.iceberg.push_down_hint'='s');
> insert into e values ('abxc');
> insert into e values (concat('a', chr(92), chr(92), 'bz'));
> insert into e values ('a%b');
> select count(*) from e where s like 'a\\b%c'; -- 0, expected 1 (1st row)
> select count(*) from e where s like 'a\\\\b%'; -- 0, expected 1 (2nd row)
> select count(*) from e where s like 'a\\%b\\'; -- 0, expected 1 (3rd row)
> {code}
> Found while working on IMPALA-15629. Master only pushes down predicates to
> Iceberg if one of them is on a partition column, or on a column listed in the
> {{impala.iceberg.push_down_hint}} table property, so the repros use these.
> IMPALA-15629 would push down predicates on any column.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
