[
https://issues.apache.org/jira/browse/HIVE-30252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30252:
----------------------------------
Description:
*Problem*
{{HiveIcebergStorageHandler.decomposePredicate}} removes the children of the
top-level filter expression that reference {{FILE__PATH}},
{{PARTITION__SPEC__ID}}, {{PARTITION__KEY}}, {{PARTITION__NAME}},
{{ROW__LINEAGE__ID}} or {{LAST__UPDATED__SEQUENCE__NUMBER}}, without checking
that the expression is an AND. For an OR this narrows the pushed predicate:
Iceberg prunes the data files that match only the
removed part, and their rows are missing from the result. There is no error.
The same narrowed predicate goes to
{{SessionStateUtil.setConflictDetectionFilter}}, so a DML with such a WHERE
clause checks too few files for conflicting concurrent writes.
*Reproducer*
{code:sql}
create table t_or(id int, v string) stored by iceberg stored as orc
tblproperties('format-version'='2');
insert into t_or values (1, 'a');
insert into t_or values (2, 'b');
select * from t_or where id = 1 or PARTITION__SPEC__ID = 0;
select * from t_or where id = 1 or FILE__PATH is not null;
select * from t_or where id = 1 or v = 'b';
{code}
- expected for all three: (1, a), (2, b)
- actual: the first two return only (1, a); the third, without a virtual
column, is correct
The result is the same with {{hive.fetch.task.conversion=none}}. EXPLAIN of
the first query:
{noformat}
filterExpr: (id = 1) (type: boolean)
predicate: ((PARTITION__SPEC__ID = 0) or (id = 1)) (type: boolean)
{noformat}
A nested OR, such as {{id = 1 or (id = 2 and FILE__PATH = 'x')}}, is narrowed
the same way.
*Proposed fix*
Remove only the conjuncts of a top-level AND that reference these columns.
Push nothing when any other expression references one.
> Iceberg: A filter that ORs a virtual column with a data column pushes down
> only the data column and loses rows
> --------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30252
> URL: https://issues.apache.org/jira/browse/HIVE-30252
> Project: Hive
> Issue Type: Bug
> Reporter: Denys Kuzmenko
> Priority: Minor
>
> *Problem*
>
>
> {{HiveIcebergStorageHandler.decomposePredicate}} removes the children of
> the top-level filter expression that reference {{FILE__PATH}},
> {{PARTITION__SPEC__ID}}, {{PARTITION__KEY}}, {{PARTITION__NAME}},
> {{ROW__LINEAGE__ID}} or {{LAST__UPDATED__SEQUENCE__NUMBER}}, without
> checking that the expression is an AND. For an OR this narrows the pushed
> predicate: Iceberg prunes the data files that match only the
> removed part, and their rows are missing from the result. There is no
> error.
>
>
>
>
> The same narrowed predicate goes to
> {{SessionStateUtil.setConflictDetectionFilter}}, so a DML with such a WHERE
> clause checks too few files for conflicting concurrent writes.
>
>
>
>
> *Reproducer*
>
>
> {code:sql}
>
>
> create table t_or(id int, v string) stored by iceberg stored as orc
> tblproperties('format-version'='2');
>
> insert into t_or values (1, 'a');
>
>
> insert into t_or values (2, 'b');
>
>
>
>
>
> select * from t_or where id = 1 or PARTITION__SPEC__ID = 0;
>
>
> select * from t_or where id = 1 or FILE__PATH is not null;
>
>
> select * from t_or where id = 1 or v = 'b';
>
>
> {code}
> - expected for all three: (1, a), (2, b)
>
>
> - actual: the first two return only (1, a); the third, without a virtual
> column, is correct
>
>
>
>
> The result is the same with {{hive.fetch.task.conversion=none}}. EXPLAIN of
> the first query:
>
> {noformat}
>
>
> filterExpr: (id = 1) (type: boolean)
>
>
> predicate: ((PARTITION__SPEC__ID = 0) or (id = 1)) (type: boolean)
>
>
> {noformat}
>
>
>
>
>
> A nested OR, such as {{id = 1 or (id = 2 and FILE__PATH = 'x')}}, is
> narrowed the same way.
>
>
>
>
> *Proposed fix*
>
>
> Remove only the conjuncts of a top-level AND that reference these columns.
> Push nothing when any other expression references one.
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
