Hello Peter Rozsa, Noemi Pap-Takacs, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24947
to look at the new patch set (#2).
Change subject: IMPALA-15421: Fix dropped conjuncts in Iceberg predicate
subsetting
......................................................................
IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
IcebergScanPlanner collected the residual expressions of the
FileScanTasks into a TreeSet ordered by
ExpressionUtil.toSanitizedString(). The sanitized string replaces
literal values with placeholders, e.g. both 'id != 11' and
'id != 21' become 'id != (2-digit-int)', so the set only kept one of
two different residuals. With ICEBERG_PREDICATE_PUSHDOWN_SUBSETTING
(on by default) the conjunct of the lost residual was then skipped
from the scan node, and the query returned rows that should have been
filtered out. This happens when non-identity partition transforms
(e.g. truncate or day) leave different residuals for different files.
This patch keys the residuals by Expression.toString(), which prints
the literals in full. It uses a TreeMap so the order of the retained
conjuncts stays deterministic. Consecutive files often have the same
residual object (e.g. every file of an unpartitioned table), so the
string conversion is skipped for them. This also removes the cost of
sanitizing the residual of every file during planning. A predicate
that is part of multiple residuals is only retained once.
Testing:
- Added EE tests with truncate and day partition transforms. Both
returned wrong results without the fix.
- Ran PlannerTest and test_iceberg.py.
Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013
Assisted-by: Claude Opus 5.5 (1M context) <[email protected]>
---
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
A
testdata/workloads/functional-query/queries/QueryTest/iceberg-predicate-subsetting.test
M tests/query_test/test_iceberg.py
3 files changed, 56 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/24947/2
--
To view, visit http://gerrit.cloudera.org:8080/24947
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013
Gerrit-Change-Number: 24947
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]>
Gerrit-Reviewer: Peter Rozsa <[email protected]>
