Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24947
Change subject: IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting ...................................................................... IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting IcebergScanPlanner collected the residual expressions of the FileScanTasks into a TreeSet ordered by ExpressionUtil.toSanitizedString(). The sanitized string replaces literal values with placeholders, e.g. both 'id != 12345' and 'id != 54321' become 'id != (5-digit-int)', so the set only kept one of two different residuals. With ICEBERG_PREDICATE_PUSHDOWN_SUBSETTING (on by default) the conjunct of the lost residual was then skipped from the scan node, and the query returned rows that should have been filtered out. This happens when non-identity partition transforms (e.g. truncate or day) leave different residuals for different files. This patch keys the residuals by Expression.toString(), which prints the literals in full. It uses a TreeMap so the order of the retained conjuncts stays deterministic. Consecutive files often have the same residual object (e.g. every file of an unpartitioned table), so the string conversion is skipped for them. This also removes the cost of sanitizing the residual of every file during planning. A predicate that is part of multiple residuals is only retained once. Testing: - Added EE tests with truncate and day partition transforms. Both returned wrong results without the fix. - Ran PlannerTest and test_iceberg.py. Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013 Assisted-by: Claude Opus 5.5 (1M context) <[email protected]> --- M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-predicate-subsetting.test M tests/query_test/test_iceberg.py 3 files changed, 56 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/24947/1 -- To view, visit http://gerrit.cloudera.org:8080/24947 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013 Gerrit-Change-Number: 24947 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
