Hello Peter Rozsa, Noemi Pap-Takacs, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24572
to look at the new patch set (#9).
Change subject: IMPALA-15171: Null out Iceberg delete file path slot after the
join
......................................................................
IMPALA-15171: Null out Iceberg delete file path slot after the join
IcebergScanPlanner materializes the INPUT__FILE__NAME (file path)
virtual slot on the data scan tuple so the position-delete join
(IcebergDeleteJoinNode) can use it as a join key. When the query does
not otherwise need the file path, the slot is still carried in the
join's output tuple and propagates through downstream EXCHANGE
operators, where the (typically long) file path string is deep-copied
and serialized for every surviving row.
The planner now detects when it materialized the file path slot solely
for the delete join, i.e. no operator above the join references it. When
the flag is set, IcebergDeleteNode nulls out the file path slot on
its output rows in GetNext() so it won't get copied downstream.
When some data files have deletes and some do not, the plan is a UNION
ALL of the delete anti-join (data files with deletes) and a plain scan
(data files without deletes). Both branches share the same tuple
descriptor, so the without-deletes scan also materializes the file path
slot even though it never goes through IcebergDeleteNode. Previously
nothing nulled that branch, so it still propagated the file path
string up through the UNION.
This is now handled symmetrically: the same flag is passed to the
without-deletes IcebergScanNode via THdfsScanNode.clear_file_path_slot,
and FileMetadataUtils::AddFileLevelVirtualColumns leaves the
INPUT__FILE__NAME slot NULL instead of assigning the filename. The tuple
layout is unchanged, so UNION pass-through still holds. The with-deletes
data scan is unaffected (it still needs the path as the join key; the
delete node nulls it after the lookup).
A new tag "clear file path slot" is added to operators (printed with
EXTENDED level or above) that are configured to clear the file path
slot.
Improvement
I used the following query (ice_lineitem has data files with and without
deletes):
SELECT count(*)
FROM ice_lineitem a, ice_lineitem b
WHERE a.l_orderkey=b.l_orderkey;
TotalBytesSent: 914.26 MB -> 246.84 MB
HASH JOIN Peak mem usage: 558.05 MB -> 238.05 MB
Testing
* added e2e test for the above query
* updated planner tests
Change-Id: I7e662cb6a98dd3e687185d384731d6be45f91b94
Assisted-by: Claude Opus 4.8 (1M context) <[email protected]>
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M be/src/exec/iceberg-delete-node.cc
M be/src/exec/iceberg-delete-node.h
M common/thrift/PlanNodes.thrift
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergDeleteJoinNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
M tests/query_test/test_iceberg.py
11 files changed, 196 insertions(+), 17 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/24572/9
--
To view, visit http://gerrit.cloudera.org:8080/24572
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7e662cb6a98dd3e687185d384731d6be45f91b94
Gerrit-Change-Number: 24572
Gerrit-PatchSet: 9
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]>
Gerrit-Reviewer: Peter Rozsa <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
