Hello Aman Sinha, Quanlong Huang, Zoltan Borok-Nagy, Csaba Ringhofer, Michael
Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24723
to look at the new patch set (#3).
Change subject: IMPALA-15280: Take referenced partitions from the final plan
......................................................................
IMPALA-15280: Take referenced partitions from the final plan
HdfsScanNode registers its partitions with the descriptor table from
computeScanRangeLocations(), which runs while the scan's subtree is
being built. The descriptor table therefore describes every scan the
planner ever constructed, not the scans that ended up in the plan. A
subtree that is replaced afterwards leaves its partitions behind, and
toThrift() ships partition descriptors that nothing reads. The TODO
above the tuple descriptors in DescriptorTable.toThrift() already names
this situation.
Have each HdfsScanNode remember the ids it registered, and once the
plan is final, keep the union over the scans still reachable from the
plan fragments. Taking what the surviving scans registered, rather than
deriving the partitions again, is what makes this safe: HdfsScanNode is
the only caller of addReferencedPartition(), so the new set is by
construction a subset of the old one, it is the same set whenever every
scan that was built survives, and no scan can lose a partition it still
reads.
Deriving the partitions again from the scan would not give the same
answer. The loop in computeScanRangeLocations() walks
getSampledOrRawPartitions(), so a TABLESAMPLE scan registers the
sampled partitions and not partitions_, and a scan under a simple limit
leaves the loop once it has rows enough and never reaches the rest.
What each scan registered is the only record of what it asked for.
The walk covers every fragment, not the getFragmentsInPlanPreorder()
subset that createPlanExecInfo() uses: that one skips fragments reached
through a join build, and a scan below one of those still runs.
A table whose scans are all gone now ends up with an empty set rather
than the partitions its replaced scan had registered. That state is not
new: getReferencedPartitions() allocates an empty set for any table
toThrift() asks about, so a table whose tuple is still materialized
while nothing scans it already ships a descriptor with no partitions -
the situation the TODO above the tuple descriptors names. The backend
takes it today.
The EmptySetNode replacements on master - createScanNode() on conjuncts
implied false, the empty SPJ result set, dropped union operands - all
happen before the discarded side is planned, so no plan is expected to
change here. This is a prerequisite for IMPALA-7996, which replaces the
pruned input of a constant-false outer join after that input has been
built and trips PlannerTestBase.testHdfsPartitionsReferenced without
it.
Testing:
- PlannerTest, which asserts on every query that each partition in the
descriptor table is covered by a scan range. PlannerTestBase used to
check only the single-node plan and now checks the distributed and
parallel plans too.
Change-Id: I553281e7c77e3297585360f37160bc3f9d69a208
Assisted-by: claude-opus-5 (Claude Code)
---
M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
5 files changed, 131 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/24723/3
--
To view, visit http://gerrit.cloudera.org:8080/24723
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I553281e7c77e3297585360f37160bc3f9d69a208
Gerrit-Change-Number: 24723
Gerrit-PatchSet: 3
Gerrit-Owner: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Aman Sinha <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
