Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24723 )
Change subject: IMPALA-15280: Take referenced partitions from the final plan ...................................................................... IMPALA-15280: Take referenced partitions from the final plan HdfsScanNode registers its partitions with the descriptor table from computeScanRangeLocations(), which runs while the scan's subtree is being built. The descriptor table therefore describes every scan the planner ever constructed, not the scans that ended up in the plan. A subtree that is replaced afterwards leaves its partitions behind, and toThrift() ships partition descriptors that nothing reads. The TODO above the tuple descriptors in DescriptorTable.toThrift() already names this situation. Have each HdfsScanNode remember the ids it registered, and once the plan is final, keep the union over the scans still reachable from the plan fragments. Taking what the surviving scans registered, rather than deriving the partitions again, is what makes this safe: HdfsScanNode is the only caller of addReferencedPartition(), so the new set is by construction a subset of the old one, it is the same set whenever every scan that was built survives, and no scan can lose a partition it still reads. Deriving the partitions again from the scan would not give the same answer. The loop in computeScanRangeLocations() walks getSampledOrRawPartitions(), so a TABLESAMPLE scan registers the sampled partitions and not partitions_, and a scan under a simple limit leaves the loop once it has rows enough and never reaches the rest. What each scan registered is the only record of what it asked for. The walk covers every fragment, not the getFragmentsInPlanPreorder() subset that createPlanExecInfo() uses: that one skips fragments reached through a join build, and a scan below one of those still runs. A table whose scans are all gone now ends up with an empty set rather than the partitions its replaced scan had registered. That state is not new: getReferencedPartitions() allocates an empty set for any table toThrift() asks about, so a table whose tuple is still materialized while nothing scans it already ships a descriptor with no partitions - the situation the TODO above the tuple descriptors names. The backend takes it today. The EmptySetNode replacements on master - createScanNode() on conjuncts implied false, the empty SPJ result set, dropped union operands - all happen before the discarded side is planned, so no plan is expected to change here. This is a prerequisite for IMPALA-7996, which replaces the pruned input of a constant-false outer join after that input has been built and trips PlannerTestBase.testHdfsPartitionsReferenced without it. Testing: - PlannerTest, which asserts on every query that each partition in the descriptor table is covered by a scan range. PlannerTestBase used to check only the single-node plan and now checks the distributed and parallel plans too. Change-Id: I553281e7c77e3297585360f37160bc3f9d69a208 Assisted-by: claude-opus-5 (Claude Code) Reviewed-on: http://gerrit.cloudera.org:8080/24723 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 5 files changed, 131 insertions(+), 15 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/24723 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I553281e7c77e3297585360f37160bc3f9d69a208 Gerrit-Change-Number: 24723 Gerrit-PatchSet: 6 Gerrit-Owner: Aleksandr Efimov <[email protected]> Gerrit-Reviewer: Aleksandr Efimov <[email protected]> Gerrit-Reviewer: Aman Sinha <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
