Aleksandr Efimov created IMPALA-15280:
-----------------------------------------
Summary: Descriptor table lists partitions of scans that are not
in the final plan
Key: IMPALA-15280
URL: https://issues.apache.org/jira/browse/IMPALA-15280
Project: IMPALA
Issue Type: Improvement
Reporter: Aleksandr Efimov
Assignee: Aleksandr Efimov
{\{HdfsScanNode}} registers the partitions it will read with the descriptor
table from
{\{computeScanRangeLocations()}}, which \{{init()}} calls while the scan's
subtree is being
built:
{code:java}
analyzer.getDescTbl().addReferencedPartition(tbl_, partition.getId());
{code}
{\{DescriptorTable.toThrift()}} then sends the backend exactly those partitions
for every
table except a table sink's target, which keeps all of them. So the set
describes every
scan the planner ever constructed rather than the scans that ended up in the
plan:
anything built and later replaced leaves its partitions behind, and the backend
gets
partition metadata that no scan reads. The TODO above the tuple descriptors in
{\{toThrift()}} already names the situation:
{code:java}
// TODO: Ideally, we should call tupleDesc.checkIsExecutable() here, but there
// currently are several situations in which we send materialized tuples without
// a mem layout to the BE, e.g., when unnesting unions or when replacing plan
// trees with an EmptySetNode.
{code}
I did not find a query on master that reproduces this. The subtree replacements
I looked
at - \{{createScanNode()}} when the conjuncts are implied false, the empty SPJ
result set
in \{{createSelectPlan()}}, and union operands dropped for the same reason -
all decide
before the discarded side is planned, so nothing is registered and then thrown
away. I did
not try to establish that no other path exists.
IMPALA-7996 does hit it. It replaces the pruned input of a constant-false outer
join after
that input has been built, and
\{{PlannerTestBase.testHdfsPartitionsReferenced}}, which
asserts that every partition in the descriptor table is covered by a scan
range, fails on
the \{{PlannerTest#testEmpty}} queries it adds.
Proposal: have each \{{HdfsScanNode}} remember the ids it registered, and once
the plan is
final - after \{{Planner.createPlans()}} in \{{Frontend.createExecRequest()}},
before the
descriptor table is serialized - keep the union over the scans still reachable
from the
plan fragments.
Keeping what the surviving scans registered, rather than deriving the
partitions again, is
what bounds the risk. \{{HdfsScanNode}} is the only caller of
\{{addReferencedPartition()}},
so the resulting set is by construction a subset of today's, and it is the same
set
whenever every scan that was built survives - which, per the paragraph above,
is every
plan on master. No plan should change here; the change is a prerequisite rather
than a fix
with its own visible effect.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
