[
https://issues.apache.org/jira/browse/IMPALA-15280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Efimov updated IMPALA-15280:
--------------------------------------
Description:
HdfsScanNode registers the partitions it will read with the descriptor table
from computeScanRangeLocations(), which init() calls while the scan's subtree
is being built:
{code:java}
analyzer.getDescTbl().addReferencedPartition(tbl_, partition.getId());
{code}
DescriptorTable.toThrift() then sends the backend exactly those partitions for
every table except a table sink's target, which keeps all of them. So the set
describes every scan the planner ever constructed rather than the scans that
ended up in the plan: anything built and later replaced leaves its partitions
behind, and the backend gets partition metadata that no scan reads. The TODO
above the tuple descriptors in toThrift() already names the situation:
{code:java}
// TODO: Ideally, we should call tupleDesc.checkIsExecutable() here, but there
// currently are several situations in which we send materialized tuples without
// a mem layout to the BE, e.g., when unnesting unions or when replacing plan
// trees with an EmptySetNode.
{code}
I did not find a query on master that reproduces this. The subtree replacements
I looked at - createScanNode() when the conjuncts are implied false, the empty
SPJ result set in createSelectPlan(), and union operands dropped for the same
reason - all decide before the discarded side is planned, so nothing is
registered and then thrown away. I did not try to establish that no other path
exists.
IMPALA-7996 does hit it. It replaces the pruned input of a constant-false outer
join after that input has been built, and
PlannerTestBase.testHdfsPartitionsReferenced, which asserts that every
partition in the descriptor table is covered by a scan range, fails on the
PlannerTest#testEmpty queries it adds.
Proposal: have each HdfsScanNode remember the ids it registered, and once the
plan is final - after Planner.createPlans() in Frontend.createExecRequest(),
before the descriptor table is serialized - keep the union over the scans still
reachable from the plan fragments.
Keeping what the surviving scans registered, rather than deriving the
partitions again, is what bounds the risk. HdfsScanNode is the only caller of
addReferencedPartition(), so the resulting set is by construction a subset of
today's, and it is the same set whenever every scan that was built survives -
which, per the paragraph above, is every plan on master. No plan should change
here; the change is a prerequisite rather than a fix with its own visible
effect.
was:
{\{HdfsScanNode}} registers the partitions it will read with the descriptor
table from
{\{computeScanRangeLocations()}}, which \{{init()}} calls while the scan's
subtree is being
built:
{code:java}
analyzer.getDescTbl().addReferencedPartition(tbl_, partition.getId());
{code}
{\{DescriptorTable.toThrift()}} then sends the backend exactly those partitions
for every
table except a table sink's target, which keeps all of them. So the set
describes every
scan the planner ever constructed rather than the scans that ended up in the
plan:
anything built and later replaced leaves its partitions behind, and the backend
gets
partition metadata that no scan reads. The TODO above the tuple descriptors in
{\{toThrift()}} already names the situation:
{code:java}
// TODO: Ideally, we should call tupleDesc.checkIsExecutable() here, but there
// currently are several situations in which we send materialized tuples without
// a mem layout to the BE, e.g., when unnesting unions or when replacing plan
// trees with an EmptySetNode.
{code}
I did not find a query on master that reproduces this. The subtree replacements
I looked
at - \{{createScanNode()}} when the conjuncts are implied false, the empty SPJ
result set
in \{{createSelectPlan()}}, and union operands dropped for the same reason -
all decide
before the discarded side is planned, so nothing is registered and then thrown
away. I did
not try to establish that no other path exists.
IMPALA-7996 does hit it. It replaces the pruned input of a constant-false outer
join after
that input has been built, and
\{{PlannerTestBase.testHdfsPartitionsReferenced}}, which
asserts that every partition in the descriptor table is covered by a scan
range, fails on
the \{{PlannerTest#testEmpty}} queries it adds.
Proposal: have each \{{HdfsScanNode}} remember the ids it registered, and once
the plan is
final - after \{{Planner.createPlans()}} in \{{Frontend.createExecRequest()}},
before the
descriptor table is serialized - keep the union over the scans still reachable
from the
plan fragments.
Keeping what the surviving scans registered, rather than deriving the
partitions again, is
what bounds the risk. \{{HdfsScanNode}} is the only caller of
\{{addReferencedPartition()}},
so the resulting set is by construction a subset of today's, and it is the same
set
whenever every scan that was built survives - which, per the paragraph above,
is every
plan on master. No plan should change here; the change is a prerequisite rather
than a fix
with its own visible effect.
> Descriptor table lists partitions of scans that are not in the final plan
> -------------------------------------------------------------------------
>
> Key: IMPALA-15280
> URL: https://issues.apache.org/jira/browse/IMPALA-15280
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Aleksandr Efimov
> Assignee: Aleksandr Efimov
> Priority: Major
>
> HdfsScanNode registers the partitions it will read with the descriptor table
> from computeScanRangeLocations(), which init() calls while the scan's subtree
> is being built:
>
> {code:java}
> analyzer.getDescTbl().addReferencedPartition(tbl_, partition.getId());
> {code}
>
> DescriptorTable.toThrift() then sends the backend exactly those partitions
> for every table except a table sink's target, which keeps all of them. So the
> set describes every scan the planner ever constructed rather than the scans
> that ended up in the plan: anything built and later replaced leaves its
> partitions behind, and the backend gets partition metadata that no scan
> reads. The TODO above the tuple descriptors in toThrift() already names the
> situation:
>
> {code:java}
> // TODO: Ideally, we should call tupleDesc.checkIsExecutable() here, but there
> // currently are several situations in which we send materialized tuples
> without
> // a mem layout to the BE, e.g., when unnesting unions or when replacing plan
> // trees with an EmptySetNode.
> {code}
>
> I did not find a query on master that reproduces this. The subtree
> replacements I looked at - createScanNode() when the conjuncts are implied
> false, the empty SPJ result set in createSelectPlan(), and union operands
> dropped for the same reason - all decide before the discarded side is
> planned, so nothing is registered and then thrown away. I did not try to
> establish that no other path exists.
>
> IMPALA-7996 does hit it. It replaces the pruned input of a constant-false
> outer join after that input has been built, and
> PlannerTestBase.testHdfsPartitionsReferenced, which asserts that every
> partition in the descriptor table is covered by a scan range, fails on the
> PlannerTest#testEmpty queries it adds.
>
> Proposal: have each HdfsScanNode remember the ids it registered, and once the
> plan is final - after Planner.createPlans() in Frontend.createExecRequest(),
> before the descriptor table is serialized - keep the union over the scans
> still reachable from the plan fragments.
>
> Keeping what the surviving scans registered, rather than deriving the
> partitions again, is what bounds the risk. HdfsScanNode is the only caller of
> addReferencedPartition(), so the resulting set is by construction a subset of
> today's, and it is the same set whenever every scan that was built survives -
> which, per the paragraph above, is every plan on master. No plan should
> change here; the change is a prerequisite rather than a fix with its own
> visible effect.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
