[
https://issues.apache.org/jira/browse/HIVE-11297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043316#comment-16043316
]
Chao Sun commented on HIVE-11297:
---------------------------------
[~kellyzly]: more changes are needed in {{SplitOpTreeForDPP}}: you also need to
make sure when splitting the OP tree, all pruning sinks will be kept. For
instance, given:
{code}
TS
|
FIL
|
SEL
/ | \
B1 B2 B3
{code}
suppose {{B2}} and {{B3}} contains pruning sinks, in {{SplitOpTreeForDPP}} we
should clone the OP tree for both of them. The result should be:
{code}
Original Tree Generated Tree
TS TS
| |
FIL FIL
| |
SEL SEL
| / \
B1 B2 B3
{code}
> Combine op trees for partition info generating tasks [Spark branch]
> -------------------------------------------------------------------
>
> Key: HIVE-11297
> URL: https://issues.apache.org/jira/browse/HIVE-11297
> Project: Hive
> Issue Type: Bug
> Affects Versions: spark-branch
> Reporter: Chao Sun
> Assignee: liyunzhang_intel
> Attachments: HIVE-11297.1.patch, HIVE-11297.2.patch
>
>
> Currently, for dynamic partition pruning in Spark, if a small table generates
> partition info for more than one partition columns, multiple operator trees
> are created, which all start from the same table scan op, but have different
> spark partition pruning sinks.
> As an optimization, we can combine these op trees and so don't have to do
> table scan multiple times.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
