[
https://issues.apache.org/jira/browse/HIVE-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122014#comment-15122014
]
Prasanth Jayachandran commented on HIVE-12893:
----------------------------------------------
The main cause of this bug is this line
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java#L305
When SDPO removes the RS (and SEL) inserted by Bucketing Sorting (this is
inserted in SemanticAnalyzer), it changes the plan from TS->SEL->RS->SEL->FS to
TS->SEL->FS. What the condition aboves says is if the schema of both SEL does
not match then it means that some columns are constant folded and
PartitionConditionRemover removed the ExprNodeColumnDesc for the partition
columns that are folded. So there will be mismatch between RowSchema of both
SEL operators. In such cases, SDPO used to bail out. Even if only a subset of
partition columns are folded the above condition will disable SDPO. I tried
fixing this condition but that lead to more fragile checks in SDPO. To avoid
that, in this patch I moved the SDPO before PartitionConditionRemover and after
first ConstantPropagation optimization. This lets me avoid fragile checks in
SDPO.
With this change, the check for schema mismatch in the above link is no longer
required. I just left it there for safety :)
The allStaticPartitions() method in the patch returns false if it finds atleast
one non-folded column in which case the optimization proceeds ahead. The newly
tests shows it clearly in the explain plan.
> Sorted dynamic partition does not work if subset of partition columns are
> constant folded
> -----------------------------------------------------------------------------------------
>
> Key: HIVE-12893
> URL: https://issues.apache.org/jira/browse/HIVE-12893
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
> Affects Versions: 1.3.0, 2.0.0
> Reporter: Yi Zhang
> Assignee: Prasanth Jayachandran
> Attachments: HIVE-12893.1.patch, HIVE-12893.2.patch,
> HIVE-12893.2.patch
>
>
> If all partition columns are constant folded then sorted dynamic partitioning
> should not be used as it is similar to static partitioning. But if only
> subset of partition columns are constant folded sorted dynamic partition
> optimization will be helpful. Currently, this optimization is disabled if
> atleast one partition column constant folded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
