[
https://issues.apache.org/jira/browse/HIVE-8024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127504#comment-14127504
]
Chao commented on HIVE-8024:
----------------------------
There's a problem again:
Suppose the operator tree is the following:
{code}
TS_0 TS_2
\ /
UNION_3
|
SEL_4
{code}
After removing UNION operator, it will look like this:
{code}
TS_0 TS_2
\ /
SEL_4
{code}
(Again, I ignored some operators, but you get the idea.)
Then, we could have MapWork 1 starts from {{TS_0}}, and MapWork 2 starts from
{{TS_2}}.
Now, when MapWork 1 initialize itself, it will initialize the operator tree,
starting from {{TS_0}, and go down the tree.
When it gets to {{SEL_4}}, it will not be able to initialize it, because not
all of {{SEL_4}}'s parent are initialize at that point.
Hence, the execution will fail.
> Find out whether it's possible to remove UnionOperator from original operator
> tree [Spark Branch]
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-8024
> URL: https://issues.apache.org/jira/browse/HIVE-8024
> Project: Hive
> Issue Type: Task
> Components: Spark
> Reporter: Chao
> Assignee: Chao
>
> Currently, after operator tree is processed, the generated works with union
> operators will go through {{GenSparkUtils::removeUnionOperators}}, which will
> clone the original operator plan associated with the work, and remove union
> operators in it. This caused some issues as seen, for example, in HIVE-7870.
> This JIRA is created to find out whether it's possible to just remove the
> union operators in the original plan.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
