[
https://issues.apache.org/jira/browse/SPARK-25168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16587133#comment-16587133
]
Wenchen Fan commented on SPARK-25168:
-------------------------------------
It's not ideal but I don't have a better idea. The unresolved plan is temporary
and will be thrown away after the plan comparison is done.
> PlanTest.comparePlans may make a supplied resolved plan unresolved.
> -------------------------------------------------------------------
>
> Key: SPARK-25168
> URL: https://issues.apache.org/jira/browse/SPARK-25168
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.1
> Reporter: Dilip Biswal
> Priority: Minor
>
> Hello,
> I came across this behaviour while writing unit test for one of my PR. We
> use the utlility class PlanTest for writing plan verification tests.
> Currently when we call comparePlans and the resolved input plan contains a
> Join operator, the plan becomes unresolved after the call to
> `normalizePlan(normalizeExprIds(plan1))`. The reason for this is that, after
> cleaning the expression ids, duplicateResolved returns false making the Join
> operator unresolved.
> {code}
> def duplicateResolved: Boolean =
> left.outputSet.intersect(right.outputSet).isEmpty
> // Joins are only resolved if they don't introduce ambiguous expression ids.
> // NaturalJoin should be ready for resolution only if everything else is
> resolved here
> lazy val resolvedExceptNatural: Boolean = {
> childrenResolved &&
> expressions.forall(_.resolved) &&
> duplicateResolved &&
> condition.forall(_.dataType == BooleanType)
> }
> {code}
> Please note that, plan verification actually works fine. It just looked
> awkward to compare two unresolved plan for equality.
> I am opening this ticket to discuss if it is an okay behaviour. If its an
> tolerable behaviour then we can close the ticket.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
