Dilip Biswal created SPARK-25168:
------------------------------------
Summary: PlanTest.comparePlans may make a supplied resolved plan
unresolved.
Key: SPARK-25168
URL: https://issues.apache.org/jira/browse/SPARK-25168
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.3.1
Reporter: Dilip Biswal
Hello,
I came across this behaviour while writing unit test for one of my PR. We use
the utlility class PlanTest for writing plan verification tests. Currently when
we call comparePlans and the resolved input plan contains a Join operator, the
plan becomes unresolved after the call to
`normalizePlan(normalizeExprIds(plan1))`. The reason for this is that, after
cleaning the expression ids, duplicateResolved returns false making the Join
operator unresolved.
{code}
def duplicateResolved: Boolean =
left.outputSet.intersect(right.outputSet).isEmpty
// Joins are only resolved if they don't introduce ambiguous expression ids.
// NaturalJoin should be ready for resolution only if everything else is
resolved here
lazy val resolvedExceptNatural: Boolean = {
childrenResolved &&
expressions.forall(_.resolved) &&
duplicateResolved &&
condition.forall(_.dataType == BooleanType)
}
{code}
Please note that, plan verification actually works fine. It just looked awkward
to compare two unresolved plan for equality.
I am opening this ticket to discuss if it is an okay behaviour. If its an
tolerable behaviour then we can close the ticket.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
