Asif created SPARK-49618:
----------------------------
Summary: Union ( & UnionExec) nodes equality not take into account
unaligned positions of branches causing NO ( reuse of exchange and cached plans)
Key: SPARK-49618
URL: https://issues.apache.org/jira/browse/SPARK-49618
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.4.3, 3.3.4, 4.0.0
Reporter: Asif
Fix For: 3.4.4
Ideally a Union( plan1, plan2) and Union (plan2, plan1) are logically equal, so
long as the output attributes of plan1 and plan2 are matching in terms of name
, data type, metdadata etc ( though differing in terms of exprId).
But because current equality and hashCode is dependent on the order of the
children, the canonicalizations do not match.
This causes reuse of exchange not happening in following situations:
Exchange 1. = Union( plan1, plan2)
Exchange 2 = Union( plan2, plan1)
similarly the cached lookup also misses picking the InMemoryRelation.
Will be submitting a PR and bug tests for the above scenarios.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
