[
https://issues.apache.org/jira/browse/SPARK-45099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18147704#comment-18147704
]
Saath commented on SPARK-45099:
-------------------------------
Hi I did not see this open and have addressed in
https://issues.apache.org/jira/browse/SPARK-62006 with a PR to fix
https://github.com/apache/spark/pull/60675
> SortMergeExec with Outer using join forgets sort information
> ------------------------------------------------------------
>
> Key: SPARK-45099
> URL: https://issues.apache.org/jira/browse/SPARK-45099
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.4.1
> Reporter: Huw
> Priority: Minor
>
> When performing a 'using' join with a sort hint in a full outer, the
> ResolveNaturalAndUsingJoin will kick in and build a new join with Equality
> conditions and a Projection like this:
> {quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l,
> r)), l.name)() }
> {quote}
> There's nothing wrong with this per se, but, SortMergeJoinExec has it's
> output ordering for a full outer join as empty, even though these join pairs
> in their final coalesced form actually are ordered.
> This means that code like this:
> {quote}frames.reduceLeft(case (l, r) => l.join(r.hint("merge"), usingColumns
> = Seq("a", "b"), joinType = "outer"))
> {quote}
> Given a non empty list of frames, will not 'stream' without a shuffle step,
> as each join forgets its sort order.
> Ideally this whole operation wouldn't require any shuffles if all the frames
> are grouped and sorted by the keys.
> (Forgive the parens instead of brackets the code snippet please, Jira was
> inferring macros)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]