[ 
https://issues.apache.org/jira/browse/SPARK-45099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18147704#comment-18147704
 ] 

Saath commented on SPARK-45099:
-------------------------------

Hi I did not see this open and have addressed in 
https://issues.apache.org/jira/browse/SPARK-62006 with a PR to fix 
https://github.com/apache/spark/pull/60675

> SortMergeExec with Outer using join forgets sort information
> ------------------------------------------------------------
>
>                 Key: SPARK-45099
>                 URL: https://issues.apache.org/jira/browse/SPARK-45099
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.4.1
>            Reporter: Huw
>            Priority: Minor
>
> When performing a 'using' join with a sort hint in a full outer, the 
> ResolveNaturalAndUsingJoin will kick in and build a new join with Equality 
> conditions and a Projection like this:
> {quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l, 
> r)), l.name)() }
> {quote}
> There's nothing wrong with this per se, but, SortMergeJoinExec has it's 
> output ordering for a full outer join as empty, even though these join pairs 
> in their final coalesced form actually are ordered.
> This means that code like this:
> {quote}frames.reduceLeft(case (l, r) => l.join(r.hint("merge"), usingColumns 
> = Seq("a", "b"), joinType = "outer"))
> {quote}
> Given a non empty list of frames, will not 'stream' without a shuffle step, 
> as each join forgets its sort order.
> Ideally this whole operation wouldn't require any shuffles if all the frames 
> are grouped and sorted by the keys.
> (Forgive the parens instead of brackets the code snippet please, Jira was 
> inferring macros)



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to