[
https://issues.apache.org/jira/browse/SPARK-21998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165403#comment-16165403
]
Takeshi Yamamuro commented on SPARK-21998:
------------------------------------------
I think the orders depend on their children, e.g.
https://github.com/apache/spark/commit/e9c91badce64731ffd3e53cbcd9f044a7593e6b8.
If you have other cases the Spark planner adds unnecessary sorts, could you
put an concrete example? Thanks!
> SortMergeJoinExec should calculate its outputOrdering independent of its
> children's outputOrdering
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-21998
> URL: https://issues.apache.org/jira/browse/SPARK-21998
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Maryann Xue
> Priority: Minor
>
> Right now SortMergeJoinExec calculates its outputOrdering based on its
> children's outputOrdering, thus oftentimes the SortMergeJoinExec's
> outputOrdering is NOT correct until after EnsureRequirements, which happens
> at a rather late stage. As a result, potential optimizations that rely on the
> required/output orderings, like SPARK-18591, will not work for
> SortMergeJoinExec.
> Unlike operators like Project or Filter, which simply preserve the ordering
> of their inputs, the SortMergeJoinExec has a behavior that generates a new
> ordering in its output regardless of the orderings of its children. I think
> the code below together with its comment is buggy.
> {code}
> /**
> * For SMJ, child's output must have been sorted on key or expressions with
> the same order as
> * key, so we can get ordering for key from child's output ordering.
> */
> private def getKeyOrdering(keys: Seq[Expression], childOutputOrdering:
> Seq[SortOrder])
> : Seq[SortOrder] = {
> keys.zip(childOutputOrdering).map { case (key, childOrder) =>
> SortOrder(key, Ascending, childOrder.sameOrderExpressions +
> childOrder.child - key)
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
