[
https://issues.apache.org/jira/browse/SPARK-21998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165765#comment-16165765
]
Maryann Xue commented on SPARK-21998:
-------------------------------------
Thank you for pointing this out, [~maropu]! Yes, you are right.
It would be more accurate to say that the outputOrdering of SortMergeJoinExec
should be a superset of (i.e., satisfy) the join key ordering.
Let's look at {{SELECT * FROM t1 JOIN t2 ON t1.a = t2.z}} for example and focus
on {{t1}} for simplicity,
1) If t1 is not sorted on a, the SMJ outputOrdering should be (a, ASC)
2) If t1 is sorted on a, the SMJ outputOrdering should be (a, ASC)
3) If t1 is sorted both on a and b, the SMJ outputOrdering should at least
include (a, ASC), and whether the SMJ output is still sorted on b remains
implementation specific, and in current Spark SQL implementation it surely is.
But we can imagine a less smart SMJ implementation which cannot avoid extra
sorting on t1 and meanwhile uses an unstable sorting can destroy the sortedness
of column b.
Anyway, my point is the current SMJ implementation will return NOTHING as
outputOrdering when the child ordering is not immediately available and can
only return the right value at a later stage after EnsureRequirements happens.
You can try my query example in
https://issues.apache.org/jira/browse/SPARK-18591?focusedCommentId=16165148&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16165148
with your proposed change for SPARK-18591 and see what's going on.
> SortMergeJoinExec should calculate its outputOrdering independent of its
> children's outputOrdering
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-21998
> URL: https://issues.apache.org/jira/browse/SPARK-21998
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Maryann Xue
> Priority: Minor
>
> Right now SortMergeJoinExec calculates its outputOrdering based on its
> children's outputOrdering, thus oftentimes the SortMergeJoinExec's
> outputOrdering is NOT correct until after EnsureRequirements, which happens
> at a rather late stage. As a result, potential optimizations that rely on the
> required/output orderings, like SPARK-18591, will not work for
> SortMergeJoinExec.
> Unlike operators like Project or Filter, which simply preserve the ordering
> of their inputs, the SortMergeJoinExec has a behavior that generates a new
> ordering in its output regardless of the orderings of its children. I think
> the code below together with its comment is buggy.
> {code}
> /**
> * For SMJ, child's output must have been sorted on key or expressions with
> the same order as
> * key, so we can get ordering for key from child's output ordering.
> */
> private def getKeyOrdering(keys: Seq[Expression], childOutputOrdering:
> Seq[SortOrder])
> : Seq[SortOrder] = {
> keys.zip(childOutputOrdering).map { case (key, childOrder) =>
> SortOrder(key, Ascending, childOrder.sameOrderExpressions +
> childOrder.child - key)
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
