andygrove commented on code in PR #6785:
URL: https://github.com/apache/datafusion-comet/pull/6785#discussion_r4601391262
##########
spark/src/main/scala/org/apache/spark/sql/comet/operators.scala:
##########
@@ -3206,11 +3205,23 @@ object CometBroadcastNestedLoopJoinExec extends
CometOperatorSerde[BroadcastNest
" DataFusion's NestedLoopJoinExec does not provide. Affects:
LeftOuter+BuildLeft," +
" RightOuter+BuildRight, FullOuter, LeftSemi+BuildLeft,
LeftAnti+BuildLeft."
+ // Spark's nested loop join keeps its streamed side's order and reports it,
so a sort above it
+ // may be gone, but DataFusion's emits its rows build row by build row.
Remove this fallback once
+ // the native join keeps that order.
+ private[apache] val streamedOrderReason: String =
+ "nested loop join over an ordered streamed side is not supported natively,
because the " +
+ "native join does not keep that order " +
+ "(https://github.com/apache/datafusion-comet/issues/6898)"
+
override def getSupportLevel(op: BroadcastNestedLoopJoinExec): SupportLevel =
(op.joinType, op.buildSide) match {
case (Inner, _) | (LeftOuter, BuildRight) | (RightOuter, BuildLeft) |
(LeftSemi, BuildRight) | (LeftAnti, BuildRight) =>
- Compatible(None)
+ if (op.outputOrdering.nonEmpty) {
Review Comment:
This makes every nested loop join over an ordered streamed side fall back,
including ones where nothing above relies on the order, so those queries get
slower than in 1.2. I agree there's no way to tell from the plan that a sort
was dropped. Could we keep #6898 open (drop it from "Closes") to track removing
this once DataFusion's nested loop join keeps probe order? Then the fallback
has an issue to point back to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]