acvictor commented on code in PR #11657:
URL:
https://github.com/apache/incubator-gluten/pull/11657#discussion_r2885368481
##########
gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala:
##########
@@ -257,7 +258,34 @@ case class WholeStageTransformer(child: SparkPlan,
materializeInput: Boolean = f
PlanBuilder.makePlan(substraitContext,
Lists.newArrayList(childCtx.root), outNames)
}
- WholeStageTransformContext(planNode, substraitContext, isCudf)
+ WholeStageTransformContext(
+ planNode,
+ substraitContext,
+ isCudf,
+ hasNonDeterministicExprInJoinProbe(child))
+ }
+
+ /**
+ * Checks whether any HashJoin's probe (streamed) side contains
non-deterministic expressions.
+ * When true, ValueStream dynamic filter pushdown must be disabled because
the dynamic filter
+ * would be applied at the scan level (below the non-deterministic Project),
changing how many
+ * times the non-deterministic expression is evaluated and thus altering its
output sequence. See
+ * SPARK-10316.
+ */
Review Comment:
I have changed the comment "the dynamic filter would be applied at the scan
level" to "if left enabled, the dynamic filter would filter rows at the
ValueStream" to make it clear that the comment describes what would go wrong
without the disable, not the disabled state itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
