peter-toth commented on code in PR #60176:
URL: https://github.com/apache/spark/pull/60176#discussion_r4494132398
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala:
##########
@@ -88,6 +88,11 @@ object InjectRuntimeFilter extends Rule[LogicalPlan] with
PredicateHelper with J
if (aggregate.containsPattern(PYTHON_UDF)) {
return filterApplicationSidePlan
}
+ // The subquery evaluates the creation side again, apart from the join. A
non-deterministic
+ // filter or key can give it different keys than the join sees.
+ if (!aggregate.deterministic) {
Review Comment:
Thanks, good point. `deterministic` doesn't see a leaf that returns other
rows when read again, like a cache whose plan isn't repeatable or a V2 scan
with a pushed limit, top-N or sample. Your `RuntimeFilterSourceAnalysis` covers
these, but only for a hinted side. This PR fixes the default path, and it has
to go back to 4.0, where none of the flags it would need exist. So I'd keep
this PR to the visible non-determinism the ticket shows, and I added the rest
as a known limit to the description. Once #58635 is in, the default path could
call the same check on master for the hidden cases, for DPP too. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
