sunchao commented on code in PR #57485:
URL: https://github.com/apache/spark/pull/57485#discussion_r3642689568
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala:
##########
@@ -54,6 +54,12 @@ object InjectRuntimeFilter extends Rule[LogicalPlan] with
PredicateHelper with J
filterApplicationSidePlan: LogicalPlan,
filterCreationSideKey: Expression,
filterCreationSidePlan: LogicalPlan): LogicalPlan = {
+ // Runtime filters are introduced after subquery optimization, so Python
UDFs in a new
+ // creation-side subquery cannot be extracted into a Python evaluation
operator.
+ if (filterCreationSidePlan.containsPattern(PYTHON_UDF)) {
Review Comment:
[P2] Check the pruned Bloom aggregate rather than the original creation plan.
A creation-side projection can contain a Python UDF used only by the outer
query output while the join and Bloom filter use the original, Python-free
column. The existing `ColumnPruning(Aggregate(...))` removes that UDF from the
actual Bloom scalar subquery, so this filter is safe; the new pre-pruning guard
nevertheless drops it. Reproduced on this exact head: the equivalent plain join
injects one Bloom filter, the output-only creation-side UDF join injects zero,
and the column-pruned Bloom aggregate contains no Python UDF. Please construct
and prune the aggregate before checking
`aggregate.containsPattern(PYTHON_UDF)`, and add ordinary Python and
Arrow/Pandas regressions that assert the safe creation-side Bloom filter is
preserved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
