wangyum opened a new pull request, #36921: URL: https://github.com/apache/spark/pull/36921
### What changes were proposed in this pull request? This PR enhances `PushPredicateThroughNonJoin` to make it do not push down complex filter condition if `grandChild` is `LeafNode`, `Filter` or `Project`. ### Why are the changes needed? Reduce complex expression evaluation. 1. Case from [SPARK-39481](https://issues.apache.org/jira/browse/SPARK-39481?focusedCommentId=17556219&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17556219): Before this PR: ``` == Optimized Logical Plan == Project [int_col#0, string_col#1, pythonUDF0#10 AS int_col_copy#5] +- ArrowEvalPython [copy_int_col(int_col#0)#4], [pythonUDF0#10], 200 +- Project [int_col#0, string_col#1] +- Filter (pythonUDF0#9 >= 3) +- ArrowEvalPython [copy_int_col(int_col#0)#4], [pythonUDF0#9], 200 +- LogicalRDD [int_col#0, string_col#1], false ``` After this PR: ``` == Optimized Logical Plan == Project [int_col#0, string_col#1, pythonUDF0#9 AS int_col_copy#5] +- Filter (isnotnull(pythonUDF0#9) AND (pythonUDF0#9 >= 3)) +- ArrowEvalPython [copy_int_col(int_col#0)#4], [pythonUDF0#9], 200 +- LogicalRDD [int_col#0, string_col#1], false ``` 2. Case from [SPARK-36290](https://github.com/apache/spark/pull/33522/files/4aebe0a19fdf84516e30ab917eb149d39c4d4ac6#r743625946): Before this PR: ``` == Optimized Logical Plan == Project [a#225, b#226, c#236, d#237] +- Join Inner, (CAST(udf(cast(a as string)) AS INT)#250 = CAST(udf(cast(c as string)) AS INT)#251) :- Project [_1#220 AS a#225, _2#221 AS b#226, cast(pythonUDF0#253 as int) AS CAST(udf(cast(a as string)) AS INT)#250] : +- BatchEvalPython [udf(cast(_1#220 as string))], [pythonUDF0#253] : +- Project [_1#220, _2#221] : +- Filter isnotnull(cast(pythonUDF0#252 as int)) : +- BatchEvalPython [udf(cast(_1#220 as string))], [pythonUDF0#252] : +- LocalRelation [_1#220, _2#221] +- Project [_1#231 AS c#236, _2#232 AS d#237, cast(pythonUDF0#255 as int) AS CAST(udf(cast(c as string)) AS INT)#251] +- BatchEvalPython [udf(cast(_1#231 as string))], [pythonUDF0#255] +- Project [_1#231, _2#232] +- Filter isnotnull(cast(pythonUDF0#254 as int)) +- BatchEvalPython [udf(cast(_1#231 as string))], [pythonUDF0#254] +- LocalRelation [_1#231, _2#232] ``` After this PR: ``` == Optimized Logical Plan == Project [a#225, b#226, c#236, d#237] +- Join Inner, (CAST(udf(cast(a as string)) AS INT)#250 = CAST(udf(cast(c as string)) AS INT)#251) :- Project [_1#220 AS a#225, _2#221 AS b#226, cast(pythonUDF0#252 as int) AS CAST(udf(cast(a as string)) AS INT)#250] : +- Filter isnotnull(cast(pythonUDF0#252 as int)) : +- BatchEvalPython [udf(cast(_1#220 as string))], [pythonUDF0#252] : +- LocalRelation [_1#220, _2#221] +- Project [_1#231 AS c#236, _2#232 AS d#237, cast(pythonUDF0#253 as int) AS CAST(udf(cast(c as string)) AS INT)#251] +- Filter isnotnull(cast(pythonUDF0#253 as int)) +- BatchEvalPython [udf(cast(_1#231 as string))], [pythonUDF0#253] +- LocalRelation [_1#231, _2#232] ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Unit test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
