AnhTtis opened a new pull request, #58115: URL: https://github.com/apache/spark/pull/58115
### What changes were proposed in this pull request? In whole-stage code generation, `FilterExec` partitions conjunctive predicates into `notNullPreds` and `otherPreds`. Previously, `notNullPreds` matched any `IsNotNull(a)` where `isNullIntolerant(a)` is true and its references are a subset of the input attributes. This inadvertently collected `IsNotNull` predicates over nested/complex expressions (such as `IsNotNull(GetStructField(b, "c"))`). However, during predicate code generation in `GeneratePredicateHelper.generatePredicateCode` (and the CSE path of `FilterExec.doConsume`), each `otherPred` only checks for guarding `IsNotNull` predicates matching its top-level `Attribute` references (`r`). Consequently, an `IsNotNull` on a nested child expression (`IsNotNull(b.c)`) never matched the root attribute `b`, and was deferred to the end of the filter (emitted after all other predicates). This allowed `null` values in intermediate nested structs to be passed into subsequent expressions/UDFs (e.g. `ScalaUDF`), causing runtime exceptions like `EXPRESSION_DECODING_FAILED` or `NullPointerException`. This PR fixes the issue by restricting `notNullPreds` partitioning in both `FilterExec` and `GeneratePredicateHelper` to `IsNotNull` on direct `Attribute` expressions only (`case IsNotNull(a: Attribute)`). `IsNotNull` predicates over nested/complex expressions remain in `otherPreds` and are evaluated in their natural short-circuit order ahead of the predicates that depend on them. Fixes [SPARK-51356](https://issues.apache.org/jira/browse/SPARK-51356). ### Why are the changes needed? Under whole-stage code generation, filtering on nested struct columns followed by expressions/UDFs on those nested fields fails with runtime errors (such as `EXPRESSION_DECODING_FAILED` or `NullPointerException`) on rows containing intermediate `null` struct fields, while interpreted execution (`spark.sql.codegen.wholeStage=false`) succeeds. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - Added a regression test case in `WholeStageCodegenSuite.scala` testing nested struct filtering under both `spark.sql.codegen.wholeStage=true` and `spark.sql.codegen.wholeStage=false`. - Existing SQL and whole-stage codegen test suites in `sql/core`. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
