jiwen624 opened a new pull request, #58058:
URL: https://github.com/apache/spark/pull/58058
### What changes were proposed in this pull request?
Currently `FilterExec` matches an `otherPred` to its guarding `IsNotNull` by
attribute reference, so an `IsNotNull` over a complex child (`IsNotNull(b.c)`)
never matches and is deferred until after the predicates that use it. This fix
emits it just before an otherPred containing that child, in both
`generatePredicateCode` and the CSE branch of `doConsume` (which must stay in
lockstep).
### Why are the changes needed?
The guard runs after the predicate it guards:
```scala
case class C(d: Int); case class B(c: C = null); case class A(b: B = null)
val ds = spark.createDataset(Seq(A(null), A(B(null)), A(B(C(0))),
A(B(C(1))))).map(identity)
.where($"b".isNotNull).select($"b".as[B])
val isDZero = udf((c: C) => c.d == 0)
ds.filter($"c".isNotNull).filter(not(isDZero($"c"))).show()
```
ScalaUDF passes null to its deserializer instead of short-circuiting, so
this throws EXPRESSION_DECODING_FAILED under whole-stage codegen while
interpreted mode returns the correct row. Only UDFs taking the nested struct
are affected.
### Does this PR introduce any user-facing change?
No API change. The failed query above now succeeds.
### How was this patch tested?
Unit test cases added.
### Was this patch authored or co-authored using generative AI tooling?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
