cloud-fan commented on code in PR #58077:
URL: https://github.com/apache/spark/pull/58077#discussion_r3813494004
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala:
##########
@@ -162,17 +179,44 @@ case class InSubqueryExec(
require(result != null || resultBroadcast != null, s"$this has not
finished")
if (result == null && resultBroadcast != null) {
result = resultBroadcast.value
+ // Recompute after broadcast deserialisation: @transient fields are not
serialised.
+ if (plan.output.length > 1 && !isResultUnavailable) {
+ hasNullInMultiColumnResult =
+ result.exists(_.asInstanceOf[InternalRow].anyNull)
+ }
}
}
override def eval(input: InternalRow): Any = {
prepareResult()
- if (isResultUnavailable) true else inSet.eval(input)
+ if (isResultUnavailable) {
+ true
+ } else {
+ val r = inSet.eval(input)
+ // InSet.hasNull misses null fields inside rows; propagate UNKNOWN on a
miss manually.
+ if (r == false && hasNullInMultiColumnResult) null else r
Review Comment:
**Blocking:**
**Blocking:** This global flag is too coarse for row-valued `IN`. For
`(1,1)` versus `(99,NULL)`, the first field comparison is FALSE, so the whole
candidate is FALSE even though the second comparison is UNKNOWN; this branch
instead returns NULL for every miss once any result row contains NULL. Please
preserve per-result-row three-valued comparison state, returning UNKNOWN only
when no candidate is TRUE and at least one candidate remains UNKNOWN, and
update the `SubquerySuite` fixture to cover both definitely-unequal and
potentially-equal nullable rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
