joelrobin18 opened a new pull request, #58407:
URL: https://github.com/apache/spark/pull/58407
### What changes were proposed in this pull request?
`GenerateUnsafeProjection` skips the null check when a value's declared
nullability says it cannot be null. For nested values (struct, array, map) the
generated writers hold the value as an object reference and dereference it
unconditionally, so a null arriving in a slot declared non-null throws an NPE
from inside the generated code instead of being written as null.
This adds a null guard at the two points where the check is currently
skipped:
- `writeExpressionsToBuffer`, for a field whose declared nullability (or
`input.isNull == FalseLiteral`) says non-null.
- `writeArrayToBuffer`, for array elements when `containsNull` is false.
The guard writes null via `setNull8Bytes`, matching what
`InterpretedUnsafeProjection` already does. It only applies to `StructType`,
`ArrayType` and `MapType` -- the values the writers hold as references.
Reference-typed atomics (string, binary) are deliberately left alone. Those
fail elsewhere, in `UnsafeWriter.write`, and their declared nullability is
inherited from their input rather than manufactured, so it stays accurate in
practice.
### Why are the changes needed?
The same logically-null value is handled two different ways depending on
which projection the query happens to use. `InterpretedUnsafeProjection` writes
null; `GenerateUnsafeProjection` throws:
```
java.lang.NullPointerException: Cannot invoke
"org.apache.spark.sql.catalyst.util.ArrayData.numElements()"
because "<localN>" is null
at ...SpecificUnsafeProjection.writeFields_..$
```
`InterpretedUnsafeProjection.generateFieldWriter` is explicit that this is
intentional on its side -- "Always wrap the writer with a null safe version".
It takes `nullable` only to derive the child writer's nullability, never to
decide whether to null-check. Which projection you get should not decide
whether the query survives.
A declared non-null is not a guarantee about the data, so this is reachable
from plain SQL:
- `CreateNamedStruct.nullable` is hardcoded `false`, and
`ArrayTransform.dataType` derives `containsNull` from `function.nullable`. So
`transform(arr, x -> named_struct(...))` always reports `containsNull = false`
regardless of the data.
- Outer joins relax nullability by mapping `withNullability(true)` over the
join output (`Join.computeOutput`). That only touches the top-level
`AttributeReference` flag; nested `StructField.nullable` and
`ArrayType.containsNull` are left claiming non-null.
### Does this PR introduce _any_ user-facing change?
Yes -- a bug fix. A query that previously failed with a
`NullPointerException` from generated code now returns null for the nested
value, which is what the interpreted path already returned. No behavior change
for data that matches its declared nullability, since the guard only fires on a
null that would otherwise have thrown.
### How was this patch tested?
New tests in `GenerateUnsafeProjectionSuite`, each covering array, map and
struct:
- a null in a top-level field declared non-nullable is written as null;
- a null in a nested struct field declared non-nullable is written as null;
- a null element in an array declared `containsNull = false` is written as
null;
- the null bit mask is not carried over between rows;
- codegen and interpreted projections produce equal rows for an inaccurate
nullability declaration.
Run locally with `build/mvn -pl sql/catalyst
-DwildcardSuites=org.apache.spark.sql.catalyst.expressions.codegen.GenerateUnsafeProjectionSuite
test`: 7 tests succeeded, 0 failed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
