[
https://issues.apache.org/jira/browse/SPARK-59097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59097:
-----------------------------------
Labels: pull-request-available (was: )
> UnsafeProjection codegen throws NullPointerException on a null in a nested
> non-nullable field, where the interpreted path writes null
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-59097
> URL: https://issues.apache.org/jira/browse/SPARK-59097
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.2, 4.2.0, 5.0.0
> Reporter: Joel Robin
> Priority: Major
> Labels: pull-request-available
>
> A struct, array or map value that is declared non-nullable but is actually
> null at runtime gets handled two different ways depending on which
> UnsafeProjection you end up with. The interpreted one writes null and carries
> on. The generated one dereferences the null and throws. Both are
> implementations of the same thing, so which one you get should not decide
> whether the query survives.
> It surfaces as a task failure like this:
> {code}
> java.lang.NullPointerException: Cannot invoke
> "org.apache.spark.sql.catalyst.util.ArrayData.numElements()" because
> "<local10>" is null
> at ...GeneratedClass$SpecificUnsafeProjection.writeFields_1_3$(Unknown
> Source)
> at ...GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
> at ...GeneratedClass$GeneratedIteratorForCodegenStage31.processNext(Unknown
> Source)
> at
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:154)
> {code}
> h3. Why codegen throws
> writeExpressionsToBuffer drops the null check whenever the declared
> nullability says the value cannot be null
> (GenerateUnsafeProjection.scala:156):
> {code:scala}
> if (!nullable || input.isNull == FalseLiteral) {
> // The value is statically known to be non-null, so skip the null check and
> the
> // (dead) setNull branch and just write the value.
> {code}
> Array elements get the same treatment when containsNull is false (:248-249).
> writeArrayToBuffer then does the dereference (:257):
> {code:java}
> final ArrayData tmpInput = <input>;
> if (tmpInput instanceof UnsafeArrayData) { ... }
> else { final int numElements = tmpInput.numElements(); // throws here
> {code}
> tmpInput is the <localN> named in the NPE message.
> InterpretedUnsafeProjection.generateFieldWriter does the opposite, and the
> comment is explicit about it (InterpretedUnsafeProjection.scala:251):
> {code:scala}
> // Always wrap the writer with a null safe version.
> {code}
> It does take nullable as a parameter, but only to work out the child writer's
> nullability. It
> never consults it when deciding whether to null-check. So the interpreted
> path tolerates an
> inaccurate declaration and codegen does not.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
