LuciferYang opened a new pull request, #56272:
URL: https://github.com/apache/spark/pull/56272
### What changes were proposed in this pull request?
`AtLeastNNonNulls.doGenCode` emitted, for every child, `if (!eval.isNull [&&
!Double.isNaN(value)]) nonnull += 1;`. For a non-nullable child the
`!eval.isNull` term is statically `false`-negated (always true), so it is
dropped: a non-float child becomes an unconditional `nonnull += 1;`, and a
float/double child keeps only the `!Double.isNaN(value)` check. Nullable
children are unchanged.
The per-child code is restructured around an `Option[String]`
count-condition (`None` = "always counts"), which also factors the shared `if
(nonnull < n) { ... } else { continue; }` wrapper out of the two former
`dataType` branches.
This follows the SPARK-56908 "non-nullable children" dead-branch pattern of
`GenArrayData.genCodeToCreateArrayData` / `CreateNamedStruct`.
### Why are the changes needed?
It removes a dead comparison/branch per non-nullable child, shrinking the
generated method. The win scales with the number of columns (e.g.
`na.drop(threshold)` over a wide non-null schema), which helps with the JVM
64KB method / constant-pool limits, Janino compile time, and JIT work — this
expression even has a "should not throw 64KiB exception" test. Part of
SPARK-56908.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing `NullExpressionsSuite` tests: "AtLeastNNonNulls" (mixes non-null
literals, including NaN floats, with null literals) and "AtLeastNNonNulls
should not throw 64KiB exception" (4000 non-null literals, exercising the split
path), via `checkEvaluation` which runs the interpreted and codegen paths.
Behavior is unchanged (the dropped term was statically true), and NaN exclusion
for non-nullable floats is still covered, so no new test is added.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
