HyukjinKwon opened a new pull request, #58721:
URL: https://github.com/apache/spark/pull/58721
### What changes were proposed in this pull request?
Follow-up to SPARK-48701. `ModeCollationAware.collationAwareTransform`
computes a collation
key for each (nested) string value in `mode`/`pandas_mode` under a
non-binary collation. The
top-level buffer key was null-guarded, but the recursion into
struct/array/map fields was not,
so a NULL string nested inside a collated complex type reached
`CollationFactory.getCollationKey(null, ...)` and threw an NPE. This adds a
single null guard
at the top of `collationAwareTransform` so a null -- top-level or nested --
folds into its own
group, and removes the now-redundant top-level guard in the buffering
function.
### Why are the changes needed?
`pandas_mode` (backing pandas-on-Spark `Series.mode()`/`DataFrame.mode()`)
with `ignoreNA=False`,
and any `mode` over a collated complex column containing nested nulls, would
throw instead of
returning a result.
### Does this PR introduce _any_ user-facing change?
Yes, within the unreleased branch: a `mode`/`pandas_mode` query over a
collated complex type with
nested nulls no longer throws. There is no change compared to released
versions.
### How was this patch tested?
Added a regression test in `CollationAggregationSuite` ("pandas_mode handles
NULL nested in a
collated struct field") that fails (NPE) without the fix. To be verified by
the Apache Spark
GitHub Actions CI on this PR.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac
Co-authored-by: Isaac <[email protected]>
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
