dongjoon-hyun opened a new pull request, #59771:
URL: https://github.com/apache/spark/pull/59771
### What changes were proposed in this pull request?
This PR aims to fix `SerializerBuildHelper.createSerializerForObject` to
check the input object for null even when it is not nullable if the struct is
empty, so that the serializer of an empty struct always refers to its input
object.
### Why are the changes needed?
An empty struct has no field serializers, so its serializer drops a
non-nullable input object. For example, `RowEncoder` serializes `s STRUCT<a:
STRUCT<> NOT NULL> NOT NULL` into `assertnotnull(named_struct(a,
assertnotnull(named_struct())))`, which does not refer to the input row. Since
`ExpressionEncoder` allows this only for a top-level empty struct
(SPARK-46539), such a nested empty struct fails with `AssertionError` at
`Dataset.collect` and `writeStream`.
```scala
spark.range(1).selectExpr("id AS i", "named_struct('a', struct()) AS
s").collect()
```
```
org.apache.spark.SparkException: [INTERNAL_ERROR] The "collect" action
failed. ...
Caused by: java.lang.AssertionError: assertion failed: each serializer
expression should contain at least one `BoundReference` or it should be an
empty struct. ...
```
### Does this PR introduce _any_ user-facing change?
Yes, compared to the released Spark versions.
- A nested empty struct can be collected and used in streaming queries.
- `RowEncoder` validates the value of an empty struct field like other
struct fields. A null value of a non-nullable empty struct field or a non-`Row`
value fails with `EXPRESSION_ENCODING_FAILED` instead of silently becoming an
empty struct.
### How was this patch tested?
Pass the CIs with the newly added test cases in `RowEncoderSuite` and
`DataFrameSuite`. All of them fail without this fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
