viirya opened a new pull request, #57607: URL: https://github.com/apache/spark/pull/57607
### What changes were proposed in this pull request? Register `org.apache.spark.unsafe.types.TimestampNanosVal` in `KryoSerializer`, next to `UTF8String` which reached the registration list the same way (SPARK-51790). ### Why are the changes needed? Since SPARK-57735, cached-batch statistics rows carry `TimestampNanosVal` min/max bounds for nanosecond-timestamp columns -- `TimestampNanosColumnStats` in the default cache serializer, and the Arrow cache serializer's vector-side statistics (SPARK-57268). The statistics `InternalRow` is a field of `DefaultCachedBatch` / `ArrowCachedBatch`, so it is serialized with the batch whenever the cache uses a serialized storage level. With `spark.serializer=KryoSerializer` and `spark.kryo.registrationRequired=true`, materializing such a cache fails at the first block write: ``` com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.unsafe.types.TimestampNanosVal ``` This is the same defect class as SPARK-51777 (`CachedBatch` classes) and SPARK-51790 (`UTF8String`): a new class reachable from the cached-batch object graph was introduced without a matching Kryo registration. Every other bound type in statistics rows (`UTF8String`, `Decimal`, primitives) is already registered; `TimestampNanosVal` was the only gap -- the remaining stats collectors for non-orderable types (Variant, CalendarInterval, Geometry) record only sizes, no value objects. Users with the default `registrationRequired=false` are unaffected (Kryo falls back to writing the class name). Nanosecond timestamp types are an unreleased 4.3.0 preview feature. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New test in `CacheTableInKryoSuite` (the suite from SPARK-51777/51790, which runs with `registrationRequired=true`): persists a `TIMESTAMP_NTZ(9)` column with `DISK_ONLY`. It fails without the registration with the exception above and passes with it. `CacheTableInKryoSuite` (4 tests) and `core`'s `*KryoSerializer*` suites (81 tests) pass. ### Was this patch authored or co-authored using generative AI tooling? Yes, this pull request and its description were written by Claude Code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
