This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new fe8ca560d749 [SPARK-58408][CORE] Register TimestampNanosVal in
KryoSerializer
fe8ca560d749 is described below
commit fe8ca560d749deae5cda26e32697a1623cbfc04e
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Tue Jul 28 16:17:08 2026 -0700
[SPARK-58408][CORE] Register TimestampNanosVal in KryoSerializer
### What changes were proposed in this pull request?
Register `org.apache.spark.unsafe.types.TimestampNanosVal` in
`KryoSerializer`, next to `UTF8String` which reached the registration list the
same way (SPARK-51790).
### Why are the changes needed?
Since SPARK-57735, cached-batch statistics rows carry `TimestampNanosVal`
min/max bounds for nanosecond-timestamp columns -- `TimestampNanosColumnStats`
in the default cache serializer, and the Arrow cache serializer's vector-side
statistics (SPARK-57268). The statistics `InternalRow` is a field of
`DefaultCachedBatch` / `ArrowCachedBatch`, so it is serialized with the batch
whenever the cache uses a serialized storage level.
With `spark.serializer=KryoSerializer` and
`spark.kryo.registrationRequired=true`, materializing such a cache fails at the
first block write:
```
com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException:
Class is not registered: org.apache.spark.unsafe.types.TimestampNanosVal
```
This is the same defect class as SPARK-51777 (`CachedBatch` classes) and
SPARK-51790 (`UTF8String`): a new class reachable from the cached-batch object
graph was introduced without a matching Kryo registration. Every other bound
type in statistics rows (`UTF8String`, `Decimal`, primitives) is already
registered; `TimestampNanosVal` was the only gap -- the remaining stats
collectors for non-orderable types (Variant, CalendarInterval, Geometry) record
only sizes, no value objects.
Users with the default `registrationRequired=false` are unaffected (Kryo
falls back to writing the class name). Nanosecond timestamp types are an
unreleased 4.3.0 preview feature.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New test in `CacheTableInKryoSuite` (the suite from SPARK-51777/51790,
which runs with `registrationRequired=true`): persists a `TIMESTAMP_NTZ(9)`
column with `DISK_ONLY`. It fails without the registration with the exception
above and passes with it. `CacheTableInKryoSuite` (4 tests) and `core`'s
`*KryoSerializer*` suites (81 tests) pass.
### Was this patch authored or co-authored using generative AI tooling?
Yes, this pull request and its description were written by Claude Code.
Closes #57607 from viirya/kryo-register-timestamp-nanos.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Liang-Chi Hsieh <[email protected]>
(cherry picked from commit 177ce56c0df55e5a21a44ff7c59105d075990a13)
Signed-off-by: Liang-Chi Hsieh <[email protected]>
---
.../main/scala/org/apache/spark/serializer/KryoSerializer.scala | 5 ++++-
.../test/scala/org/apache/spark/sql/CacheTableInKryoSuite.scala | 8 ++++++++
2 files changed, 12 insertions(+), 1 deletion(-)
diff --git
a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
index ea7af01121a1..65f0ffcbe18e 100644
--- a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
+++ b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
@@ -47,7 +47,7 @@ import org.apache.spark.internal.io.FileCommitProtocol._
import org.apache.spark.network.util.ByteUnit
import org.apache.spark.scheduler.{CompressedMapStatus,
HighlyCompressedMapStatus}
import org.apache.spark.storage._
-import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.unsafe.types.{TimestampNanosVal, UTF8String}
import org.apache.spark.util.{BoundedPriorityQueue, ByteBufferInputStream,
NextIterator, SerializableConfiguration, SerializableJobConf, Utils}
import org.apache.spark.util.collection.{BitSet, CompactBuffer}
import org.apache.spark.util.io.ChunkedByteBuffer
@@ -236,6 +236,9 @@ class KryoSerializer(conf: SparkConf)
kryo.register(classOf[ArrayBuffer[Any]])
kryo.register(classOf[Array[Array[Byte]]])
kryo.register(classOf[UTF8String])
+ // Nanosecond-timestamp min/max bounds inside cached-batch statistics rows
+ // (TimestampNanosColumnStats and the Arrow cache's vector-side stats).
+ kryo.register(classOf[TimestampNanosVal])
// We can't load those class directly in order to avoid unnecessary jar
dependencies.
// We load them safely, ignore it if the class not found.
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/CacheTableInKryoSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/CacheTableInKryoSuite.scala
index 72a2da160547..feee2bee298f 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/CacheTableInKryoSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/CacheTableInKryoSuite.scala
@@ -57,6 +57,14 @@ class CacheTableInKryoSuite extends SharedSparkSession {
}
}
+ test("SPARK-58408: TimestampNanosVal stats should be registered in
KryoSerializer") {
+ // Cached-batch statistics rows carry TimestampNanosVal min/max bounds for
+ // nanosecond-timestamp columns (TimestampNanosColumnStats), so persisting
such a cache
+ // through Kryo with registrationRequired must find the class registered.
+ val df = sql("SELECT CAST('2025-01-06 12:30:45.123456789' AS
TIMESTAMP_NTZ(9)) AS ts")
+ assert(df.persist(StorageLevel.DISK_ONLY).count() === 1)
+ }
+
test("SPARK-51813 DefaultCachedBatchKryoSerializer do not propagate nulls") {
val ks = new KryoSerializer(this.sparkConf)
val kryo = ks.newKryo()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]