This is an automated email from the ASF dual-hosted git repository.

viirya pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new fe8ca560d749 [SPARK-58408][CORE] Register TimestampNanosVal in 
KryoSerializer
fe8ca560d749 is described below

commit fe8ca560d749deae5cda26e32697a1623cbfc04e
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Tue Jul 28 16:17:08 2026 -0700

    [SPARK-58408][CORE] Register TimestampNanosVal in KryoSerializer
    
    ### What changes were proposed in this pull request?
    
    Register `org.apache.spark.unsafe.types.TimestampNanosVal` in 
`KryoSerializer`, next to `UTF8String` which reached the registration list the 
same way (SPARK-51790).
    
    ### Why are the changes needed?
    
    Since SPARK-57735, cached-batch statistics rows carry `TimestampNanosVal` 
min/max bounds for nanosecond-timestamp columns -- `TimestampNanosColumnStats` 
in the default cache serializer, and the Arrow cache serializer's vector-side 
statistics (SPARK-57268). The statistics `InternalRow` is a field of 
`DefaultCachedBatch` / `ArrowCachedBatch`, so it is serialized with the batch 
whenever the cache uses a serialized storage level.
    
    With `spark.serializer=KryoSerializer` and 
`spark.kryo.registrationRequired=true`, materializing such a cache fails at the 
first block write:
    
    ```
    com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException:
    Class is not registered: org.apache.spark.unsafe.types.TimestampNanosVal
    ```
    
    This is the same defect class as SPARK-51777 (`CachedBatch` classes) and 
SPARK-51790 (`UTF8String`): a new class reachable from the cached-batch object 
graph was introduced without a matching Kryo registration. Every other bound 
type in statistics rows (`UTF8String`, `Decimal`, primitives) is already 
registered; `TimestampNanosVal` was the only gap -- the remaining stats 
collectors for non-orderable types (Variant, CalendarInterval, Geometry) record 
only sizes, no value objects.
    
    Users with the default `registrationRequired=false` are unaffected (Kryo 
falls back to writing the class name). Nanosecond timestamp types are an 
unreleased 4.3.0 preview feature.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    New test in `CacheTableInKryoSuite` (the suite from SPARK-51777/51790, 
which runs with `registrationRequired=true`): persists a `TIMESTAMP_NTZ(9)` 
column with `DISK_ONLY`. It fails without the registration with the exception 
above and passes with it. `CacheTableInKryoSuite` (4 tests) and `core`'s 
`*KryoSerializer*` suites (81 tests) pass.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Yes, this pull request and its description were written by Claude Code.
    
    Closes #57607 from viirya/kryo-register-timestamp-nanos.
    
    Authored-by: Liang-Chi Hsieh <[email protected]>
    Signed-off-by: Liang-Chi Hsieh <[email protected]>
    (cherry picked from commit 177ce56c0df55e5a21a44ff7c59105d075990a13)
    Signed-off-by: Liang-Chi Hsieh <[email protected]>
---
 .../main/scala/org/apache/spark/serializer/KryoSerializer.scala   | 5 ++++-
 .../test/scala/org/apache/spark/sql/CacheTableInKryoSuite.scala   | 8 ++++++++
 2 files changed, 12 insertions(+), 1 deletion(-)

diff --git 
a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala 
b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
index ea7af01121a1..65f0ffcbe18e 100644
--- a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
+++ b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
@@ -47,7 +47,7 @@ import org.apache.spark.internal.io.FileCommitProtocol._
 import org.apache.spark.network.util.ByteUnit
 import org.apache.spark.scheduler.{CompressedMapStatus, 
HighlyCompressedMapStatus}
 import org.apache.spark.storage._
-import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.unsafe.types.{TimestampNanosVal, UTF8String}
 import org.apache.spark.util.{BoundedPriorityQueue, ByteBufferInputStream, 
NextIterator, SerializableConfiguration, SerializableJobConf, Utils}
 import org.apache.spark.util.collection.{BitSet, CompactBuffer}
 import org.apache.spark.util.io.ChunkedByteBuffer
@@ -236,6 +236,9 @@ class KryoSerializer(conf: SparkConf)
     kryo.register(classOf[ArrayBuffer[Any]])
     kryo.register(classOf[Array[Array[Byte]]])
     kryo.register(classOf[UTF8String])
+    // Nanosecond-timestamp min/max bounds inside cached-batch statistics rows
+    // (TimestampNanosColumnStats and the Arrow cache's vector-side stats).
+    kryo.register(classOf[TimestampNanosVal])
 
     // We can't load those class directly in order to avoid unnecessary jar 
dependencies.
     // We load them safely, ignore it if the class not found.
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/CacheTableInKryoSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/CacheTableInKryoSuite.scala
index 72a2da160547..feee2bee298f 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/CacheTableInKryoSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/CacheTableInKryoSuite.scala
@@ -57,6 +57,14 @@ class CacheTableInKryoSuite extends SharedSparkSession {
     }
   }
 
+  test("SPARK-58408: TimestampNanosVal stats should be registered in 
KryoSerializer") {
+    // Cached-batch statistics rows carry TimestampNanosVal min/max bounds for
+    // nanosecond-timestamp columns (TimestampNanosColumnStats), so persisting 
such a cache
+    // through Kryo with registrationRequired must find the class registered.
+    val df = sql("SELECT CAST('2025-01-06 12:30:45.123456789' AS 
TIMESTAMP_NTZ(9)) AS ts")
+    assert(df.persist(StorageLevel.DISK_ONLY).count() === 1)
+  }
+
   test("SPARK-51813 DefaultCachedBatchKryoSerializer do not propagate nulls") {
     val ks = new KryoSerializer(this.sparkConf)
     val kryo = ks.newKryo()


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to