Luka Erceg created SPARK-60330:
----------------------------------
Summary: Register classification tree nodes with Kryo
Key: SPARK-60330
URL: https://issues.apache.org/jira/browse/SPARK-60330
Project: Spark
Issue Type: Improvement
Components: ML, Spark Core
Affects Versions: 5.0.0
Reporter: Luka Erceg
Spark's built-in Kryo registrations omit the classes needed to serialize
classification tree node graphs. With spark.kryo.registrationRequired=true,
serializing an Array[org.apache.spark.ml.tree.Node] fails unless users register
these built-in classes manually. Without strict registration, Kryo writes
class-name metadata for the unregistered types.
Register Node arrays, InternalNode, LeafNode, ContinuousSplit,
CategoricalSplit, GiniCalculator, and EntropyCalculator using the existing
optional-class loading mechanism, preserving the separation between Spark Core
and MLlib.
The PR adds NodeKryoSuite to verify round-trip serialization with strict
registration enabled for Gini and entropy trees, checking predictions, leaf
indices, and class-count statistics across continuous and categorical splits.
PR: https://github.com/apache/spark/pull/59539
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
