Fangchen Li created SPARK-55152:
-----------------------------------
Summary: Evaluate serialization strategy for Scala 3 support
Key: SPARK-55152
URL: https://issues.apache.org/jira/browse/SPARK-55152
Project: Spark
Issue Type: Sub-task
Components: Spark Core
Affects Versions: 4.2.0
Reporter: Fangchen Li
The current Kryo serialization depends on Twitter's chill library, which has no
Scala 3 support and is unmaintained (last release: June 2021).
We have a few options here:
# Fork chill and take on its maintenance. This requires the least change in
Spark, but adds extra responsibility and maintenance burden.
# Switch to
[scala-kryo-serialization|https://github.com/altoo-ag/scala-kryo-serialization],
which supports Scala 3 but requires Kryo 5.
# Consider adding [Apache Fory|https://github.com/apache/fory] as an
alternative serializer. It's much better maintained and much faster than Kryo.
But we probably still need to keep Kryo for backward compatibility.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
