Marek Simunek created BEAM-6021:
-----------------------------------
Summary: Use Kryo spark.serializer instead of JavaSerializer
Key: BEAM-6021
URL: https://issues.apache.org/jira/browse/BEAM-6021
Project: Beam
Issue Type: Improvement
Components: runner-spark
Affects Versions: 2.8.0
Reporter: Marek Simunek
Assignee: Marek Simunek
Fix For: 2.9.0
By default is set spark.serializer=org.apache.spark.serializer.JavaSerializer.
Because all objects from user are using Beam coders it will affect only
internal objects for spark translation.
So why not use more optimal {{org.apache.spark.serializer.KryoSerializer}} and
force spark runner contributors to register classes in
{{BeamSparkRunnerRegistrator}} by setting
{{spark.kryo.registrationRequired=true)}}.
More information about benefits of [kryo
serialization|https://spark.apache.org/docs/latest/tuning.html#data-serialization]
over java serializer.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
