[
https://issues.apache.org/jira/browse/BEAM-6021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chamikara Jayalath updated BEAM-6021:
-------------------------------------
Fix Version/s: (was: 2.9.0)
2.10.0
> Use Kryo spark.serializer instead of JavaSerializer
> ---------------------------------------------------
>
> Key: BEAM-6021
> URL: https://issues.apache.org/jira/browse/BEAM-6021
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
> Affects Versions: 2.8.0
> Reporter: Marek Simunek
> Assignee: Marek Simunek
> Priority: Major
> Fix For: 2.10.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> By default is set spark.serializer=org.apache.spark.serializer.JavaSerializer.
> Because all objects from user are using Beam coders it will affect only
> internal objects for spark translation.
> So why not use more optimal {{org.apache.spark.serializer.KryoSerializer}}
> and force spark runner contributors to register classes in
> {{BeamSparkRunnerRegistrator}} by setting
> {{spark.kryo.registrationRequired=true)}}.
> More information about benefits of [kryo
> serialization|https://spark.apache.org/docs/latest/tuning.html#data-serialization]
> over java serializer.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
