[
https://issues.apache.org/jira/browse/SPARK-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15238171#comment-15238171
]
Dubkov Mikhail commented on SPARK-12414:
----------------------------------------
[~srowen], [~andrewor14],
As I see, you just hard coded spark.closure.serializer use JavaSerializer
implementation, that's why I have a question:
On our project we use custom spark.serializer that has own requirements for
objects serialization which can differs from JavaSerializer requirements. For
example, our serializer not requires "implements Serializable", unless
JavaSerializer does.
The question is will application fails once we upgrade to Spark 2.0.0 ?
Because now, if we don't define serializer as "spark.closure.serializer"
application fails with 'Caused by: java.io.NotSerializableException'
Could you please explain how it will work after changes in scope of this task?
Thanks!
> Remove closure serializer
> -------------------------
>
> Key: SPARK-12414
> URL: https://issues.apache.org/jira/browse/SPARK-12414
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 1.0.0
> Reporter: Andrew Or
> Assignee: Sean Owen
> Fix For: 2.0.0
>
>
> There is a config `spark.closure.serializer` that accepts exactly one value:
> the java serializer. This is because there are currently bugs in the Kryo
> serializer that make it not a viable candidate. This was uncovered by an
> unsuccessful attempt to make it work: SPARK-7708.
> My high level point is that the Java serializer has worked well for at least
> 6 Spark versions now, and it is an incredibly complicated task to get other
> serializers (not just Kryo) to work with Spark's closures. IMO the effort is
> not worth it and we should just remove this documentation and all the code
> associated with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
