Andrew Or created SPARK-12414:
---------------------------------
Summary: Remove closure serializer
Key: SPARK-12414
URL: https://issues.apache.org/jira/browse/SPARK-12414
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.0.0
Reporter: Andrew Or
There is a config `spark.closure.serializer` that accepts exactly one value:
the java serializer. This is because there are currently bugs in the Kryo
serializer that make it not a viable candidate. This was uncovered by an
unsuccessful attempt to make it work: SPARK-7708.
My high level point is that the Java serializer has worked well for at least 6
Spark versions now, and it is an incredibly complicated task to get other
serializers (not just Kryo) to work with Spark's closures. IMO the effort is
not worth it and we should just remove this documentation and all the code
associated with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
