Josh Rosen created SPARK-4898:
---------------------------------
Summary: Replace cloudpickle with Dill
Key: SPARK-4898
URL: https://issues.apache.org/jira/browse/SPARK-4898
Project: Spark
Issue Type: Bug
Components: PySpark
Reporter: Josh Rosen
We should consider replacing our modified version of {{cloudpickle}} with
[Dill|https://github.com/uqfoundation/dill], since it supports both Python 2
and 3 and might do a better job of handling certain corner-cases.
I attempted to do this a few months ago but ran into cases where Dill had
issues pickling objects defined in doctests, which broke our test suite:
https://github.com/uqfoundation/dill/issues/50. This issue may have been
resolved now; I haven't checked.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
