[ 
https://issues.apache.org/jira/browse/SPARK-23159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Cutler updated SPARK-23159:
---------------------------------
    Description: 
Update PySpark's version of Cloudpickle to match version 0.4.3.  The reasons 
for doing this are:
 * Pick up bug fixes, improvements with newer version
 * Match a specific version as close as possible (Spark has additional changes 
that might be necessary) to make future upgrades easier

There are newer versions of Cloudpickle that can fix bugs with NamedTuple 
pickling (that Spark currently has workarounds for), but these include other 
changes that need some verification before bringing into Spark.  Upgrading 
first to 0.4.3 will help make this verification easier.

Discussion on the mailing list: 
[http://apache-spark-developers-list.1001551.n3.nabble.com/Thoughts-on-Cloudpickle-Update-td23188.html]

Upgrading to the recent release of v0.4.3 will include the following:
 * Fix pickling of named tuples 
[https://github.com/cloudpipe/cloudpickle/pull/113]
 * Built in type constructors for PyPy compatibility 
[here]([https://github.com/cloudpipe/cloudpickle/commit/d84980ccaafc7982a50d4e04064011f401f17d1b])
 * Fix memoryview support [https://github.com/cloudpipe/cloudpickle/pull/122]
 * Improved compatibility with other cloudpickle versions 
[https://github.com/cloudpipe/cloudpickle/pull/128]
 * Several cleanups [https://github.com/cloudpipe/cloudpickle/pull/121] and 
[here]([https://github.com/cloudpipe/cloudpickle/commit/c91aaf110441991307f5097f950764079d0f9652])
 * [MRG] Regression on pickling classes from the __main__ module 
[https://github.com/cloudpipe/cloudpickle/pull/149]
 * BUG: Handle instance methods of builtin types 
[https://github.com/cloudpipe/cloudpickle/pull/154]
 * Fix #129 : do not silence RuntimeError in dump() 
[https://github.com/cloudpipe/cloudpickle/pull/153]

  was:
Update PySpark's version of Cloudpickle to match version 0.4.2.  The reasons 
for doing this are:
 * Pick up bug fixes, improvements with newer version
 * Match a specific version as close as possible (Spark has additional changes 
that might be necessary) to make future upgrades easier

There are newer versions of Cloudpickle that can fix bugs with NamedTuple 
pickling (that Spark currently has workarounds for), but these include other 
changes that need some verification before bringing into Spark.  Upgrading 
first to 0.4.2 will help make this verification easier.

Discussion on the mailing list: 
http://apache-spark-developers-list.1001551.n3.nabble.com/Thoughts-on-Cloudpickle-Update-td23188.html


> Update Cloudpickle to match version 0.4.3
> -----------------------------------------
>
>                 Key: SPARK-23159
>                 URL: https://issues.apache.org/jira/browse/SPARK-23159
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>    Affects Versions: 2.3.0
>            Reporter: Bryan Cutler
>            Priority: Major
>
> Update PySpark's version of Cloudpickle to match version 0.4.3.  The reasons 
> for doing this are:
>  * Pick up bug fixes, improvements with newer version
>  * Match a specific version as close as possible (Spark has additional 
> changes that might be necessary) to make future upgrades easier
> There are newer versions of Cloudpickle that can fix bugs with NamedTuple 
> pickling (that Spark currently has workarounds for), but these include other 
> changes that need some verification before bringing into Spark.  Upgrading 
> first to 0.4.3 will help make this verification easier.
> Discussion on the mailing list: 
> [http://apache-spark-developers-list.1001551.n3.nabble.com/Thoughts-on-Cloudpickle-Update-td23188.html]
> Upgrading to the recent release of v0.4.3 will include the following:
>  * Fix pickling of named tuples 
> [https://github.com/cloudpipe/cloudpickle/pull/113]
>  * Built in type constructors for PyPy compatibility 
> [here]([https://github.com/cloudpipe/cloudpickle/commit/d84980ccaafc7982a50d4e04064011f401f17d1b])
>  * Fix memoryview support [https://github.com/cloudpipe/cloudpickle/pull/122]
>  * Improved compatibility with other cloudpickle versions 
> [https://github.com/cloudpipe/cloudpickle/pull/128]
>  * Several cleanups [https://github.com/cloudpipe/cloudpickle/pull/121] and 
> [here]([https://github.com/cloudpipe/cloudpickle/commit/c91aaf110441991307f5097f950764079d0f9652])
>  * [MRG] Regression on pickling classes from the __main__ module 
> [https://github.com/cloudpipe/cloudpickle/pull/149]
>  * BUG: Handle instance methods of builtin types 
> [https://github.com/cloudpipe/cloudpickle/pull/154]
>  * Fix #129 : do not silence RuntimeError in dump() 
> [https://github.com/cloudpipe/cloudpickle/pull/153]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to