HyukjinKwon commented on a change in pull request #20691: [SPARK-18161]
[Python] Update cloudpickle to v0.6.1
URL: https://github.com/apache/spark/pull/20691#discussion_r249231927
##########
File path: python/pyspark/broadcast.py
##########
@@ -110,7 +110,7 @@ def __init__(self, sc=None, value=None,
pickle_registry=None, path=None,
def dump(self, value, f):
try:
- pickle.dump(value, f, 2)
+ pickle.dump(value, f, pickle.HIGHEST_PROTOCOL)
Review comment:
Ah, yea. this PR was previously setting the protocol to highest one to
support 4gb+ pickle alone in the regular pickle (not including cloudpickle).
So I suggested to target upgrade Cloudpickle because upper Cloudpickle has
that change to use highest protocol even though upgrading Cloudpickle is
slightly orthogonal.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
