Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/20373#discussion_r163419648
--- Diff: python/pyspark/cloudpickle.py ---
@@ -237,28 +262,14 @@ def dump(self, obj):
if 'recursion' in e.args[0]:
msg = """Could not pickle object as excessively deep
recursion required."""
raise pickle.PicklingError(msg)
- except pickle.PickleError:
- raise
- except Exception as e:
- emsg = _exception_message(e)
- if "'i' format requires" in emsg:
- msg = "Object too large to serialize: %s" % emsg
- else:
- msg = "Could not serialize object: %s: %s" %
(e.__class__.__name__, emsg)
- print_exec(sys.stderr)
- raise pickle.PicklingError(msg)
-
def save_memoryview(self, obj):
- """Fallback to save_string"""
- Pickler.save_string(self, str(obj))
+ self.save(obj.tobytes())
+ dispatch[memoryview] = save_memoryview
--- End diff --
Some cleanups, fix memoryview support
https://github.com/cloudpipe/cloudpickle/commit/f8187e90aed7e1b96ffaae85cdf4b37108c75d3f
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
