Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/20373#discussion_r163658600
--- Diff: python/pyspark/cloudpickle.py ---
@@ -237,28 +262,14 @@ def dump(self, obj):
if 'recursion' in e.args[0]:
msg = """Could not pickle object as excessively deep
recursion required."""
raise pickle.PicklingError(msg)
- except pickle.PickleError:
- raise
- except Exception as e:
- emsg = _exception_message(e)
- if "'i' format requires" in emsg:
- msg = "Object too large to serialize: %s" % emsg
- else:
- msg = "Could not serialize object: %s: %s" %
(e.__class__.__name__, emsg)
- print_exec(sys.stderr)
- raise pickle.PicklingError(msg)
-
def save_memoryview(self, obj):
- """Fallback to save_string"""
- Pickler.save_string(self, str(obj))
+ self.save(obj.tobytes())
+ dispatch[memoryview] = save_memoryview
--- End diff --
so you mean change to this
```
def save_memoryview(self, obj):
self.save(obj.tobytes())
dispatch[memoryview] = save_memoryview
```
That format is done in a few places and I saw a couple other formatting
issues.. so yeah I can submit a PR to do those
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
