Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/23166#discussion_r237231750
--- Diff: python/pyspark/broadcast.py ---
@@ -118,8 +121,16 @@ def dump(self, value, f):
f.close()
def load_from_path(self, path):
- with open(path, 'rb', 1 << 20) as f:
- return self.load(f)
+ # we only need to decrypt it here if its on the driver since
executor
+ # decryption handled already
+ if self._sc is not None and self._sc._encryption_enabled:
--- End diff --
makes sense, will move it there
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
