Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/23166#discussion_r237228157
--- Diff: python/pyspark/broadcast.py ---
@@ -118,8 +121,16 @@ def dump(self, value, f):
f.close()
def load_from_path(self, path):
- with open(path, 'rb', 1 << 20) as f:
- return self.load(f)
+ # we only need to decrypt it here if its on the driver since
executor
+ # decryption handled already
+ if self._sc is not None and self._sc._encryption_enabled:
--- End diff --
can you move the entire conditional check into `value()` instead, and keep
`load_from_path` like it was before? `value()` is already checking for
different scenarios, and keeps the meaning of this function more
straightforward.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
