Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/23166#discussion_r237738802
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -708,16 +709,36 @@ private[spark] class PythonBroadcast(@transient var
path: String) extends Serial
override def handleConnection(sock: Socket): Unit = {
val env = SparkEnv.get
val in = sock.getInputStream()
- val dir = new File(Utils.getLocalDir(env.conf))
- val file = File.createTempFile("broadcast", "", dir)
- path = file.getAbsolutePath
- val out = env.serializerManager.wrapForEncryption(new
FileOutputStream(path))
+ val abspath = new File(path).getAbsolutePath
+ val out = env.serializerManager.wrapForEncryption(new
FileOutputStream(abspath))
--- End diff --
yeah I see how it was wrong before. I'm saying, after you add
`setupDecryptionServer`, then that decryption server would still be reading
from the value of `path` which gets updated here, since its the same object in
the driver's JVM.
anyway, this isn't a big deal, I think its better with your change.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
