Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/23166#discussion_r237535187
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -708,16 +709,36 @@ private[spark] class PythonBroadcast(@transient var
path: String) extends Serial
override def handleConnection(sock: Socket): Unit = {
val env = SparkEnv.get
val in = sock.getInputStream()
- val dir = new File(Utils.getLocalDir(env.conf))
- val file = File.createTempFile("broadcast", "", dir)
- path = file.getAbsolutePath
- val out = env.serializerManager.wrapForEncryption(new
FileOutputStream(path))
+ val abspath = new File(path).getAbsolutePath
+ val out = env.serializerManager.wrapForEncryption(new
FileOutputStream(abspath))
--- End diff --
just want to make sure I understand this part -- this change isn't
necessary, right? even in the old version, `path` gets updated here, so
`setupDecryptionServer` would know where to read the data from.
that said, I do think your change makes more sense -- not sure why I didn't
just use the supplied path in the first place.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
