Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/23166#discussion_r237875851
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -708,16 +709,36 @@ private[spark] class PythonBroadcast(@transient var
path: String) extends Serial
override def handleConnection(sock: Socket): Unit = {
val env = SparkEnv.get
val in = sock.getInputStream()
- val dir = new File(Utils.getLocalDir(env.conf))
- val file = File.createTempFile("broadcast", "", dir)
- path = file.getAbsolutePath
- val out = env.serializerManager.wrapForEncryption(new
FileOutputStream(path))
+ val abspath = new File(path).getAbsolutePath
+ val out = env.serializerManager.wrapForEncryption(new
FileOutputStream(abspath))
--- End diff --
ok I think we agree its good this way, (just to verify though I won't
commit until you +1 it), but yes you are correct, now that we are using the
decryption server which reads from the path in PythonBroadcast the path change
isn't strictly necessary, but the value of self._path in broadcast.py doesn't
match the path in PythonBroadcast so I think its better to have those match.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
