Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20519#discussion_r167018107
  
    --- Diff: 
core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala ---
    @@ -180,18 +181,52 @@ private[spark] class PythonWorkerFactory(pythonExec: 
String, envVars: Map[String
             return
           }
     
    +      var serverSocket: ServerSocket = null
           try {
    +        // get a server socket so that the launched daemon can tell us its 
server port
    +        serverSocket = new ServerSocket(0, 0, 
InetAddress.getByAddress(Array(127, 0, 0, 1)))
    +        val serverPort = serverSocket.getLocalPort
    +
    +        // generate an 'auth token' for the daemon to pass back to us. 
This will
    +        // allow us to confirm that the we are truly communicating with 
the newly
    +        // launched daemon
    +        val expectedAuthToken = (new Random()).nextInt()
    +
             // Create and start the daemon
    -        val pb = new ProcessBuilder(Arrays.asList(pythonExec, "-m", 
daemonModule))
    +        val pb = new ProcessBuilder(Arrays.asList(pythonExec, "-m", 
daemonModule,
    +          serverPort.toString))
             val workerEnv = pb.environment()
             workerEnv.putAll(envVars.asJava)
             workerEnv.put("PYTHONPATH", pythonPath)
             // This is equivalent to setting the -u flag; we use it because 
ipython doesn't support -u:
             workerEnv.put("PYTHONUNBUFFERED", "YES")
    +        workerEnv.put("PYSPARK_DAEMON_TOKEN", expectedAuthToken.toString)
             daemon = pb.start()
     
    +        // get the local port of the daemon's server socket,
    +        // but don't wait forever for the daemon to connect
    +        serverSocket.setSoTimeout(10000)
    +        var socketToDaemon: Socket = null
    --- End diff --
    
    This one can be a `val ... = serverSocket.accept()`, and you don't need the 
null check in the finally for it.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to