Repository: spark Updated Branches: refs/heads/master 354f4582b -> c1840a862
[SPARK-7736] [CORE] Fix a race introduced in PythonRunner. The fix for SPARK-7736 introduced a race where a port value of "-1" could be passed down to the pyspark process, causing it to fail to connect back to the JVM. This change adds code to fix that race. Author: Marcelo Vanzin <van...@cloudera.com> Closes #8258 from vanzin/SPARK-7736. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c1840a86 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c1840a86 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c1840a86 Branch: refs/heads/master Commit: c1840a862eb548bc4306e53ee7e9f26986b31832 Parents: 354f458 Author: Marcelo Vanzin <van...@cloudera.com> Authored: Tue Aug 18 11:36:36 2015 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Tue Aug 18 11:36:36 2015 -0700 ---------------------------------------------------------------------- .../main/scala/org/apache/spark/deploy/PythonRunner.scala | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/c1840a86/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala b/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala index 4277ac2..23d01e9 100644 --- a/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala +++ b/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala @@ -52,10 +52,16 @@ object PythonRunner { gatewayServer.start() } }) - thread.setName("py4j-gateway") + thread.setName("py4j-gateway-init") thread.setDaemon(true) thread.start() + // Wait until the gateway server has started, so that we know which port is it bound to. + // `gatewayServer.start()` will start a new thread and run the server code there, after + // initializing the socket, so the thread started above will end as soon as the server is + // ready to serve connections. + thread.join() + // Build up a PYTHONPATH that includes the Spark assembly JAR (where this class is), the // python directories in SPARK_HOME (if set), and any files in the pyFiles argument val pathElements = new ArrayBuffer[String] --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
