HyukjinKwon commented on code in PR #47434:
URL: https://github.com/apache/spark/pull/47434#discussion_r1733782694
##########
core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala:
##########
@@ -74,12 +80,22 @@ object PythonRunner {
// Launch Python process
val builder = new ProcessBuilder((Seq(pythonExec, formattedPythonFile) ++
otherArgs).asJava)
val env = builder.environment()
+ if (sparkConf.getOption("spark.remote").nonEmpty) {
+ // For non-local remote, pass configurations to environment variables so
+ // Spark Connect client sets them. For local remotes, they will be set
+ // via Py4J.
Review Comment:
For `bin/pyspark --remote local`, it still launches Spark Connect server via
Py4J. This should be fine because the Py4J jars and other dependencies should
be present to support `bin/pyspark --remote local` run.
Pure Python packaged ones would not support `bin/pyspark --remote local` and
won't need Py4J.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
