hvanhovell commented on code in PR #47434:
URL: https://github.com/apache/spark/pull/47434#discussion_r1733489649
##########
core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala:
##########
@@ -74,12 +80,22 @@ object PythonRunner {
// Launch Python process
val builder = new ProcessBuilder((Seq(pythonExec, formattedPythonFile) ++
otherArgs).asJava)
val env = builder.environment()
+ if (sparkConf.getOption("spark.remote").nonEmpty) {
+ // For non-local remote, pass configurations to environment variables so
+ // Spark Connect client sets them. For local remotes, they will be set
+ // via Py4J.
Review Comment:
Why still take a dependency on Py4j in connect mode?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
