HyukjinKwon commented on code in PR #47434:
URL: https://github.com/apache/spark/pull/47434#discussion_r1689052244
##########
core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala:
##########
@@ -74,6 +77,16 @@ object PythonRunner {
// Launch Python process
val builder = new ProcessBuilder((Seq(pythonExec, formattedPythonFile) ++
otherArgs).asJava)
val env = builder.environment()
+ if (!Utils.isLocalRemote(sparkConf)) {
+ // For non-local remote, pass configurations to environment variables so
+ // Spark Connect client sets them. For local remotes, they will be set
+ // via Py4J.
+ // For PySpark specifically, we can't send other configurations through
properties.
+ // So, here we should send it together.
+ val confs = sparkConf.getAll.filter(p =>
Review Comment:
This specific code path is only used in Spark Connect client (so I only pass
`spark.sql.*`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
