This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new 6df196622e1d [SPARK-50195][CORE] Fix `StandaloneRestServer` to
propagate `spark.app.name` to `SparkSubmit` properly
6df196622e1d is described below
commit 6df196622e1d20fb9d57a0bae105ea2c4f7b51d8
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 31 20:29:49 2024 -0700
[SPARK-50195][CORE] Fix `StandaloneRestServer` to propagate
`spark.app.name` to `SparkSubmit` properly
This PR aims to fix `StandaloneRestServer` to propagate `spark.app.name` to
`SparkSubmit` properly.
This is a long-standing bug which PySpark job didn't get a proper
`spark.app.name` propagation unlike Scala/Java Spark jobs. Since PySpark jobs
are invoked indirectly via `SparkSubmit`, we need to hand over `spark.app.name`
via `-c` configuration.
This is a bug fix. The new behavior is the expected bahavior.
Pass the CIs with the newly added test case.
No.
Closes #48729 from dongjoon-hyun/SPARK-50195.
Lead-authored-by: Dongjoon Hyun <[email protected]>
Co-authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit ce899404f59d63b83e24ad24d65628676f85c773)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../org/apache/spark/deploy/rest/StandaloneRestServer.scala | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala
b/core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala
index c060ef9da8c1..12413698d283 100644
---
a/core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala
+++
b/core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala
@@ -21,7 +21,7 @@ import java.io.File
import javax.servlet.http.HttpServletResponse
import org.apache.spark.{SPARK_VERSION => sparkVersion, SparkConf}
-import org.apache.spark.deploy.{Command, DeployMessages, DriverDescription}
+import org.apache.spark.deploy.{Command, DeployMessages, DriverDescription,
SparkSubmit}
import org.apache.spark.deploy.ClientArguments._
import org.apache.spark.internal.config
import org.apache.spark.launcher.SparkLauncher
@@ -168,9 +168,16 @@ private[rest] class StandaloneSubmitRequestServlet(
val extraJavaOpts =
driverExtraJavaOptions.map(Utils.splitCommandString).getOrElse(Seq.empty)
val sparkJavaOpts = Utils.sparkJavaOpts(conf)
val javaOpts = sparkJavaOpts ++ defaultJavaOpts ++ extraJavaOpts
+ val sparkSubmitOpts = if (mainClass.equals(classOf[SparkSubmit].getName)) {
+ sparkProperties.get("spark.app.name")
+ .map { v => Seq("-c", s"spark.app.name=$v") }
+ .getOrElse(Seq.empty[String])
+ } else {
+ Seq.empty[String]
+ }
val command = new Command(
"org.apache.spark.deploy.worker.DriverWrapper",
- Seq("{{WORKER_URL}}", "{{USER_JAR}}", mainClass) ++ appArgs, // args to
the DriverWrapper
+ Seq("{{WORKER_URL}}", "{{USER_JAR}}", mainClass) ++ sparkSubmitOpts ++
appArgs,
environmentVariables, extraClassPath, extraLibraryPath, javaOpts)
val actualDriverMemory =
driverMemory.map(Utils.memoryStringToMb).getOrElse(DEFAULT_MEMORY)
val actualDriverCores = driverCores.map(_.toInt).getOrElse(DEFAULT_CORES)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
