This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e242bcfc1183 [SPARK-53020][DEPLOY] JPMS args should also apply to
non-SparkSubmit process
e242bcfc1183 is described below
commit e242bcfc1183bfdad9c1024b6f1fffec32183baa
Author: Cheng Pan <cheng...@apache.org>
AuthorDate: Wed Jul 30 10:21:10 2025 -0700
[SPARK-53020][DEPLOY] JPMS args should also apply to non-SparkSubmit process
### What changes were proposed in this pull request?
Currently, JPMS args are only applied to `SparkSubmit`-based processes, for
non-`SparkSubmit` processes, e.g. Spark History Server, BeeLine, JPMS args are
not applied automatically.
For example, when trying run Spark History Server with Java 24, I see [JEP
472](https://openjdk.org/jeps/472) complains
```
$ spark-4.1.0-SNAPSHOT-bin-hadoop3 SPARK_NO_DAEMONIZE=1
sbin/start-history-server.sh
starting org.apache.spark.deploy.history.HistoryServer, logging to
/Users/chengpan/app/spark-4.1.0-SNAPSHOT-bin-hadoop3/logs/spark-chengpan-org.apache.spark.deploy.history.HistoryServer-1-H27212-MAC-01.local.out
Spark Command: /Users/chengpan/.sdkman/candidates/java/24.0.2-tem/bin/java
-cp
/Users/chengpan/app/spark-4.1.0-SNAPSHOT-bin-hadoop3/conf/:/Users/chengpan/app/spark-4.1.0-SNAPSHOT-bin-hadoop3/jars/slf4j-api-2.0.17.jar:/Users/chengpan/app/spark-4.1.0-SNAPSHOT-bin-hadoop3/jars/*
-Xmx1g org.apache.spark.deploy.history.HistoryServer
========================================
Using Spark's default log4j profile:
org/apache/spark/log4j2-defaults.properties
25/07/30 21:28:39 INFO HistoryServer: Started daemon with process name:
54576H27212-MAC-01.local
25/07/30 21:28:39 INFO SignalUtils: Registering signal handler for TERM
25/07/30 21:28:39 INFO SignalUtils: Registering signal handler for HUP
25/07/30 21:28:39 INFO SignalUtils: Registering signal handler for INT
WARNING: A restricted method in java.lang.System has been called
WARNING: java.lang.System::loadLibrary has been called by
org.apache.hadoop.util.NativeCodeLoader in an unnamed module
(file:/Users/chengpan/app/spark-4.1.0-SNAPSHOT-bin-hadoop3/jars/hadoop-client-api-3.4.1.jar)
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for
callers in this module
WARNING: Restricted methods will be blocked in a future release unless
native access is enabled
25/07/30 21:28:39 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile:
org/apache/spark/log4j2-defaults.properties
...
```
### Why are the changes needed?
I think the JPMS args defined at `JavaModuleOptions.DEFAULT_MODULE_OPTIONS`
are intended to be applied to all Spark process, so I would classify this as a
bug.
### Does this PR introduce _any_ user-facing change?
Seems no, as I haven't seen Spark deamon process failures caused by missing
JPMS args in our real use cases.
### How was this patch tested?
UT is added. Also verified Spark History Server start command:
```
spark-4.1.0-SNAPSHOT-bin-SPARK-53020 SPARK_NO_DAEMONIZE=1
sbin/start-history-server.sh
starting org.apache.spark.deploy.history.HistoryServer, logging to
/Users/chengpan/app/spark-4.1.0-SNAPSHOT-bin-SPARK-53020/logs/spark-chengpan-org.apache.spark.deploy.history.HistoryServer-1-H27212-MAC-01.local.out
Spark Command: /Users/chengpan/.sdkman/candidates/java/24.0.2-tem/bin/java
-cp
/Users/chengpan/app/spark-4.1.0-SNAPSHOT-bin-SPARK-53020/conf/:/Users/chengpan/app/spark-4.1.0-SNAPSHOT-bin-SPARK-53020/jars/slf4j-api-2.0.17.jar:/Users/chengpan/app/spark-4.1.0-SNAPSHOT-bin-SPARK-53020/jars/*
-Xmx1g -XX:+IgnoreUnrecognizedVMOptions --add-modules=jdk.incubator.vector
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.lan [...]
========================================
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: package sun.security.action not in java.base
Using Spark's default log4j profile:
org/apache/spark/log4j2-defaults.properties
25/07/30 22:06:38 INFO HistoryServer: Started daemon with process name:
60622H27212-MAC-01.local
25/07/30 22:06:38 INFO SignalUtils: Registering signal handler for TERM
25/07/30 22:06:38 INFO SignalUtils: Registering signal handler for HUP
25/07/30 22:06:38 INFO SignalUtils: Registering signal handler for INT
25/07/30 22:06:38 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile:
org/apache/spark/log4j2-defaults.properties
...
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #51725 from pan3793/SPARK-53020.
Authored-by: Cheng Pan <cheng...@apache.org>
Signed-off-by: Dongjoon Hyun <dongj...@apache.org>
---
.../main/java/org/apache/spark/launcher/SparkClassCommandBuilder.java | 1 +
.../java/org/apache/spark/launcher/SparkClassCommandBuilderSuite.java | 1 +
2 files changed, 2 insertions(+)
diff --git
a/launcher/src/main/java/org/apache/spark/launcher/SparkClassCommandBuilder.java
b/launcher/src/main/java/org/apache/spark/launcher/SparkClassCommandBuilder.java
index 8d95bc06d7a7..d7d10d486e3f 100644
---
a/launcher/src/main/java/org/apache/spark/launcher/SparkClassCommandBuilder.java
+++
b/launcher/src/main/java/org/apache/spark/launcher/SparkClassCommandBuilder.java
@@ -99,6 +99,7 @@ class SparkClassCommandBuilder extends AbstractCommandBuilder
{
String mem = firstNonEmpty(memKey != null ? System.getenv(memKey) : null,
DEFAULT_MEM);
cmd.add("-Xmx" + mem);
+ addOptionString(cmd, JavaModuleOptions.defaultModuleOptions());
cmd.add(className);
cmd.addAll(classArgs);
return cmd;
diff --git
a/launcher/src/test/java/org/apache/spark/launcher/SparkClassCommandBuilderSuite.java
b/launcher/src/test/java/org/apache/spark/launcher/SparkClassCommandBuilderSuite.java
index 76a5e23d4693..23ebbfc098ac 100644
---
a/launcher/src/test/java/org/apache/spark/launcher/SparkClassCommandBuilderSuite.java
+++
b/launcher/src/test/java/org/apache/spark/launcher/SparkClassCommandBuilderSuite.java
@@ -33,6 +33,7 @@ public class SparkClassCommandBuilderSuite extends BaseSuite {
SparkClassCommandBuilder builder =
new SparkClassCommandBuilder("org.apache.hive.beeline.BeeLine", args);
List<String> strings = builder.buildCommand(new HashMap<>());
+
assertTrue(strings.containsAll(Arrays.asList(JavaModuleOptions.defaultModuleOptionArray())));
assertTrue(strings.contains("-DmyKey=yourValue"));
assertTrue(strings.contains("myBeelineArg"));
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
