This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 4d1261171b9a [SPARK-57451][CORE] Remove the check rejecting
`spark.authenticate.secret` with Master REST server
4d1261171b9a is described below
commit 4d1261171b9a5cf08cc1c1a33ff01d020913c434
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Jun 14 23:12:34 2026 -0700
[SPARK-57451][CORE] Remove the check rejecting `spark.authenticate.secret`
with Master REST server
### What changes were proposed in this pull request?
This PR removes the standalone `Master` check-code that rejects
`spark.authenticate.secret` when the Master REST server
(`spark.master.rest.enabled`) is enabled.
https://github.com/apache/spark/blob/088071d869dee0cb433c5e72ba2e7851e332b391/core/src/main/scala/org/apache/spark/deploy/master/Master.scala#L138-L144
For the record, the check was introduced at Apache Spark 2.4.0. And,
currently, it's outdated.
- https://github.com/apache/spark/pull/22071
### Why are the changes needed?
`spark.authenticate.secret` (the RPC authentication secret) and
`spark.master.rest.enabled` (the standalone submission REST server) are
independent concerns, but the removed check-code coupled them by failing Master
startup whenever both were set.
Since Apache Spark 4.1.0, `spark.master.rest.enabled` defaults to `true`,
this check-code forced any cluster using RPC authentication to disable the REST
server. This is wrong. We don't need to block like this because the REST server
is protected independently like the following.
- https://github.com/apache/spark/pull/47595 (Apache Spark 4.0.0)
- https://github.com/apache/spark/pull/47596
- https://github.com/apache/spark/pull/49894 (Apache Spark 4.1.0)
### Does this PR introduce _any_ user-facing change?
Previously, starting a standalone Master with `spark.authenticate.secret`
set and `spark.master.rest.enabled=true` (the default) failed with an
`IllegalArgumentException`. After this PR, the Master starts normally with both
configured securely.
Although this is a bug fix by enabling a previous-blocked code path. So,
technically there is no loss from the user perspective.
### How was this patch tested?
Added a unit test in `MasterSuite` that verifies a `Master` can be created
with both `spark.master.rest.enabled=true` and `spark.authenticate.secret` set.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
Closes #56511 from dongjoon-hyun/SPARK-57451.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit ff36aacb41c0ed225c606238593976d4472a6073)
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 0a952431f3ca49b5c4670b96bdb00918450ac55a)
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit a6480c17137bdf43c89a4d470123ae10af4a9442)
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 5b84d39a1289f90d32a67192b7b011aa7c2d4cea)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../src/main/scala/org/apache/spark/deploy/master/Master.scala | 8 --------
.../scala/org/apache/spark/deploy/master/MasterSuite.scala | 10 ++++++++++
2 files changed, 10 insertions(+), 8 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
b/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
index 617d073e8c8b..ff69b80c8715 100644
--- a/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
@@ -135,14 +135,6 @@ private[deploy] class Master(
private var restServer: Option[StandaloneRestServer] = None
private var restServerBoundPort: Option[Int] = None
- {
- val authKey = SecurityManager.SPARK_AUTH_SECRET_CONF
- require(conf.getOption(authKey).isEmpty || !restServerEnabled,
- s"The RestSubmissionServer does not support authentication via
${authKey}. Either turn " +
- "off the RestSubmissionServer with spark.master.rest.enabled=false, or
do not use " +
- "authentication.")
- }
-
override def onStart(): Unit = {
logInfo(log"Starting Spark master at ${MDC(LogKeys.MASTER_URL,
masterUrl)}")
logInfo(log"Running Spark version" +
diff --git
a/core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala
b/core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala
index e64bc724cfba..c138d1142a47 100644
--- a/core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala
+++ b/core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala
@@ -249,4 +249,14 @@ class MasterSuite extends MasterSuiteBase {
eventLogCodec = None)
assert(master.invokePrivate(_createApplication(desc, null)).id ===
"spark-45756")
}
+
+ test("SPARK-57451: Allows REST server and spark.authenticate.secret to be
enabled together") {
+ val conf = new SparkConf()
+ .set(MASTER_REST_SERVER_ENABLED, true)
+ .set(AUTH_SECRET, "secret")
+ // Creating a Master must not fail when both the REST server and an auth
secret are set.
+ noException should be thrownBy {
+ makeMaster(conf)
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
