This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 5b84d39a1289 [SPARK-57451][CORE] Remove the check rejecting
`spark.authenticate.secret` with Master REST server
5b84d39a1289 is described below
commit 5b84d39a1289f90d32a67192b7b011aa7c2d4cea
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Jun 14 23:12:34 2026 -0700
[SPARK-57451][CORE] Remove the check rejecting `spark.authenticate.secret`
with Master REST server
### What changes were proposed in this pull request?
This PR removes the standalone `Master` check-code that rejects
`spark.authenticate.secret` when the Master REST server
(`spark.master.rest.enabled`) is enabled.
https://github.com/apache/spark/blob/088071d869dee0cb433c5e72ba2e7851e332b391/core/src/main/scala/org/apache/spark/deploy/master/Master.scala#L138-L144
For the record, the check was introduced at Apache Spark 2.4.0. And,
currently, it's outdated.
- https://github.com/apache/spark/pull/22071
### Why are the changes needed?
`spark.authenticate.secret` (the RPC authentication secret) and
`spark.master.rest.enabled` (the standalone submission REST server) are
independent concerns, but the removed check-code coupled them by failing Master
startup whenever both were set.
Since Apache Spark 4.1.0, `spark.master.rest.enabled` defaults to `true`,
this check-code forced any cluster using RPC authentication to disable the REST
server. This is wrong. We don't need to block like this because the REST server
is protected independently like the following.
- https://github.com/apache/spark/pull/47595 (Apache Spark 4.0.0)
- https://github.com/apache/spark/pull/47596
- https://github.com/apache/spark/pull/49894 (Apache Spark 4.1.0)
### Does this PR introduce _any_ user-facing change?
Previously, starting a standalone Master with `spark.authenticate.secret`
set and `spark.master.rest.enabled=true` (the default) failed with an
`IllegalArgumentException`. After this PR, the Master starts normally with both
configured securely.
Although this is a bug fix by enabling a previous-blocked code path. So,
technically there is no loss from the user perspective.
### How was this patch tested?
Added a unit test in `MasterSuite` that verifies a `Master` can be created
with both `spark.master.rest.enabled=true` and `spark.authenticate.secret` set.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
Closes #56511 from dongjoon-hyun/SPARK-57451.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit ff36aacb41c0ed225c606238593976d4472a6073)
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 0a952431f3ca49b5c4670b96bdb00918450ac55a)
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit a6480c17137bdf43c89a4d470123ae10af4a9442)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../src/main/scala/org/apache/spark/deploy/master/Master.scala | 8 --------
.../scala/org/apache/spark/deploy/master/MasterSuite.scala | 10 ++++++++++
2 files changed, 10 insertions(+), 8 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
b/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
index 95778771d423..e95c4bd8c622 100644
--- a/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
@@ -135,14 +135,6 @@ private[deploy] class Master(
private var restServer: Option[StandaloneRestServer] = None
private var restServerBoundPort: Option[Int] = None
- {
- val authKey = SecurityManager.SPARK_AUTH_SECRET_CONF
- require(conf.getOption(authKey).isEmpty || !restServerEnabled,
- s"The RestSubmissionServer does not support authentication via
${authKey}. Either turn " +
- "off the RestSubmissionServer with spark.master.rest.enabled=false, or
do not use " +
- "authentication.")
- }
-
override def onStart(): Unit = {
logInfo(log"Starting Spark master at ${MDC(LogKeys.MASTER_URL,
masterUrl)}")
logInfo(log"Running Spark version" +
diff --git
a/core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala
b/core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala
index e64bc724cfba..c138d1142a47 100644
--- a/core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala
+++ b/core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala
@@ -249,4 +249,14 @@ class MasterSuite extends MasterSuiteBase {
eventLogCodec = None)
assert(master.invokePrivate(_createApplication(desc, null)).id ===
"spark-45756")
}
+
+ test("SPARK-57451: Allows REST server and spark.authenticate.secret to be
enabled together") {
+ val conf = new SparkConf()
+ .set(MASTER_REST_SERVER_ENABLED, true)
+ .set(AUTH_SECRET, "secret")
+ // Creating a Master must not fail when both the REST server and an auth
secret are set.
+ noException should be thrownBy {
+ makeMaster(conf)
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
