This is an automated email from the ASF dual-hosted git repository.
vanzin pushed a commit to branch branch-2.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.3 by this push:
new 96c2c3b [SPARK-26998][CORE] Remove SSL configuration from executors
96c2c3b is described below
commit 96c2c3b4ddc2911a8c7f93bd0fbe6b42e9ac5a39
Author: Gabor Somogyi <gabor.g.somo...@gmail.com>
AuthorDate: Tue Apr 2 09:18:43 2019 -0700
[SPARK-26998][CORE] Remove SSL configuration from executors
## What changes were proposed in this pull request?
Different SSL passwords shown up as command line argument on executor side
in standalone mode:
* keyStorePassword
* keyPassword
* trustStorePassword
In this PR I've removed SSL configurations from executors.
## How was this patch tested?
Existing + additional unit tests.
Additionally tested with standalone mode and checked the command line
arguments:
```
[gaborsomogyi:~/spark] SPARK-26998(+4/-0,3)+ ± jps
94803 CoarseGrainedExecutorBackend
94818 Jps
90149 RemoteMavenServer
91925 Nailgun
94793 SparkSubmit
94680 Worker
94556 Master
398
[gaborsomogyi:~/spark] SPARK-26998(+4/-1,3)+ ± ps -ef | egrep
"94556|94680|94793|94803"
502 94556 1 0 2:02PM ttys007 0:07.39
/Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/bin/java -cp
/Users/gaborsomogyi/spark/conf/:/Users/gaborsomogyi/spark/assembly/target/scala-2.12/jars/*
-Xmx1g org.apache.spark.deploy.master.Master --host gsomogyi-MBP.local --port
7077 --webui-port 8080 --properties-file conf/spark-defaults.conf
502 94680 1 0 2:02PM ttys007 0:07.27
/Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/bin/java -cp
/Users/gaborsomogyi/spark/conf/:/Users/gaborsomogyi/spark/assembly/target/scala-2.12/jars/*
-Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081
--properties-file conf/spark-defaults.conf spark://gsomogyi-MBP.local:7077
502 94793 94782 0 2:02PM ttys007 0:35.52
/Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/bin/java -cp
/Users/gaborsomogyi/spark/conf/:/Users/gaborsomogyi/spark/assembly/target/scala-2.12/jars/*
-Dscala.usejavacp=true -Xmx1g org.apache.spark.deploy.SparkSubmit --master
spark://gsomogyi-MBP.local:7077 --class org.apache.spark.repl.Main --name Spark
shell spark-shell
502 94803 94680 0 2:03PM ttys007 0:05.20
/Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/bin/java -cp
/Users/gaborsomogyi/spark/conf/:/Users/gaborsomogyi/spark/assembly/target/scala-2.12/jars/*
-Xmx1024M -Dspark.ssl.ui.port=0 -Dspark.driver.port=60902
org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
spark://CoarseGrainedScheduler172.30.65.186:60902 --executor-id 0 --hostname
172.30.65.186 --cores 8 --app-id app-20190326140311-0000 --worker-u [...]
502 94910 57352 0 2:05PM ttys008 0:00.00 egrep
94556|94680|94793|94803
```
Closes #24170 from gaborgsomogyi/SPARK-26998.
Authored-by: Gabor Somogyi <gabor.g.somo...@gmail.com>
Signed-off-by: Marcelo Vanzin <van...@cloudera.com>
(cherry picked from commit 57aff93886ac7d02b88294672ce0d2495b0942b8)
Signed-off-by: Marcelo Vanzin <van...@cloudera.com>
---
core/src/main/scala/org/apache/spark/SparkConf.scala | 1 -
core/src/test/scala/org/apache/spark/SparkConfSuite.scala | 11 +++++++++++
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/SparkConf.scala
b/core/src/main/scala/org/apache/spark/SparkConf.scala
index cfceedf..5670f7d 100644
--- a/core/src/main/scala/org/apache/spark/SparkConf.scala
+++ b/core/src/main/scala/org/apache/spark/SparkConf.scala
@@ -703,7 +703,6 @@ private[spark] object SparkConf extends Logging {
*/
def isExecutorStartupConf(name: String): Boolean = {
(name.startsWith("spark.auth") && name !=
SecurityManager.SPARK_AUTH_SECRET_CONF) ||
- name.startsWith("spark.ssl") ||
name.startsWith("spark.rpc") ||
name.startsWith("spark.network") ||
isSparkPortConf(name)
diff --git a/core/src/test/scala/org/apache/spark/SparkConfSuite.scala
b/core/src/test/scala/org/apache/spark/SparkConfSuite.scala
index 1f68a0d..cc67b7f 100644
--- a/core/src/test/scala/org/apache/spark/SparkConfSuite.scala
+++ b/core/src/test/scala/org/apache/spark/SparkConfSuite.scala
@@ -339,6 +339,17 @@ class SparkConfSuite extends SparkFunSuite with
LocalSparkContext with ResetSyst
}
}
+ test("SPARK-26998: SSL configuration not needed on executors") {
+ val conf = new SparkConf(false)
+ conf.set("spark.ssl.enabled", "true")
+ conf.set("spark.ssl.keyPassword", "password")
+ conf.set("spark.ssl.keyStorePassword", "password")
+ conf.set("spark.ssl.trustStorePassword", "password")
+
+ val filtered = conf.getAll.filter { case (k, _) =>
SparkConf.isExecutorStartupConf(k) }
+ assert(filtered.isEmpty)
+ }
+
test("SPARK-27244 toDebugString redacts sensitive information") {
val conf = new SparkConf(loadDefaults = false)
.set("dummy.password", "dummy-password")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
