Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9287#discussion_r44047478
--- Diff: core/src/main/scala/org/apache/spark/deploy/Client.scala ---
@@ -80,11 +81,30 @@ private class ClientEndpoint(
val extraJavaOptsConf = "spark.driver.extraJavaOptions"
val extraJavaOpts = sys.props.get(extraJavaOptsConf)
.map(Utils.splitCommandString).getOrElse(Seq.empty)
- val sparkJavaOpts = Utils.sparkJavaOpts(conf)
+
+ // we don't want to send any secrets as system properties because
they would be translated
+ // to Java options which are passed in command-line and thus
visible for other users
+ val propsToExclude = Set(Namespace.Blank, Namespace.Submission,
Namespace.Application)
+ .map(_.inConf(SPARK_AUTH_SECRET_CONF))
+ val sparkJavaOpts = Utils.sparkJavaOpts(conf, key =>
!propsToExclude.contains(key))
val javaOpts = sparkJavaOpts ++ extraJavaOpts
- val command = new Command(mainClass,
- Seq("{{WORKER_URL}}", "{{USER_JAR}}", driverArgs.mainClass) ++
driverArgs.driverOptions,
- sys.env, classPathEntries, libraryPathEntries, javaOpts)
+
+ // we want to transport secrets only from submission and app
namespaces; they will be
+ // transported as environment variables though
+ val envSecrets = Set(Namespace.Submission,
Namespace.Application).flatMap { ns =>
+ sys.env.get(ns.inEnv(ENV_AUTH_SECRET))
+ .orElse(conf.getOption(ns.inConf(SPARK_AUTH_SECRET_CONF)))
+ .map(v => ns.inEnv(ENV_AUTH_SECRET) -> v)
+ }
+
+ val command = new Command(
+ mainClass = mainClass,
+ arguments = Seq("{{WORKER_URL}}", "{{USER_JAR}}",
driverArgs.mainClass)
+ ++ driverArgs.driverOptions,
+ environment = sys.env ++ envSecrets - ENV_AUTH_SECRET,
--- End diff --
what's the precedence order here? can you add parentheses to make that
explicit (and make sure the code does what's intended)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
