LuciferYang commented on code in PR #58010:
URL: https://github.com/apache/spark/pull/58010#discussion_r4303948989
##########
docs/security.md:
##########
@@ -183,6 +183,28 @@ Unlike the other SSL settings for the UI, the RPC SSL is
*not* automatically ena
`spark.ssl.enabled` is set. It must be explicitly enabled, to ensure a safe
migration path for users
upgrading Spark versions.
+On Kubernetes, RPC SSL passwords set in the Spark configuration are propagated
to executor pods
+using environment variables, like the authentication secret described under
+[Kubernetes](#kubernetes) above, so any user who can list pods in the
namespace can read them. In
+cluster mode they are also stored in plaintext in the driver's ConfigMap. To
avoid both, store the
+passwords in a Kubernetes Secret and bind the following variables on the
driver and the executors
+using `spark.kubernetes.driver.secretKeyRef.<name>` and
+`spark.kubernetes.executor.secretKeyRef.<name>`, instead of setting them in
the Spark configuration:
+
+* `_SPARK_SSL_RPC_KEY_STORE_PASSWORD`
+* `_SPARK_SSL_RPC_KEY_PASSWORD`
+* `_SPARK_SSL_RPC_PRIVATE_KEY_PASSWORD`
+* `_SPARK_SSL_RPC_TRUST_STORE_PASSWORD`
+
+For example:
+`spark.kubernetes.executor.secretKeyRef._SPARK_SSL_RPC_KEY_STORE_PASSWORD=<secret-name>:<key>`.
+
+A variable already bound on the executors, through `secretKeyRef`,
`spark.executorEnv` or the
+executor pod template, is not overwritten. Passwords the driver reads from a
Hadoop credential
+provider, from its environment or through variable substitution are not
propagated and must be
Review Comment:
The docs say passwords set through variable substitution must be provided to
the executors directly, which in context means binding `_SPARK_SSL_RPC_*`. That
is not enough: the executor's main SparkEnv re-resolves
`spark.ssl.rpc.keyStorePassword=${env:KS_PASS}` from the conf the driver sends,
`KS_PASS` is unset there, so the password becomes the literal `${env:KS_PASS}`
and the keystore fails to load. Binding only `KS_PASS` fails too, because the
bootstrap fetch reads `_SPARK_SSL_RPC_*`.
Could the substitution case get its own sentence, saying that both the
referenced variable and the matching `_SPARK_SSL_RPC_*` have to be bound on the
executors?
##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala:
##########
@@ -142,6 +143,32 @@ private[spark] class BasicExecutorFeatureStep(
case _ => Nil
}.getOrElse(Nil)
+ // spark.ssl.* passwords are withheld from the executor conf, so they go
through env.
+ // Unlike standalone, a name the user already binds wins; for
secretKeyRef the skip only
+ // keeps a plaintext copy out of the pod spec. envFrom keys are unknown
and lose to env,
+ // so any envFrom disables the injection.
+ val container = Option(pod.container)
+ val userBoundEnvNames = kubernetesConf.secretEnvNamesToKeyRefs.keySet ++
+ kubernetesConf.environment.keySet ++
+ container.flatMap(c => Option(c.getEnv))
+ .map(_.asScala.map(_.getName).toSet).getOrElse(Set.empty)
+ val unboundSslRpcPasswords = kubernetesConf.sslRpcPasswordEnvs.filterNot
{
+ case (name, _) => userBoundEnvNames.contains(name)
+ }
+ val hasEnvFrom = container.flatMap(c =>
Option(c.getEnvFrom)).exists(!_.isEmpty)
+ val sslRpcPasswords = if (hasEnvFrom) {
+ if (unboundSslRpcPasswords.nonEmpty &&
+ BasicExecutorFeatureStep.sslRpcEnvFromWarned.compareAndSet(false,
true)) {
+ val names = unboundSslRpcPasswords.keys.toSeq.sorted.mkString(", ")
+ logWarning(log"Executor container uses envFrom, so RPC SSL passwords
are not " +
+ log"injected. Bind ${MDC(LogKeys.EXECUTOR_ENVS, names)} on the
executors unless " +
+ log"envFrom provides them.")
+ }
+ Seq.empty
+ } else {
+ unboundSslRpcPasswords.toSeq
Review Comment:
The injected passwords go verbatim into `EnvVar.value`, and Kubernetes
expands that field: `$$` becomes `$`, and `$(NAME)` is replaced by a variable
defined earlier in the container. With a keystore password of `pa$$word`, the
executor reads `pa$word`, fails to load the keystore, and dies the same way as
before this fix; a job whose executors already got the password through an
`envFrom` that spark-operator appends (a template `envFrom` disables the
injection, so it is unaffected) goes from working to failing.
Escaping `$` as `$$` in these passwords before injecting (`v.replace("$",
"$$")`) gives back the exact value after expansion; `spark.executorEnv` values
have always been passed to Kubernetes as is, so escaping them would change
existing behavior. A test with a password containing `$$` and
`$(SPARK_EXECUTOR_ID)` that asserts the escaped value in the pod spec would pin
it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
