Github user aditanase commented on a diff in the pull request:
https://github.com/apache/spark/pull/22904#discussion_r239732258
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
---
@@ -67,8 +66,16 @@ private[spark] object SparkKubernetesClientFactory {
val dispatcher = new Dispatcher(
ThreadUtils.newDaemonCachedThreadPool("kubernetes-dispatcher"))
- // TODO [SPARK-25887] Create builder in a way that respects
configurable context
- val config = new ConfigBuilder()
+ // Allow for specifying a context used to auto-configure from the
users K8S config file
+ val kubeContext = sparkConf.get(KUBERNETES_CONTEXT).filter(c =>
StringUtils.isNotBlank(c))
+ logInfo(s"Auto-configuring K8S client using " +
+ s"${if (kubeContext.isEmpty) s"context ${kubeContext.get}" else
"current context"}" +
+ s" from users K8S config file")
+
+ // Start from an auto-configured config with the desired context
+ // Fabric 8 uses null to indicate that the users current context
should be used so if no
+ // explicit setting pass null
+ val config = new
ConfigBuilder(autoConfigure(kubeContext.getOrElse(null)))
--- End diff --
This somewhat answers my previous question, and it makes sense. Seems like
Fabric8 is behaving consistently with other SDKs (e.g. AWS comes to mind as it
will try 3-4 default configuration options, including env vars, iam machine
role, etc).
That being said, if you were to deploy an app inside a docker container, it
won't see any of those and one still needs to inject some env vars or config
maps for the AWS SDK to pick up.
Long story short, I think this enhancement does not apply to client mode,
and we should make that clearer in the docs, while providing alternatives for
issues like https://issues.apache.org/jira/browse/SPARK-26295.
Hopefully I'm making sense.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
