Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19954#discussion_r157893811
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala
---
@@ -45,6 +45,59 @@ private[spark] class KubernetesClusterManager extends
ExternalClusterManager wit
masterURL: String,
scheduler: TaskScheduler): SchedulerBackend = {
val sparkConf = sc.getConf
+ val maybeInitContainerConfigMap =
sparkConf.get(INIT_CONTAINER_CONFIG_MAP_NAME)
+ val maybeInitContainerConfigMapKey =
sparkConf.get(INIT_CONTAINER_CONFIG_MAP_KEY_CONF)
+
+ if (maybeInitContainerConfigMap.isEmpty) {
+ logWarning("The executor's init-container config map was not
specified. Executors will " +
+ "therefore not attempt to fetch remote or submitted dependencies.")
+ }
+
+ if (maybeInitContainerConfigMapKey.isEmpty) {
+ logWarning("The executor's init-container config map key was not
specified. Executors will " +
+ "therefore not attempt to fetch remote or submitted dependencies.")
+ }
+
+ // Only set up the bootstrap if they've provided both the config map
key and the config map
+ // name. The config map might not be provided if init-containers
aren't being used to
+ // bootstrap dependencies.
+ val maybeInitContainerBootstrap = for {
--- End diff --
Optimally, you (the orchestrator?) would just instantiate all the steps and
each step would be responsible for figuring out if it has anything to do...
that would simplify code in the call sites (like here).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
