Github user liyinan926 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19954#discussion_r157913485
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala
---
@@ -45,6 +45,59 @@ private[spark] class KubernetesClusterManager extends
ExternalClusterManager wit
masterURL: String,
scheduler: TaskScheduler): SchedulerBackend = {
val sparkConf = sc.getConf
+ val maybeInitContainerConfigMap =
sparkConf.get(INIT_CONTAINER_CONFIG_MAP_NAME)
+ val maybeInitContainerConfigMapKey =
sparkConf.get(INIT_CONTAINER_CONFIG_MAP_KEY_CONF)
+
+ if (maybeInitContainerConfigMap.isEmpty) {
+ logWarning("The executor's init-container config map was not
specified. Executors will " +
+ "therefore not attempt to fetch remote or submitted dependencies.")
+ }
+
+ if (maybeInitContainerConfigMapKey.isEmpty) {
+ logWarning("The executor's init-container config map key was not
specified. Executors will " +
+ "therefore not attempt to fetch remote or submitted dependencies.")
+ }
+
+ // Only set up the bootstrap if they've provided both the config map
key and the config map
+ // name. The config map might not be provided if init-containers
aren't being used to
+ // bootstrap dependencies.
+ val maybeInitContainerBootstrap = for {
--- End diff --
Agreed. This part definitely needs refactoring once we start using an
orchestrator pattern for the executors.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
