Github user mccheah commented on a diff in the pull request:
https://github.com/apache/spark/pull/21366#discussion_r195513995
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala
---
@@ -56,17 +58,44 @@ private[spark] class KubernetesClusterManager extends
ExternalClusterManager wit
Some(new File(Config.KUBERNETES_SERVICE_ACCOUNT_TOKEN_PATH)),
Some(new File(Config.KUBERNETES_SERVICE_ACCOUNT_CA_CRT_PATH)))
- val allocatorExecutor = ThreadUtils
- .newDaemonSingleThreadScheduledExecutor("kubernetes-pod-allocator")
val requestExecutorsService = ThreadUtils.newDaemonCachedThreadPool(
"kubernetes-executor-requests")
+
+ val bufferSnapshotsExecutor = ThreadUtils
+
.newDaemonSingleThreadScheduledExecutor("kubernetes-executor-snapshots-buffer")
+ val snapshotsStore = new
ExecutorPodsSnapshotsStoreImpl(bufferSnapshotsExecutor)
+ val removedExecutorsCache = CacheBuilder.newBuilder()
+ .expireAfterWrite(3, TimeUnit.MINUTES)
--- End diff --
The cache is only for a best effort attempt to not remove the same executor
from the scheduler backend multiple times, but at the end of the day even if we
do accidentally remove multiple times the only noticeable result is noisy logs.
The scheduler backend properly handles multiple attempts to remove but we'd
prefer it if we didn't have to.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
