attilapiros commented on a change in pull request #30675:
URL: https://github.com/apache/spark/pull/30675#discussion_r541197579
##########
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleManager.scala
##########
@@ -109,33 +112,33 @@ private[spark] class ExecutorPodsLifecycleManager(
// Reconcile the case where Spark claims to know about an executor but the
corresponding pod
// is missing from the cluster. This would occur if we miss a deletion
event and the pod
// transitions immediately from running to absent. We only need to check
against the latest
- // snapshot for this, and we don't do this for executors in the deleted
executors cache or
- // that we just removed in this round.
- val lostExecutors = if (snapshots.nonEmpty) {
- schedulerBackend.getExecutorIds().map(_.toLong).toSet --
+ // fresh full snapshot (coming from ExecutorPodsPollingSnapshotSource) for
this, and we don't
+ // do this for executors in the deleted executors cache or that we just
removed in this round.
Review comment:
The "we just removed in this round" means those executors we have just
received the `PodDeleted` for, see this line:
https://github.com/apache/spark/blob/add1c6441981e412c13c539665baa1568f7f23bb/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleManager.scala#L71.
Of course this goes to the deleted executors cache too.
So the old comment just ensures us the current round deleted executors is
handled here as well (by skipping them; they are not forgotten from the logic).
The `registrationTs` is the timestamp when the executor is registered to the
**scheduler backend**.
There is no updates. See this place and class containing it (also this is
only place where executor data is added to this collection):
https://github.com/apache/spark/blob/add1c6441981e412c13c539665baa1568f7f23bb/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L246
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
