attilapiros commented on a change in pull request #30675:
URL: https://github.com/apache/spark/pull/30675#discussion_r554415649
##########
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleManager.scala
##########
@@ -109,33 +112,33 @@ private[spark] class ExecutorPodsLifecycleManager(
// Reconcile the case where Spark claims to know about an executor but the
corresponding pod
// is missing from the cluster. This would occur if we miss a deletion
event and the pod
// transitions immediately from running to absent. We only need to check
against the latest
- // snapshot for this, and we don't do this for executors in the deleted
executors cache or
- // that we just removed in this round.
- val lostExecutors = if (snapshots.nonEmpty) {
- schedulerBackend.getExecutorIds().map(_.toLong).toSet --
+ // fresh full snapshot (coming from ExecutorPodsPollingSnapshotSource) for
this, and we don't
+ // do this for executors in the deleted executors cache or that we just
removed in this round.
+ if (snapshots.nonEmpty && lastFullSnapshotTs !=
snapshots.last.fullSnapshotTs) {
+ lastFullSnapshotTs = snapshots.last.fullSnapshotTs
+ val lostExecutorsWithRegistrationTs =
+ schedulerBackend.getExecutorsWithRegistrationTs().map(t =>
(t._1.toLong, t._2)) --
Review comment:
In the `CoarseGrainedSchedulerBackend` the executor ID is a string. See
the other methods, like:
https://github.com/apache/spark/blob/4a4853ae369859bec6fef395c0ee7d405b8ed36b/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L628-L630
https://github.com/apache/spark/blob/4a4853ae369859bec6fef395c0ee7d405b8ed36b/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L636
So to keep the consistency I would leave `getExecutorsWithRegistrationTs` as
it is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
