LuciferYang commented on PR #58489: URL: https://github.com/apache/spark/pull/58489#issuecomment-5548580986
**3. The lister poll doesn't check hasSynced(), so an unsynced empty cache would wipe the snapshot store** `resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsListerSnapshotSource.scala:73` `PollRunnable` unconditionally calls `replaceSnapshot(lister.list())`, but the informer's local cache is empty until the initial LIST finishes. `replaceSnapshot` replaces wholesale with a fresh `fullSnapshotTs`, so `ExecutorPodsLifecycleManager`'s missing-pod reconcile fires on every poll and removes every executor registered more than `missingPodDetectDelta` (30s) ago via `doRemoveExecutor`, while the allocator re-requests a full batch seeing zero known executors. Today the window is masked by the blocking `run()` (by the time backend.start() returns, the cache is synced), but it opens the moment start becomes async per my startup comment: any namespace where the initial sync is slower than `listerPollingInterval` hits it. Skipping the round when `!informer.hasSynced()` fixes it; the "Empty list of pods" test should then assert no replacement happens while unsynced. Note this check needs the force-retry exceptionHandler from my exceptionHandler comment as well — a never-synced, dead informer would otherwise mean polls skipped forever. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
