littlexyw opened a new pull request, #58489: URL: https://github.com/apache/spark/pull/58489
### What changes were proposed in this pull request? Add an opt-in `SharedIndexInformer` + `Lister` based path for tracking executor pod state, as an alternative to `ExecutorPodsWatchSnapshotSource` + `ExecutorPodsPollingSnapshotSource`. A new trait `ExecutorPodsSnapshotSource` is extracted so the existing watch/poll sources and the new informer/lister sources share one interface. The two paths are mutually exclusive and selected by a new config; the default is unchanged. This is a follow-up to the closed #51396 by @ForVic, incorporating review from @dongjoon-hyun and @skonto: dropped the pluggable custom-source surface in favor of a single boolean switch, renamed classes to drop the `Custom` infix, `ExecutorPodsSnapshotSource` no longer extends `Logging`, and the informer is filtered server-side by label (`app-id + role=executor + !inactive`) at construction, so its local cache — and therefore `Lister.list()` — only contains this application's executor pods. ### Why are the changes needed? `ExecutorPodsPollingSnapshotSource` issues a full `pods().list()` against the apiserver every `spark.kubernetes.executor.apiPollingInterval` (30s default) as a safety net for missed watch events. LIST hits the apiserver's in-memory watch cache and does not go to etcd, but the watch cache indexes only certain fields (namespace, name, nodeName…) and not labels, so the label selector is applied by scanning every Pod in the namespace and matching in memory. Per-request cost therefore scales with the namespace's total pod count rather than with the result size. When a shared K8s cluster hosts many concurrent Spark applications, the aggregate steady-state LIST QPS from all drivers is material. `SharedIndexInformer` does one initial LIST and then keeps a local `Indexer` in sync via a single long-lived WATCH, resuming from the last observed `resourceVersion` on disconnect (only re-listing on HTTP 410). `Lister.list()` reads that local `Indexer`, so the periodic snapshot has zero apiserver cost. Optional `informerResyncInterval` replays from the local cache; it does not re-list against the apiserver. ### Does this PR introduce _any_ user-facing change? No behavior change by default. Adds `spark.kubernetes.executor.enableInformer` (off by default) plus `spark.kubernetes.executor.listerPollingInterval` (default `30s`) and `spark.kubernetes.executor.informerResyncInterval` (default `0s`, disabled). ### How was this patch tested? New unit tests: `ExecutorPodsInformerSnapshotSourceSuite`, `ExecutorPodsListerSnapshotSourceSuite`, `InformerManagerSuite`. Manually verified on an internal Spark on K8s cluster. ### Was this patch authored or co-authored using generative AI tooling? Assisted-by: Claude Opus 4.7 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
