[
https://issues.apache.org/jira/browse/SPARK-60480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60480:
-----------------------------------
Labels: pull-request-available (was: )
> Avoid scanning healthy executor pods in ExecutorPodsLifecycleManager
> --------------------------------------------------------------------
>
> Key: SPARK-60480
> URL: https://issues.apache.org/jira/browse/SPARK-60480
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 5.0.0
> Reporter: Victor Sunderland
> Priority: Major
> Labels: pull-request-available
>
> {{ExecutorPodsLifecycleManager.onNewSnapshots()}} scans the full executor pod
> map of every snapshot twice: once to collect failed executors and once to
> handle final and inactive pods. Healthy (pending or running) pods never need
> lifecycle handling, so with many executors and frequent watch updates most of
> this work is wasted.
> Each {{ExecutorPodsSnapshot}} can keep an index of its final and inactive
> pods, maintained incrementally in {{withUpdate()}}, so the manager only
> visits the pods it acts on. Every snapshot is still processed, so failures
> seen in intermediate snapshots and retries of failed pod deletions are
> preserved. Missing-pod detection keeps using the full pod map.
> In a local synthetic benchmark with 10,000 executor pods and 100 watch
> updates per batch, snapshot updates plus lifecycle processing for healthy-pod
> batches dropped from 14.9 ms to 0.011 ms.
> This is similar to SPARK-55963, which optimized snapshot traversal in
> {{ExecutorPodsAllocator}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
