[
https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471288#comment-16471288
]
Yinan Li commented on SPARK-24248:
----------------------------------
Just realized one thing: solely replying on the watcher poses risks of losing
executor pod updates. This can potentially happen for example if the API server
gets restarted or if the watch connection is interrupted temporarily while the
pods are running. So periodic polling is still needed. This is referred to as
resync in controller term. Enabling resync is almost always a good thing.
> [K8S] Use the Kubernetes cluster as the backing store for the state of pods
> ---------------------------------------------------------------------------
>
> Key: SPARK-24248
> URL: https://issues.apache.org/jira/browse/SPARK-24248
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 2.3.0
> Reporter: Matt Cheah
> Priority: Major
>
> We have a number of places in KubernetesClusterSchedulerBackend right now
> that maintains the state of pods in memory. However, the Kubernetes API can
> always give us the most up to date and correct view of what our executors are
> doing. We should consider moving away from in-memory state as much as can in
> favor of using the Kubernetes cluster as the source of truth for pod status.
> Maintaining less state in memory makes it so that there's a lower chance that
> we accidentally miss updating one of these data structures and breaking the
> lifecycle of executors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
