Matt Cheah created SPARK-24248:
----------------------------------
Summary: [K8S] Use the Kubernetes cluster as the backing store for
the state of pods
Key: SPARK-24248
URL: https://issues.apache.org/jira/browse/SPARK-24248
Project: Spark
Issue Type: Improvement
Components: Kubernetes
Affects Versions: 2.3.0
Reporter: Matt Cheah
We have a number of places in KubernetesClusterSchedulerBackend right now that
maintains the state of pods in memory. However, the Kubernetes API can always
give us the most up to date and correct view of what our executors are doing.
We should consider moving away from in-memory state as much as can in favor of
using the Kubernetes cluster as the source of truth for pod status. Maintaining
less state in memory makes it so that there's a lower chance that we
accidentally miss updating one of these data structures and breaking the
lifecycle of executors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
