[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516368#comment-16516368 ] Matt Cheah commented on SPARK-24248: I've summarized what we ended up going with after some

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481217#comment-16481217 ] Apache Spark commented on SPARK-24248: -- User 'mccheah' has created a pull request for this issue:

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477825#comment-16477825 ] Yinan Li commented on SPARK-24248: -- Re-sync is not a fallback nor a replacement, but a complement to the

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477812#comment-16477812 ] Erik Erlandson commented on SPARK-24248: Is the design above using re-sync as the fallback for

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474774#comment-16474774 ] Anirudh Ramanathan commented on SPARK-24248: Somewhat related -

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471479#comment-16471479 ] Yinan Li commented on SPARK-24248: -- I think it's both more robust and easier to implement with a

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471295#comment-16471295 ] Matt Cheah commented on SPARK-24248: I see - I suppose if the watch connection drops, we should try

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471288#comment-16471288 ] Yinan Li commented on SPARK-24248: -- Just realized one thing: solely replying on the watcher poses risks

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471280#comment-16471280 ] Matt Cheah commented on SPARK-24248: I thought about it a bit more, and believe that we can do most

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471259#comment-16471259 ] Yinan Li commented on SPARK-24248: -- Actually even if the fabric8 client does not support caching, we can

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471244#comment-16471244 ] Yinan Li commented on SPARK-24248: -- It's potentially possible to get rid of the in-memory state in favor

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471206#comment-16471206 ] Matt Cheah commented on SPARK-24248: [~foxish] [~liyinan926] curious as to what you think about this