Gaurangi94 commented on pull request #29630: URL: https://github.com/apache/spark/pull/29630#issuecomment-688991583
> I don't think Spark has the concept of "clusters". Even I don't think Spark has the concept of "cluster", unless you use standalone mode. More specifically, there's no strong relation between applications, and there's no sort of control plane for Spark side to control all applications in the cluster. The cluster is actually resource scheduler's cluster. > > If the rationalization of [SPARK-32097](https://issues.apache.org/jira/browse/SPARK-32097) and [SPARK-32135](https://issues.apache.org/jira/browse/SPARK-32135) is to make SHS be cluster-wise, then probably the concept of "cluster" needs to be defined and introduced, instead of having some fixes for workaround. > > Everyone has different views on being "cluster-wise". e.g. If SHS is cluster-wise and supports multi-clusters, I would prefer "isolated view" per cluster, like selecting the cluster first, and see filtered view for the cluster. I wouldn't prefer listing all applications from all clusters in the same list. This would be the opposite view you're proposing in [SPARK-32135](https://issues.apache.org/jira/browse/SPARK-32135). So that's not a trivial thing. It warrants the discussion, including we really want to make it be "cluster-wise". Hey, I have updated the JIRA with a more elaborate description of our use-case. Initially, I had tried to keep it general but I noticed that has caused a lot of unintended confusion. Apologies for that. Could you review our use-case again and we can discuss what you think would be a better way to support it. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
