Hi all, I just created a JIRA ticket and a work in progress PR. Here is the link to the JIRA ticket - https://issues.apache.org/jira/browse/SPARK-52041 Here is the link to the GitHub PR - https://github.com/apache/spark/pull/50770
I kindly ask for feedback. Kind regards On Wed, Feb 19, 2025 at 11:30 AM Enrico Minack <enrico-min...@gmx.de.invalid> wrote: > Hi devs, > > Let me pull some spark-submit developers into this discussion. > > @dongjoon-hyun @HyukjinKwon @cloud-fan > > What are your thoughts on making spark-submit fully and generically > support ExternalClusterManager implementations? > > The current situation is that the only way to submit a Spark job via an > ExternalClusterManager implementation is to use *client mode* and a *"local" > URL*, which seems counterintuitive. > > Here is an example for our Armada cluster manager, but this discussion is > meant to be generic for any ExternalClusterManager implementation: > > spark-submit --deploy-mode *client* --master *local://* > armada://IP:PORT ... > > What you would expect is this to work: > > spark-submit --deploy-mode cluster --master armada://IP:PORT ... > > What are your feelings on making spark-submit support master URLs that are > handled by an ExternalClusterManager implementation in cluster mode? > > We are happy to contribute a generic fix for that. > > Cheers, > Enrico > > > > Am 07.02.25 um 16:26 schrieb George J: > > To me, this seems like a gap in the "pluggable cluster manager" > implementation. > > What is the value of making cluster managers pluggable, if spark-submit > doesn't accept jobs on those cluster managers? > > It seems to me, for pluggable cluster managers to work, you would want > some parts of spark-submit to be "pluggable" as well. > > Thanks! > George Jahad > G-Research Developer > > >
