Hi devs, Let me pull some spark-submit developers into this discussion.
@dongjoon-hyun @HyukjinKwon @cloud-fan What are your thoughts on making spark-submit fully and generically support ExternalClusterManager implementations? The current situation is that the only way to submit a Spark job via an ExternalClusterManager implementation is to use *client mode* and a *"local" URL*, which seems counterintuitive. Here is an example for our Armada cluster manager, but this discussion is meant to be generic for any ExternalClusterManager implementation: spark-submit --deploy-mode *client* --master *local://*armada://IP:PORT ... What you would expect is this to work: spark-submit --deploy-mode cluster --master armada://IP:PORT ... What are your feelings on making spark-submit support master URLs that are handled by an ExternalClusterManager implementation in cluster mode? We are happy to contribute a generic fix for that. Cheers, Enrico Am 07.02.25 um 16:26 schrieb George J:
To me, this seems like a gap in the "pluggable cluster manager" implementation. What is the value of making cluster managers pluggable, if spark-submit doesn't accept jobs on those cluster managers? It seems to me, for pluggable cluster managers to work, you would want some parts of spark-submit to be "pluggable" as well. Thanks! George Jahad G-Research Developer
