Well, everything is possible. Please initiate a discussion on the matter of a proposal to "Create a pluggable cluster manager" and put it to the community.
See some examples here https://lists.apache.org/list.html?dev@spark.apache.org HTH Dr Mich Talebzadeh, Architect | Data Science | Financial Crime | Forensic Analysis | GDPR view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> On Fri, 7 Feb 2025 at 15:50, Dejan Pejchev <de...@gr-oss.io> wrote: > This External Cluster Manager is an amazing concept and I really like the > separation. > > Would it be possible to include a broader group and discuss an approach on > how to make Spark more pluggable? It is a bit far fetched but we would be > very much interested in working on this if this resonates well within the > Spark community. > > On Fri, Feb 7, 2025 at 4:45 PM Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > >> Agreed, If the goal is to make Spark truly pluggable, the spark-submit >> tool itself should be more flexible in handling different cluster managers >> and their specific requirements. >> >> 1. Back in the days, Spark's initial development focused on a limited >> set of cluster managers (Standalone, YARN). >> 2. Implementing *a full pluggability for spark-submit *would require >> redesign and implementation to handle the diverse requirements of >> different >> cluster managers which I think will be a major project for itself >> >> HTH >> >> Dr Mich Talebzadeh, >> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR >> >> view my Linkedin profile >> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >> >> >> >> >> >> On Fri, 7 Feb 2025 at 15:27, George J <georgej...@gmail.com> wrote: >> >>> To me, this seems like a gap in the "pluggable cluster manager" >>> implementation. >>> >>> What is the value of making cluster managers pluggable, if spark-submit >>> doesn't accept jobs on those cluster managers? >>> >>> It seems to me, for pluggable cluster managers to work, you would want >>> some parts of spark-submit to be "pluggable" as well. >>> >>> Thanks! >>> George Jahad >>> G-Research Developer >>> >>
