Re: Extending Spark with a custom ExternalClusterManager

[Jules Damji]

Yes, if this becomes a need that surfaces time and again, then it’s worthwhile to start a broader discussion in a manner of high-level proposal, which could trigger favorable discussion leading to next steps.  Cheers Jules  — Sent from my iPhone Pardon the dumb thumb typos :) On Feb 7, 2025, at 8:00 AM, Mich Talebzadeh <mich.talebza...@gmail.com> wrote:  Well, everything is possible. Please initiate a discussion on the matter of a proposal to "Create a pluggable cluster manager" and put it to the community.  See some examples here https://lists.apache.org/list.html?dev@spark.apache.org HTH Dr Mich Talebzadeh, Architect | Data Science | Financial Crime | Forensic Analysis | GDPR    view my Linkedin profile   On Fri, 7 Feb 2025 at 15:50, Dejan Pejchev <de...@gr-oss.io> wrote: This External Cluster Manager is an amazing concept and I really like the separation. Would it be possible to include a broader group and discuss an approach on how to make Spark more pluggable? It is a bit far fetched but we would be very much interested in working on this if this resonates well within the Spark community. On Fri, Feb 7, 2025 at 4:45 PM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: Agreed, If the goal is to make Spark truly pluggable, the spark-submit tool itself should be more flexible in handling different cluster managers and their specific requirements. Back in the days, Spark's initial development focused on a limited set of cluster managers (Standalone, YARN). Implementing a full pluggability for spark-submit would require redesign and implementation to handle the diverse requirements of different cluster managers which I think will be a major project for itself HTH Dr Mich Talebzadeh, Architect | Data Science | Financial Crime | Forensic Analysis | GDPR    view my Linkedin profile   On Fri, 7 Feb 2025 at 15:27, George J <georgej...@gmail.com> wrote: To me, this seems like a gap in the "pluggable cluster manager" implementation. What is the value of making cluster managers pluggable, if spark-submit doesn't accept jobs on those cluster managers? It seems to me, for pluggable cluster managers to work, you would want some parts of spark-submit to be "pluggable" as well. Thanks! George Jahad G-Research Developer