[
https://issues.apache.org/jira/browse/SPARK-52041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-52041:
----------------------------------
Target Version/s: (was: 4.1.0)
> Add better support for integrating with external cluster manager #50770
> -----------------------------------------------------------------------
>
> Key: SPARK-52041
> URL: https://issues.apache.org/jira/browse/SPARK-52041
> Project: Spark
> Issue Type: Improvement
> Components: Spark Submit
> Affects Versions: 4.1.0
> Reporter: Dejan Zele Pejchev
> Priority: Minor
> Labels: pull-request-available
>
> h3. What changes were proposed?
> * Remove previous checks in {{SparkSubmit}} that restricted use of external
> managers to specific master URLs.
> * Add new configuration field {{{}spark.submit.external.class{}}}.
> h3. Why are the changes needed?
> SparkSubmit currently only supports built-in cluster managers (YARN,
> Kubernetes, Standalone), making it difficult to integrate custom or
> third-party schedulers without forking the codebase.
> By removing hard-coded restrictions, this patch enables easier integration
> with various cluster managers.
> Now the following is possible (assuming the implementation is loaded):
> bin/spark-submit \
> --master armada://localhost:30002 \
> --deploy-mode cluster
> Does this change introduce _any_ user-facing change?
> Yes, users can easily integrate with their custom cluster managers.
> h3. How was this patch tested?
> Added unit tests and tested with
> [armadaproject/armada-spark|https://github.com/armadaproject/armada-spark]
> custom integration and with manual testing while developing the Armada
> integration.
> h3. Was this patch authored or co-authored using generative AI tooling?
> No.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
