astelrastogi opened a new issue, #59426: URL: https://github.com/apache/spark/issues/59426
The spark public docs [here](https://spark.apache.org/docs/latest/declarative-pipelines-programming-guide.html#the-spark-pipelines-command-line-interface) mention that > spark-pipelines is built on top of spark-submit, meaning that it supports all cluster managers supported by spark-submit. It supports all spark-submit arguments except for --class. but if I try to use a `local` cluster manager: ``` spark-pipelines run \ --master "local[*]" \ --name "my-local-sdp-run" \ --spec spark-pipeline.yml ``` I get `Exception in thread "main" org.apache.spark.SparkException: Remote cannot be specified with master and/or deploy mode.` because SDP starts its own spark connect server (sets `isRemote` to true) and master is set as well. Trying to understand how would I use a custom resource manager with spark declarative pipelines. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
