akshatb1 opened a new pull request #33134: URL: https://github.com/apache/spark/pull/33134
### What changes were proposed in this pull request? - This PR allows to add a custom implementation of YARN_CLUSTER_SUBMIT_CLASS as a configuration. - This is useful when there is a custom variant of Yarn cluster running which requires a modified Yarn client. - Users can implement a class similar to YarnClusterApplication which can invoke a custom Yarn client. Users should add the relevant dependencies in the classpath. - Default value of the configuration is set to the current class: org.apache.spark.deploy.yarn.YarnClusterApplication. ### Why are the changes needed? This is required when there is a custom variant of Yarn cluster running which requires a modified Yarn client. For example, there may be different handling of credentials required in a custom Yarn cluster. ### Does this PR introduce _any_ user-facing change? Yes, this PR introduces a new configuration key but the behaviour is unchanged without explicitly modifying the config. Documentation is added for the new configuration. ### How was this patch tested? Tested following in a local Yarn cluster: Tested without the config added - Existing client is used to submit application Tested with config added - New client is used to submit application Tested with config added but class not present - Spark-submit throws error: Could not load YARN classes. Unit tests: Added unit tests when the configuration is enabled. Existing unit tests which validate the current YarnClusterApplication class name are passing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
