xkrogen opened a new pull request #34084: URL: https://github.com/apache/spark/pull/34084
### What changes were proposed in this pull request? Add environment variable resolution logic to `yarn.Client.getUserClasspathUrls`, which allows for users to specify JAR paths (e.g. from `spark.jars`) which contain references to environment variables. This is a best-effort attempt to mimic the variable resolution logic used by a typical shell (implemented in `yarn.Client.replaceEnvVars`). ### Why are the changes needed? In PR #32810 the way user JAR classpaths were passed around was changed to avoid passing them via the command line, which is prone to exceeding maximum argument length limitations. However, as a result, the classpaths are no longer interpreted by the shell, so environment variables are not resolved. This is explicitly called out in the docs of `spark.yarn.config.gatewayPath` as a use case, so we definitely need to continue supporting it. There are more details in the comments of SPARK-36572. ### Does this PR introduce _any_ user-facing change? Yes, using environment variables in `spark.jars` or `spark.yarn.config.replacementPath` will work again, as it did before PR #32810. ### How was this patch tested? New unit tests added for this specific case in `YarnClusterSuite`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
