Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/2232#issuecomment-56042913
From the other pr we need to keep backwards compatibility for the env
variables, the configs should be fine. I believe this pr matches the behavior
described below. Please correct me if I'm wrong as I haven't looked at the code
in detail.
The behavior we want is and should have been the behavior as of pr969:
- --archives/--files via sparksubmit defaults to use file:// if not
specified, for both yarn-client and yarn-cluster
- spark.yarn.dist.archives/spark.yarn.dist.files defaults to use file:// if
not specified, for both yarn-client and yarn-cluster
- env variable SPARK_YARN_DIST_ARCHIVES/SPARK_YARN_DIST_FILES set in
yarn-client then it should default to hdfs://
- --files/--archives specified from spark-class in yarn-cluster mode then
it should default to hdfs://
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
