Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/969#issuecomment-46191458
@witgo As I stated above, the config for client mode is not meant for the
end user, it is for internal use that is why I consider this an improvement and
not a bug.
Note that just removing resolveUri's makes the behavior inconsistent with
the sparkSubmit --archives/--files options.
The behavior we want if we add these configs:
- --archives/--files via sparksubmit defaults to use file:// if not
specified, for both yarn-client and yarn-cluster
- spark.yarn.dist.archives/spark.yarn.dist.files defaults to use file://
if not specified, for both yarn-client and yarn-cluster
- env variable SPARK_YARN_DIST_ARCHIVES/SPARK_YARN_DIST_FILES set in
yarn-client then it should default to hdfs://
- --files/--archives specified from spark-class in yarn-cluster mode then
it should default to hdfs://
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
