Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19130#discussion_r139576814
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -367,6 +368,54 @@ object SparkSubmit extends CommandLineUtils with
Logging {
}.orNull
}
+ // When running in YARN, for some remote resources with scheme:
--- End diff --
This is a problem for YARN currently, because YARN uses dist cache to
distribute resources to yarn cluster, dist cache requires supported Hadoop FS
to copy resources, if our resource scheme is http, it will try to find http FS
to handle such resource, which will be failed since no http FS supported in
current Hadoop.
In standalone and Mesos cluster, we use Spark's internal logic to handle
http resources, this logic handles well for the http(s) resources, so there
should be no issue for standalone and mesos mode.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
