Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19130#discussion_r139608374
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -385,4 +385,14 @@ package object config {
.checkValue(v => v > 0 && v <= Int.MaxValue,
s"The buffer size must be greater than 0 and less than
${Int.MaxValue}.")
.createWithDefault(1024 * 1024)
+
+ private[spark] val FORCE_DOWNLOAD_SCHEMES =
+ ConfigBuilder("spark.yarn.dist.forceDownloadSchemes")
+ .doc("Comma-separated list of schemes for which files will be
downloaded to the " +
+ "local disk prior to being added to YARN's distributed cache. For
use in cases " +
+ "where the YARN service does not support schemes that are
supported by Spark, like http, " +
+ "https, ftp.")
--- End diff --
It is not required, we still want to leverage Hadoop's http(s) FS to
distribute resources by default if it is supported in Hadoop 2.9+
(https://issues.apache.org/jira/browse/HADOOP-14383)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
