Repository: spark Updated Branches: refs/heads/branch-1.3 29e39e178 -> febb12308
[SPARK-6313] Add config option to disable file locks/fetchFile cache to ... ...support NFS mounts. This is a work around for now with the goal to find a more permanent solution. https://issues.apache.org/jira/browse/SPARK-6313 Author: nemccarthy <[email protected]> Closes #5036 from nemccarthy/master and squashes the following commits: 2eaaf42 [nemccarthy] [SPARK-6313] Update config wording doc for spark.files.useFetchCache 5de7eb4 [nemccarthy] [SPARK-6313] Add config option to disable file locks/fetchFile cache to support NFS mounts (cherry picked from commit 4cca3917dc30ee907e6cbd6a569b6ac58af963f7) Signed-off-by: Josh Rosen <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/febb1230 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/febb1230 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/febb1230 Branch: refs/heads/branch-1.3 Commit: febb12308dac94af20279a31e8a6013690f42f24 Parents: 29e39e1 Author: nemccarthy <[email protected]> Authored: Tue Mar 17 09:33:11 2015 -0700 Committer: Josh Rosen <[email protected]> Committed: Tue Mar 17 09:33:28 2015 -0700 ---------------------------------------------------------------------- core/src/main/scala/org/apache/spark/util/Utils.scala | 3 ++- docs/configuration.md | 12 ++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/febb1230/core/src/main/scala/org/apache/spark/util/Utils.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/util/Utils.scala b/core/src/main/scala/org/apache/spark/util/Utils.scala index 568ff3b..75a40f1 100644 --- a/core/src/main/scala/org/apache/spark/util/Utils.scala +++ b/core/src/main/scala/org/apache/spark/util/Utils.scala @@ -402,7 +402,8 @@ private[spark] object Utils extends Logging { useCache: Boolean) { val fileName = url.split("/").last val targetFile = new File(targetDir, fileName) - if (useCache) { + val fetchCacheEnabled = conf.getBoolean("spark.files.useFetchCache", defaultValue = true) + if (useCache && fetchCacheEnabled) { val cachedFileName = s"${url.hashCode}${timestamp}_cache" val lockFileName = s"${url.hashCode}${timestamp}_lock" val localDir = new File(getLocalDir(conf)) http://git-wip-us.apache.org/repos/asf/spark/blob/febb1230/docs/configuration.md ---------------------------------------------------------------------- diff --git a/docs/configuration.md b/docs/configuration.md index a7116fb..9f97dd2 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -746,6 +746,18 @@ Apart from these, the following properties are also available, and may be useful </td> </tr> <tr> + <td><code>spark.files.useFetchCache</code></td> + <td>true</td> + <td> + If set to true (default), file fetching will use a local cache that is shared by executors + that belong to the same application, which can improve task launching performance when + running many executors on the same host. If set to false, these caching optimizations will + be disabled and all executors will fetch their own copies of files. This optimization may be + disabled in order to use Spark local directories that reside on NFS filesystems (see + <a href="https://issues.apache.org/jira/browse/SPARK-6313";>SPARK-6313</a> for more details). + </td> +</tr> +<tr> <td><code>spark.files.overwrite</code></td> <td>false</td> <td> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
