Adding disclaimer for shuffle file consolidation
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/b9451acd Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/b9451acd Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/b9451acd Branch: refs/heads/scala-2.10 Commit: b9451acdf4c73ab4b0b9d9827ab1ccedbcfb144f Parents: 241336a Author: Patrick Wendell <[email protected]> Authored: Fri Dec 6 19:25:28 2013 -0800 Committer: Patrick Wendell <[email protected]> Committed: Fri Dec 6 19:25:28 2013 -0800 ---------------------------------------------------------------------- docs/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/b9451acd/docs/configuration.md ---------------------------------------------------------------------- diff --git a/docs/configuration.md b/docs/configuration.md index 22abe1c..65b41b4 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -331,7 +331,7 @@ Apart from these, the following properties are also available, and may be useful <td>spark.shuffle.consolidateFiles</td> <td>false</td> <td> - If set to "true", consolidates intermediate files created during a shuffle. Creating fewer files can improve filesystem performance if you run shuffles with large numbers of reduce tasks. + If set to "true", consolidates intermediate files created during a shuffle. Creating fewer files can improve filesystem performance for shuffles with large numbers of reduce tasks. It is reccomended to set this to "true" when using ext4 or xfs filesystems. On ext3, this option might degrade performance on machines with many (>8) cores due to filesystem limitations. </td> </tr> <tr>
