Merge pull request #236 from pwendell/shuffle-docs Adding disclaimer for shuffle file consolidation (cherry picked from commit 1b38f5f2774982d524742e987b6cef26ccaae676)
Signed-off-by: Patrick Wendell <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/cfca70e4 Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/cfca70e4 Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/cfca70e4 Branch: refs/heads/branch-0.8 Commit: cfca70e4b285a321e9a4543f3965430e6d72b2fd Parents: 80cc4ff Author: Patrick Wendell <[email protected]> Authored: Fri Dec 6 20:16:15 2013 -0800 Committer: Patrick Wendell <[email protected]> Committed: Sat Dec 7 01:15:20 2013 -0800 ---------------------------------------------------------------------- docs/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/cfca70e4/docs/configuration.md ---------------------------------------------------------------------- diff --git a/docs/configuration.md b/docs/configuration.md index 0edbac2..e86b9ea 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -331,7 +331,7 @@ Apart from these, the following properties are also available, and may be useful <td>spark.shuffle.consolidateFiles</td> <td>false</td> <td> - If set to "true", consolidates intermediate files created during a shuffle. Creating fewer files can improve filesystem performance if you run shuffles with large numbers of reduce tasks. + If set to "true", consolidates intermediate files created during a shuffle. Creating fewer files can improve filesystem performance for shuffles with large numbers of reduce tasks. It is reccomended to set this to "true" when using ext4 or xfs filesystems. On ext3, this option might degrade performance on machines with many (>8) cores due to filesystem limitations. </td> </tr> <tr>
