waleedfateem commented on a change in pull request #29541:
URL: https://github.com/apache/spark/pull/29541#discussion_r477389757
##########
File path: docs/configuration.md
##########
@@ -1761,11 +1761,16 @@ Apart from these, the following properties are also
available, and may be useful
</tr>
<tr>
<td><code>spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version</code></td>
- <td>1</td>
+ <td>Dependent on environment</td>
<td>
The file output committer algorithm version, valid algorithm version
number: 1 or 2.
Version 2 may have better performance, but version 1 may handle failures
better in certain situations,
as per <a
href="https://issues.apache.org/jira/browse/MAPREDUCE-4815";>MAPREDUCE-4815</a>.
+ The default value depends on the Hadoop version used in an environment:
+ 1 for Hadoop versions lower than 3.0
+ 2 for Hadoop versions 3.0 and higher
+ It's important to note that this can change back to 1 again in the future
once <a
href="https://issues/apache.org.jira/browse/MAPREDUCE-7282";>MAPREDUCE-7282</a>
Review comment:
Whoops! Fixed that and pushed changes. I made sure the links work this
time.
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
