steveloughran commented on a change in pull request #2399:
URL: https://github.com/apache/hadoop/pull/2399#discussion_r522098602
##########
File path:
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md
##########
@@ -535,20 +535,28 @@ Conflict management is left to the execution engine
itself.
| Option | Magic | Directory | Partitioned | Meaning | Default |
|--------|-------|-----------|-------------|---------|---------|
-| `mapreduce.fileoutputcommitter.marksuccessfuljobs` | X | X | X | Write a
`_SUCCESS` file at the end of each job | `true` |
+| `mapreduce.fileoutputcommitter.marksuccessfuljobs` | X | X | X | Write a
`_SUCCESS` file on the successful completion of the job. | `true` |
+| `fs.s3a.buffer.dir` | X | X | X | Local filesystem directory for data being
written and/or staged. | `${hadoop.tmp.dir}/s3a` |
+| `fs.s3a.committer.magic.enabled` | X | | | Enable "magic committer" support
in the filesystem. | `false` |
+| `fs.s3a.committer.abort.pending.uploads` | X | X | X | list and abort all
pending uploads under the destination path when the job is committed or
aborted. | `true` |
| `fs.s3a.committer.threads` | X | X | X | Number of threads in committers for
parallel operations on files. | 8 |
-| `fs.s3a.committer.staging.conflict-mode` | | X | X | Conflict resolution:
`fail`, `append` or `replace`| `append` |
-| `fs.s3a.committer.staging.unique-filenames` | | X | X | Generate unique
filenames | `true` |
-| `fs.s3a.committer.magic.enabled` | X | | | Enable "magic committer" support
in the filesystem | `false` |
+| `fs.s3a.committer.generate.uuid` | | X | X | Generate a Job UUID if none is
passed down from Spark | `false` |
+| `fs.s3a.committer.require.uuid` | | X | X | Require the Job UUID to be
passed down from Spark | `false` |
+Staging committer (Directory and Partitioned) options
| Option | Magic | Directory | Partitioned | Meaning | Default |
|--------|-------|-----------|-------------|---------|---------|
-| `fs.s3a.buffer.dir` | X | X | X | Local filesystem directory for data being
written and/or staged. | |
-| `fs.s3a.committer.staging.tmp.path` | | X | X | Path in the cluster
filesystem for temporary data | `tmp/staging` |
+| `fs.s3a.committer.staging.conflict-mode` | | X | X | Conflict resolution:
`fail`, `append` or `replace`| `append` |
Review comment:
done. Also reviewed both tables, removed those columns about which
committer supports what option, now they are split into common and staging
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
