[
https://issues.apache.org/jira/browse/HADOOP-14971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260082#comment-16260082
]
ASF GitHub Bot commented on HADOOP-14971:
-----------------------------------------
Github user ajfabbri commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/282#discussion_r152149374
--- Diff:
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md ---
@@ -550,6 +551,102 @@ Conflict management is left to the execution engine
itself.
| `fs.s3a.committer.staging.tmp.path` | | X | X | Path in the cluster
filesystem for temporary data | `tmp/staging` |
+```xml
+<property>
+ <name>fs.s3a.committer.name</name>
+ <value>file</value>
+ <description>
+ Committer to create for output to S3A, one of:
+ "file", "directory", "partitioned", "magic".
+ </description>
+</property>
+
+<property>
+ <name>fs.s3a.committer.magic.enabled</name>
+ <value>false</value>
+ <description>
+ Enable support in the filesystem for the S3 "Magic" committer.
+ When working with AWS S3, S3Guard must be enabled for the destination
+ bucket, as consistent metadata listings are required.
+ </description>
+</property>
+
+<property>
+ <name>fs.s3a.committer.threads</name>
+ <value>8</value>
+ <description>
+ Number of threads in committers for parallel operations on files
+ (upload, commit, abort, delete...)
+ </description>
+</property>
+
+<property>
+ <name>fs.s3a.committer.staging.tmp.path</name>
+ <value>tmp/staging</value>
+ <description>
+ Path in the cluster filesystem for temporary data.
+ This is for HDFS, not the local filesystem.
+ It is only for the summary data of each file, not the actual
+ data being committed.
+ Using an unqualified path guarantees that the full path will be
+ generated relative to the home directory of the user creating the job,
+ hence private (assuming home directory permissions are secure).
+ </description>
+</property>
+
+<property>
+ <name>fs.s3a.committer.staging.unique-filenames</name>
+ <value>true</value>
+ <description>
+ Option for final files to have a unique name through job attempt info,
+ or the value of fs.s3a.committer.staging.uuid
+ When writing data with the "append" conflict option, this guarantees
+ that new data will not overwrite any existing data.
+ </description>
+</property>
+
+<property>
+ <name>fs.s3a.committer.staging.conflict-mode</name>
+ <value>fail</value>
+ <description>
+ Staging committer conflict resolution policy.
+ Supported: "fail", "append", "replace".
+ </description>
+</property>
+
+<property>
+ <name>s.s3a.committer.staging.abort.pending.uploads</name>
--- End diff --
typo here
> Merge S3A committers into trunk
> -------------------------------
>
> Key: HADOOP-14971
> URL: https://issues.apache.org/jira/browse/HADOOP-14971
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-13786-040.patch, HADOOP-13786-041.patch
>
>
> Merge the HADOOP-13786 committer into trunk. This branch is being set up as a
> github PR for review there & to keep it out the mailboxes of the watchers on
> the main JIRA
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
