[
https://issues.apache.org/jira/browse/HADOOP-17833?focusedWorklogId=779066&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-779066
]
ASF GitHub Bot logged work on HADOOP-17833:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 07/Jun/22 12:37
Start Date: 07/Jun/22 12:37
Worklog Time Spent: 10m
Work Description: steveloughran commented on code in PR #3289:
URL: https://github.com/apache/hadoop/pull/3289#discussion_r891168186
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitConstants.java:
##########
@@ -222,13 +226,18 @@ private CommitConstants() {
/**
* Number of threads in committers for parallel operations on files
* (upload, commit, abort, delete...): {@value}.
+ * Two thread pools this size are created, one for the outer
+ * task-level parallelism, and one for parallel execution
+ * within tasks (POSTs to commit individual uploads)
+ * If the value is negative, it is inverted and then multiplied
+ * by the number of cores in the CPU.
*/
public static final String FS_S3A_COMMITTER_THREADS =
"fs.s3a.committer.threads";
/**
* Default value for {@link #FS_S3A_COMMITTER_THREADS}: {@value}.
*/
- public static final int DEFAULT_COMMITTER_THREADS = 8;
+ public static final int DEFAULT_COMMITTER_THREADS = -4;
Review Comment:
yeah, i was thinking about this feature. maybe we will go back to a fixed
number, just a larger one. prevents "surprises" on different deployments.
fixed to 32. larger, but not too large. another strategy would be to somehow
just pick up those in the s3a instance, but limit it to a given value. benefit:
recycling across jobs and tasks.
the reason they are out of that pool is just that the code for parallel
commit came from netflix and we never tried to merge
Issue Time Tracking
-------------------
Worklog Id: (was: 779066)
Time Spent: 10h 10m (was: 10h)
> Improve Magic Committer Performance
> -----------------------------------
>
> Key: HADOOP-17833
> URL: https://issues.apache.org/jira/browse/HADOOP-17833
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 3.3.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 10h 10m
> Remaining Estimate: 0h
>
> Magic committer tasks can be slow because every file created with
> overwrite=false triggers a HEAD (verify there's no file) and a LIST (that
> there's no dir). And because of delayed manifestations, it may not behave as
> expected.
> ParquetOutputFormat is one example of a library which does this.
> we could fix parquet to use overwrite=true, but (a) there may be surprises in
> other uses (b) it'd still leave the list and (c) do nothing for other formats
> call
> Proposed: createFile() under a magic path to skip all probes for file/dir at
> end of path
> Only a single task attempt Will be writing to that directory and it should
> know what it is doing. If there is conflicting file names and parts across
> tasks that won't even get picked up at this point. Oh and none of the
> committers ever check for this: you'll get the last file manifested (s3a) or
> renamed (file)
> If we skip the checks we will save 2 HTTP requests/file.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
