[
https://issues.apache.org/jira/browse/HADOOP-17833?focusedWorklogId=778700&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-778700
]
ASF GitHub Bot logged work on HADOOP-17833:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 06/Jun/22 17:00
Start Date: 06/Jun/22 17:00
Worklog Time Spent: 10m
Work Description: steveloughran commented on PR #3289:
URL: https://github.com/apache/hadoop/pull/3289#issuecomment-1147670480
Wrapping up createFile() with the ability to set headers on the file
builder.must("fs.s3a.create.header.my-header","my-value");
this will set a user metadata entry "my-header" which xattr will return
as "header.my-header".
This forced me to make PutOptions something passed around more rigorously and
used as the parameter class to innerCreateFile. This is ultimately good as it
lets us set new options later..encryption, storage class etc. if we so
choose.
also: s3a path capabilities let you probe for this prefix, as well as the
performance one...think this should be the standard behaviour for custom
options.
also: fixed the various style/javadoc warnings
Issue Time Tracking
-------------------
Worklog Id: (was: 778700)
Time Spent: 8h 50m (was: 8h 40m)
> Improve Magic Committer Performance
> -----------------------------------
>
> Key: HADOOP-17833
> URL: https://issues.apache.org/jira/browse/HADOOP-17833
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 3.3.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 8h 50m
> Remaining Estimate: 0h
>
> Magic committer tasks can be slow because every file created with
> overwrite=false triggers a HEAD (verify there's no file) and a LIST (that
> there's no dir). And because of delayed manifestations, it may not behave as
> expected.
> ParquetOutputFormat is one example of a library which does this.
> we could fix parquet to use overwrite=true, but (a) there may be surprises in
> other uses (b) it'd still leave the list and (c) do nothing for other formats
> call
> Proposed: createFile() under a magic path to skip all probes for file/dir at
> end of path
> Only a single task attempt Will be writing to that directory and it should
> know what it is doing. If there is conflicting file names and parts across
> tasks that won't even get picked up at this point. Oh and none of the
> committers ever check for this: you'll get the last file manifested (s3a) or
> renamed (file)
> If we skip the checks we will save 2 HTTP requests/file.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
