[
https://issues.apache.org/jira/browse/HADOOP-17318?focusedWorklogId=512301&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-512301
]
ASF GitHub Bot logged work on HADOOP-17318:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 16/Nov/20 12:11
Start Date: 16/Nov/20 12:11
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2399:
URL: https://github.com/apache/hadoop/pull/2399#issuecomment-727939820
@rdblue
yes, I did a bit more than was needed because I had to also let > 1 magic
committer commit work side-by-side (all that active upload warning), and the
IDE was trying to keep me in check too, on a piece of code which hasn't been
revisited for a while.
While I had the files open in the IDE, I moved to passing FileStatus down to
line up with the changes in #2168 -if you open a file through the
JsonSerializer by passing in the FileStatus, that will be handed off to the
FileSystem's implementation of openFile(status.path).withFileStatus(status),
and so be used by S3A FS to skip the initial HEAD request. Means if we are
reading 1000 .pendingset files in S3A, we eliminate 1000 HEAD calls, which
should have tangible benefits for committers using S3 as the place to keep
those files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 512301)
Time Spent: 7h 10m (was: 7h)
> S3A committer to support concurrent jobs with same app attempt ID & dest dir
> ----------------------------------------------------------------------------
>
> Key: HADOOP-17318
> URL: https://issues.apache.org/jira/browse/HADOOP-17318
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> Reported failure of magic committer block uploads as pending upload ID is
> unknown. Likely cause: it's been aborted by another job
> # Make it possible to turn off cleanup of pending uploads in magic committer
> # log more about uploads being deleted in committers
> # and upload ID in the S3aBlockOutputStream errors
> There are other concurrency issues when you look close, see SPARK-33230
> * magic committer uses app attempt ID as path under __magic; if there are
> duplicate then they will conflict
> * staging committer local temp dir uses app attempt id
> Fix will be to have a job UUID which for spark will be picked up from the
> SPARK-33230 changes, (option to self-generate in job setup for hadoop 3.3.1+
> older spark builds); fall back to app-attempt *unless that fallback has been
> disabled*
> MR: configure to use app attempt ID
> Spark: configure to fail job setup if app attempt ID is the source of a job
> uuid
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
