[
https://issues.apache.org/jira/browse/HADOOP-17318?focusedWorklogId=511353&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-511353
]
ASF GitHub Bot logged work on HADOOP-17318:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 13/Nov/20 13:24
Start Date: 13/Nov/20 13:24
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#2399:
URL: https://github.com/apache/hadoop/pull/2399#discussion_r522947215
##########
File path:
hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
##########
@@ -1925,20 +1925,13 @@
</property>
<property>
- <name>fs.s3a.committer.staging.abort.pending.uploads</name>
+ <name>fs.s3a.committer.abort.pending.uploads</name>
<value>true</value>
<description>
- Should the staging committers abort all pending uploads to the destination
+ Should the committers abort all pending uploads to the destination
directory?
- Changing this if more than one partitioned committer is
- writing to the same destination tree simultaneously; otherwise
- the first job to complete will cancel all outstanding uploads from the
- others. However, it may lead to leaked outstanding uploads from failed
- tasks. If disabled, configure the bucket lifecycle to remove uploads
- after a time period, and/or set up a workflow to explicitly delete
- entries. Otherwise there is a risk that uncommitted uploads may run up
- bills.
+ Set to false if more than one job is writing to the same directory tree.
Review comment:
taskAbort, yet. JobAbort/cleanup is where things are more trouble,
because the job doesn't know what specific task attempts have uploaded.
with the staging committer, there's no files uploaded until task commit.
Tasks which fail before that moment don't have any pending uploads to cancel.
with the magic committer, because the files are written direct to S3, there
is more risk of pending uploads collecting.
I'm not sure about spark here, but on MR when a task is considered to have
failed, abortTask is called in the AM to abort that specific task; for the
magic committer the task's set of .pending files is determined by listing the
task attempt dir, and those operations cancelled. If that operation is called
reliably, only the current upload is pending.
Of course, if an entire job fails: no cleanup at all.
The best thing to do is simply to tell everyone to have a scheduled cleanup.
FWIW, the most leakage I see in the real world is actually from incomplete
S3ABlockOutputStream writes as again, they accrue bills. Everyone needs a
lifecycle rule to delete old ones. The sole exception there is one which our QE
team used which (unknown to them) I'd use for testing the scalability of the
"hadoop s3guard uploads" command -how well does it work when there are many,
many incomplete uploads, can it still delete them all etc. If they had a rule
then it'd screw up my test runs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 511353)
Time Spent: 6h (was: 5h 50m)
> S3A committer to support concurrent jobs with same app attempt ID & dest dir
> ----------------------------------------------------------------------------
>
> Key: HADOOP-17318
> URL: https://issues.apache.org/jira/browse/HADOOP-17318
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 6h
> Remaining Estimate: 0h
>
> Reported failure of magic committer block uploads as pending upload ID is
> unknown. Likely cause: it's been aborted by another job
> # Make it possible to turn off cleanup of pending uploads in magic committer
> # log more about uploads being deleted in committers
> # and upload ID in the S3aBlockOutputStream errors
> There are other concurrency issues when you look close, see SPARK-33230
> * magic committer uses app attempt ID as path under __magic; if there are
> duplicate then they will conflict
> * staging committer local temp dir uses app attempt id
> Fix will be to have a job UUID which for spark will be picked up from the
> SPARK-33230 changes, (option to self-generate in job setup for hadoop 3.3.1+
> older spark builds); fall back to app-attempt *unless that fallback has been
> disabled*
> MR: configure to use app attempt ID
> Spark: configure to fail job setup if app attempt ID is the source of a job
> uuid
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
