[
https://issues.apache.org/jira/browse/HADOOP-9184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13568935#comment-13568935
]
Steve Loughran commented on HADOOP-9184:
----------------------------------------
The patch won't apply as it's in the branch 1 layout and build, trunk is split
up and Mavenized. If you could do it for there and try and reduce the odd
spurious change, that would be good.
I see three possible causes
# you've hit eventual consistency at play in S3.
* Which S3 location are you working with? US-default?
* Have a go on US-west and see if replicates there.
# There's just a race condition between {{isFile()}} (which calls
{{getFileStatus()}} and the {{getFileStatus()}} in the following condition.
I'd switch to a single {{getFileStatus()}} up front to eliminate that race, and
eliminate the cost of another round trip (which is more expensive to remote
filesystems)
# Atomicity: {{mv}} and {{rm -rf}} aren't atomic on blobstores. We know about
this and do need to come up with a better solution at the end of MR jobs for
filesystems that declare they are blobstores.
Steve
> Some reducers failing to write final output file to s3.
> -------------------------------------------------------
>
> Key: HADOOP-9184
> URL: https://issues.apache.org/jira/browse/HADOOP-9184
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.20.2
> Reporter: Jeremy Karn
> Attachments: example.pig, HADOOP-9184-branch-0.20.patch,
> hadoop-9184.patch, task_log.txt
>
>
> We had a Hadoop job that was running 100 reducers with most of the reducers
> expected to write out an empty file. When the final output was to an S3
> bucket we were finding that sometimes we were missing a final part file.
> This was happening approximately 1 job in 3 (so approximately 1 reducer out
> of 300 was failing to output the data properly). I've attached the pig script
> we were using to reproduce the bug.
> After an in depth look and instrumenting the code we traced the problem to
> moveTaskOutputs in FileOutputCommitter.
> The code there looked like:
> {code}
> if (fs.isFile(taskOutput)) {
> … do stuff …
> } else if(fs.getFileStatus(taskOutput).isDir()) {
> … do stuff …
> }
> {code}
> And what we saw happening is that for the problem jobs neither path was being
> exercised. I've attached the task log of our instrumented code. In this
> version we added an else statement and printed out the line "THIS SEEMS LIKE
> WE SHOULD NEVER GET HERE …".
> The root cause of this seems to be an eventual consistency issue with S3.
> You can see in the log that the first time moveTaskOutputs is called it finds
> that the taskOutput is a directory. It goes into the isDir() branch and
> successfully retrieves the list of files in that directory from S3 (in this
> case just one file). This triggers a recursive call to moveTaskOutputs for
> the file found in the directory. But in this pass through moveTaskOutput the
> temporary output file can't be found resulting in both branches of the above
> if statement being skipped and the temporary file never being moved to the
> final output location.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
