[
https://issues.apache.org/jira/browse/HADOOP-9577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13661395#comment-13661395
]
Harsh J commented on HADOOP-9577:
---------------------------------
Thanks for the report Joshua! What would you suggest as a general solution for
this? Disable use of temporary output locations (i.e. no FOC) for S3
automatically since it can never be consistent (which is the real bother)?
> Actual data loss using s3n
> --------------------------
>
> Key: HADOOP-9577
> URL: https://issues.apache.org/jira/browse/HADOOP-9577
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1.0.3
> Reporter: Joshua Caplan
> Priority: Critical
>
> The implementation of needsTaskCommit() assumes that the FileSystem used for
> writing temporary outputs is consistent. That happens not to be the case
> when using the S3 native filesystem in the US Standard region. It is
> actually quite common in larger jobs for the exists() call to return false
> even if the task attempt wrote output minutes earlier, which essentially
> cancels the commit operation with no error. That's real life data loss right
> there, folks.
> The saddest part is that the Hadoop APIs do not seem to provide any
> legitimate means for the various RecordWriters to communicate with the
> OutputCommitter. In my projects I have created a static map of semaphores
> keyed by TaskAttemptID, which all my custom RecordWriters have to be aware
> of. That's pretty lame.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
