[
https://issues.apache.org/jira/browse/HADOOP-9577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13661397#comment-13661397
]
Parviz Deyhim commented on HADOOP-9577:
---------------------------------------
just as a clarification, US-Standard is eventually consistent. Other S3
regions/endpoints have read-your-write consistency:
http://docs.aws.amazon.com/AmazonS3/latest/dev/LocationSelection.html
> Actual data loss using s3n
> --------------------------
>
> Key: HADOOP-9577
> URL: https://issues.apache.org/jira/browse/HADOOP-9577
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1.0.3
> Reporter: Joshua Caplan
> Priority: Critical
>
> The implementation of needsTaskCommit() assumes that the FileSystem used for
> writing temporary outputs is consistent. That happens not to be the case
> when using the S3 native filesystem in the US Standard region. It is
> actually quite common in larger jobs for the exists() call to return false
> even if the task attempt wrote output minutes earlier, which essentially
> cancels the commit operation with no error. That's real life data loss right
> there, folks.
> The saddest part is that the Hadoop APIs do not seem to provide any
> legitimate means for the various RecordWriters to communicate with the
> OutputCommitter. In my projects I have created a static map of semaphores
> keyed by TaskAttemptID, which all my custom RecordWriters have to be aware
> of. That's pretty lame.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
