[
https://issues.apache.org/jira/browse/HADOOP-9577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13661430#comment-13661430
]
Steve Loughran commented on HADOOP-9577:
----------------------------------------
Joshua,
Tthe problems of blob store consistency is something I became more aware of
during HADOOP-8545; where delete's eventual consistency was visible between
tests cases -some of the tests would get the data of the previous run. This was
avoided by not reusing blob names, and using OpenStack Swift's {{X-NEWEST}}
header, which so far has met its promise of "returning the newest value".
Even so, the other issue with blobstores is that most don't meet Hadoop's
expectations of atomicity of directory create, rename and delete; rename is
used during the commits of MR jobs. We have expectations of a filesystem
"presents a consistent single-instance view of a hierarchical filesystem across
the cluster" that are very much broken by blobstores "a set of blobs with
variable consistency guarantees, onto which we have to mimic directories
through bulk blob operations".
This is why we're trying to spell out our expectations more clearly in
HADOOP-9361, under which I'm thinking we need to add an explicit {{Blobstore}}
interface to the blobstores to indicate -at least to MapReduce- that what
implements the FileSystem isn't really, and that we need to commit differently.
If you can help define and fix the behaviour -and verify it works- we'd love
your participation.
For now, something we should think about spelling out in more detail is "commit
to HDFS then copy to S3", and "if you do output to a blobstore, make sure
speculation is turned off".
Making sure that distCP works reliably with blobstores should be a first step
in blobstore-aware work.
Now
# was this Apache Hadoop 1.0.3 or the Amazon variant?
# was speculative execution enabled?
> Actual data loss using s3n
> --------------------------
>
> Key: HADOOP-9577
> URL: https://issues.apache.org/jira/browse/HADOOP-9577
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1.0.3
> Reporter: Joshua Caplan
> Priority: Critical
>
> The implementation of needsTaskCommit() assumes that the FileSystem used for
> writing temporary outputs is consistent. That happens not to be the case
> when using the S3 native filesystem in the US Standard region. It is
> actually quite common in larger jobs for the exists() call to return false
> even if the task attempt wrote output minutes earlier, which essentially
> cancels the commit operation with no error. That's real life data loss right
> there, folks.
> The saddest part is that the Hadoop APIs do not seem to provide any
> legitimate means for the various RecordWriters to communicate with the
> OutputCommitter. In my projects I have created a static map of semaphores
> keyed by TaskAttemptID, which all my custom RecordWriters have to be aware
> of. That's pretty lame.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
