[
https://issues.apache.org/jira/browse/HADOOP-9371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599411#comment-13599411
]
Matthew Farrellee commented on HADOOP-9371:
-------------------------------------------
Page 2, Concurrency, you mention "mkdir/mkdirs is atomic"
It seems reasonable that mkdir is atomic.
I've been researching mkdirs(), with a focus on idempotence and atomicity.
ClientProtocol.java:mkdirs() clearly labels it as @Idempotent, and the
documentation and various implementations support that claim. It's also a
property that is relatively straight-forward to implement on many back-end
filesystems.
I'm having more difficulty tracking down the atomicity of mkdirs(). The LocalFS
implementations are not themselves atomic. I tracked the HDFS implementation
back to FSNamesystem.java:mkdirsInt(), which appears to provide an atomic
implementation. However, the atomic nature of mkdirsInt() appears to come from
HDFS-988, which looks to fix a bug by making mkdirs() atomic rather having an
explicit purpose of making mkdirs() atomic by design.
How are you getting to mkdirs() as atomic?
A mild concern of mine is that even if mkdirs() isn't atomic by design, for
HDFS it has been implemented as atomic and who knows who may silently be
relying on the not-by-design atomic property. That said, given mkdirs() is
idempotent it isn't suitable for use as a locking mechanism.
> Define Semantics of FileSystem and FileContext more rigorously
> --------------------------------------------------------------
>
> Key: HADOOP-9371
> URL: https://issues.apache.org/jira/browse/HADOOP-9371
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
> Affects Versions: 1.2.0, 3.0.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HadoopFilesystemContract.pdf
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The semantics of {{FileSystem}} and {{FileContext}} are not completely
> defined in terms of
> # core expectations of a filesystem
> # consistency requirements.
> # concurrency requirements.
> # minimum scale limits
> Furthermore, methods are not defined strictly enough in terms of their
> outcomes and failure modes.
> The requirements and method semantics should be defined more strictly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
