[
https://issues.apache.org/jira/browse/HADOOP-11452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14261260#comment-14261260
]
Steve Loughran commented on HADOOP-11452:
-----------------------------------------
rename is the most troublesome and inconsistent bit of the filesystem; its the
bit we're scared of.
# There's a risk that we rely over-much on the current atomicity guarantees of
HDFS, some of which are merely implementation details. Example: {{mkdirs}} is
only atomic because of lock optimisation, not by design.
# There's no guarantee that not only will any other FS support these
guarantees, but that we may not be able to support them in the future if we
ever went to more distributed metadata storage.
# Accordingly, I'd worry about exposing blindly what may just be implementation
details.
That said: atomic rename with fail-on-overwrite is a lovely way to implement
end-of-job synchronization. Things like speculative execution rely on atomicity
already, so can fail spectacularly on those filesystems whose rename operations
are both non-atomic and take O(files) or worse to complete.
Which is why {{FileContext}} APIs support rename with public options
today...its features like a more consistent rename which should motivate people
to move onto the new APIs.
accordingly, the definition of rename should be driven by those
{{FileContext}} APIs, maybe exposed in the existing {{FileSystem}} class.
Oh, and I'll be expecting updates to the HADOOP-9361 specifications of rename,
with tests. As I've noted: its the trickiest bit of filesystems, and the
specification —and I'm not sure that specification is correct.
> Revisit FileSystem#rename
> -------------------------
>
> Key: HADOOP-11452
> URL: https://issues.apache.org/jira/browse/HADOOP-11452
> Project: Hadoop Common
> Issue Type: Task
> Components: fs
> Reporter: Yi Liu
> Assignee: Yi Liu
>
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected
> and with _deprecated_ annotation. And the default implementation is not
> atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a
> good and atomic implementation. (Also an interesting thing in {{DFSClient}},
> the _deprecated_ annotations for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since
> it's atomic for rename+overwrite, also it saves RPC calls if user desires
> rename+overwrite.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
