[
https://issues.apache.org/jira/browse/HADOOP-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430412#comment-16430412
]
Steve Loughran commented on HADOOP-15361:
-----------------------------------------
As the docs say, "nobody really understands rename"
Some of the stores (ADL I think) probably do raise an exception on source not
found, Lookingat S3A it catches and downgrades to a fialsoe.
# if the current RawLocal raises an FNFE then it's what people using file://
expect: leave as is.
# if it currently copies HDFS and returns false, do the same.
bq. filesystem.md just states what is happening, these behaviors are not
intended.
It tries to document what HDFS does, based on review and experiments, on the
basis that it's the reference implementation; but the local FS is the RI of
what Posix does.
in testing.md:
bq. "Look at the implementations of a class/interface/method to see what they
do, especially HDFS and local. These are the documentation of what is done
today."
The fact that rename() doesn't fail on many problems is a flaw in the API, one
you can see if you look through applications to see how it's generally used,
usually something like
{code}
if(!rename(src, dest)} { throw new IOException("Rename failed"); }
{code}
That is: an exception is eventually raised —but one with no meaningful
information.
The {{FileSystem.rename(path, path, options)}} operation used by FileContext
does throw exceptions on all failures, which is why HADOOP-11452 pushes for it
to be specified tested and made public: so people can use it. And, because its
out there, easy to backpport. Have a look at that to see where we could go in
future.
Let's see what [~sanjay.radia] has to say on this topic.
> RawLocalFileSystem should use Java nio framework for rename
> -----------------------------------------------------------
>
> Key: HADOOP-15361
> URL: https://issues.apache.org/jira/browse/HADOOP-15361
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Andras Bokor
> Assignee: Andras Bokor
> Priority: Major
> Labels: incompatibleChange
> Attachments: HADOOP-15361.01.patch
>
>
> Currently RawLocalFileSystem uses a fallback logic for cross-volume renames.
> The fallback logic is a copy-on-fail logic so when rename fails it copies the
> source then delete it.
> An additional fallback logic was needed for Windows to provide POSIX rename
> behavior.
> Due to the fallback logic RawLocalFileSystem does not pass the contract tests
> (HADOOP-13082).
> With using Java nio framework both could be eliminated since it is not
> platform dependent and provides cross-volume rename.
> In addition the fallback logic for Windows is not correct since Java io
> overrides the destination only if the source is also a directory but
> handleEmptyDstDirectoryOnWindows method checks only the destination. That
> means rename allows to override a directory with a file on Windows but not on
> Unix.
> File#renameTo and Files#move are not 100% compatible:
> If the source is a directory and the destination is an empty directory
> File#renameTo overrides the source but Files#move is does not. We have to use
> {{StandardCopyOption.REPLACE_EXISTING}} but it overrides the destination even
> if the source or the destination is a file. So to make them compatible we
> have to check that the either the source or the destination is a directory
> before we add the copy option.
> I think the correct strategy is
> * Where the contract test passed so far it should pass after this
> * Where the contract test failed because of Java specific think and not
> because of the fallback logic we should keep the original behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
