[
https://issues.apache.org/jira/browse/HADOOP-901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12466364
]
Tom White commented on HADOOP-901:
----------------------------------
I've attached a patch which does some additional checks and uses a new method
listDeepSubPaths to retrieve all subpaths in one S3 operation. It includes unit
tests.
Note that, unlike Mike's code, a rename will fail (return false) if the parent
directory of the destination does not exist, which I believe is consistent with
HDFS. Mike - could you check that this patch works for your use case please?
> Make S3FileSystem do recursive renames
> --------------------------------------
>
> Key: HADOOP-901
> URL: https://issues.apache.org/jira/browse/HADOOP-901
> Project: Hadoop
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.10.1
> Reporter: Tom White
> Attachments: hadoop-901.patch
>
>
> From Mike Smith:
> I went through the S3FileSystem.java codes and fixed the renameRaw() method.
> Now, it iterates through the folders recursively and rename those. Also, in
> the case of existing destination folder, it moves the src folder under the
> dst folder.
> Here is the piece code that should be replaced in S3FileSystem.java.
> renameRaw() method should be replaced by the following methods:
> @Override
> public boolean renameRaw(Path src, Path dst) throws IOException {
> Path absoluteDst = makeAbsolute(dst);
> Path absoluteSrc = makeAbsolute(src);
> INode inode = store.getINode(absoluteDst);
> // checking to see of dst folder exist. In this case moves the
> // src folder under the existing path.
> if (inode != null && inode.isDirectory()) {
> Path newDst = new Path(absoluteDst.toString
> ()+"/"+absoluteSrc.getName());
> return renameRaw(src,newDst,src);
> } else {
> // if the dst folder does not exist, then the dst folder will be created.
> return renameRaw(src,dst,src);
> }
> }
> // recursively goes through all the subfolders and rename those.
> public boolean renameRaw(Path src, Path dst,Path orgSrc) throws
> IOException {
> Path absoluteSrc = makeAbsolute(src);
> Path newDst = new Path(src.toString().replaceFirst(orgSrc.toString(),
> dst.toString()));
> Path absoluteDst = makeAbsolute(newDst);
> LOG.info(absoluteSrc.toString());
> INode inode = store.getINode (absoluteSrc);
> if (inode == null) {
> return false;
> }
> if (inode.isFile()) {
> store.storeINode(makeAbsolute(absoluteDst), inode);
> } else {
> store.storeINode (makeAbsolute(absoluteDst), inode);
> Path[] contents = listPathsRaw(absoluteSrc);
> if (contents == null) {
> return false;
> }
> for (Path p : contents) {
> if (! renameRaw(p,dst,orgSrc)) {
> return false;
> }
> }
> }
> store.deleteINode(absoluteSrc);
> return true;
> }
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
