[ https://issues.apache.org/jira/browse/HDFS-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16911953#comment-16911953 ]
hemanthboyina commented on HDFS-13123: -------------------------------------- hi [~jojochuang] , thanks for the review _You should make sure both directories on the source and destination are snapshottable before running this tool._ we are making source and destination snapshottable , before doing any operation {code:java} srcFs.allowSnapshot(srcmountPath); destFs.allowSnapshot(destFolderPath);{code} _Probably not a good idea to hard code the snapshot name as "s1" and "s2". Use randomly generated name instead._ will change this _I don't understand why you create two snapshots in the source cluster almost immediately. If you do so, you only update the files added/deleted during the two snapshots_. According to design in the document : "Do a first copy first, then put the lock, and do a second copy to capture any new changes there." so created snapshot and did disctp now data was in destination , if there any files added/deleted during this time , we take a second snapshot and do the diff _The state of "s1" snapshot on the source should be exactly the same as the state of "s1" snapshot on the destination. You'll hit various strange issues if the destination is not a mirror of source._ After using distcp , we check the return code of distcp , if the copy was not successfull we delete the copied folder in destination ,we revert back the changes we have done(allowsnapshot , createsnapshot) and will return (didnt updated in the code , should modify it ) _make sure you delete the snapshots even if the prior steps hit errors._ have done that , please check in this part of code . {code:java} if (exitCode != 0) { srcFs.deleteSnapshot(srcmountPath, "s1"); srcFs.disallowSnapshot(srcmountPath); if (distcpUpdateExitCode != 0) { srcFs.deleteSnapshot(srcmountPath, "s1"); srcFs.deleteSnapshot(srcmountPath, "s2"); srcFs.disallowSnapshot(srcmountPath); destFs.deleteSnapshot(destFolderPath, "s1"); destFs.disallowSnapshot(destFolderPath); destFs.delete(new Path(destpath), true);{code} > RBF: Add a balancer tool to move data across subcluster > -------------------------------------------------------- > > Key: HDFS-13123 > URL: https://issues.apache.org/jira/browse/HDFS-13123 > Project: Hadoop HDFS > Issue Type: Improvement > Reporter: Wei Yan > Assignee: hemanthboyina > Priority: Major > Attachments: HDFS Router-Based Federation Rebalancer.pdf, > HDFS-13123.patch > > > Follow the discussion in HDFS-12615. This Jira is to track effort for > building a rebalancer tool, used by router-based federation to move data > among subclusters. -- This message was sent by Atlassian Jira (v8.3.2#803003) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org