[
https://issues.apache.org/jira/browse/HDFS-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16911953#comment-16911953
]
hemanthboyina commented on HDFS-13123:
--------------------------------------
hi [~jojochuang] , thanks for the review
_You should make sure both directories on the source and destination
are snapshottable before running this tool._
we are making source and destination snapshottable , before doing any operation
{code:java}
srcFs.allowSnapshot(srcmountPath);
destFs.allowSnapshot(destFolderPath);{code}
_Probably not a good idea to hard code the snapshot name as "s1" and
"s2". Use randomly generated name instead._
will change this
_I don't understand why you create two snapshots in the source cluster
almost immediately. If you do so, you only update the files added/deleted
during the two snapshots_.
According to design in the document : "Do a first copy first, then put the
lock, and do a second copy to capture any new changes there." so created
snapshot and did disctp now data was in destination , if there any files
added/deleted during this time , we take a second snapshot and do the diff
_The state of "s1" snapshot on the source should be exactly the same as the
state of "s1" snapshot on the destination. You'll hit various strange issues if
the destination is not a mirror of source._
After using distcp , we check the return code of distcp , if the copy was not
successfull we delete the copied folder in destination ,we revert back the
changes we have done(allowsnapshot , createsnapshot) and will return (didnt
updated in the code , should modify it )
_make sure you delete the snapshots even if the prior steps hit errors._
have done that , please check in this part of code .
{code:java}
if (exitCode != 0) {
srcFs.deleteSnapshot(srcmountPath, "s1");
srcFs.disallowSnapshot(srcmountPath);
if (distcpUpdateExitCode != 0) {
srcFs.deleteSnapshot(srcmountPath, "s1");
srcFs.deleteSnapshot(srcmountPath, "s2");
srcFs.disallowSnapshot(srcmountPath);
destFs.deleteSnapshot(destFolderPath, "s1");
destFs.disallowSnapshot(destFolderPath);
destFs.delete(new Path(destpath), true);{code}
> RBF: Add a balancer tool to move data across subcluster
> --------------------------------------------------------
>
> Key: HDFS-13123
> URL: https://issues.apache.org/jira/browse/HDFS-13123
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Wei Yan
> Assignee: hemanthboyina
> Priority: Major
> Attachments: HDFS Router-Based Federation Rebalancer.pdf,
> HDFS-13123.patch
>
>
> Follow the discussion in HDFS-12615. This Jira is to track effort for
> building a rebalancer tool, used by router-based federation to move data
> among subclusters.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
