[ https://issues.apache.org/jira/browse/HDFS-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15543686#comment-15543686 ]
Andrew Wang commented on HDFS-9820: ----------------------------------- Hi Yongjun, thanks for sticking with this one. I had a few meta comments, and some code comments: Meta comments: * I see a TODO about HDFS-10263. How much would it simplify the code here if we implemented that first? There's a lot of logic about source vs. target due to how we flip it for the rdiff, and overall not much code sharing with the existing snapshot diff mechanism. * Is there a situation where we'd want to pass two different clusters for "src" and "tgt" to rdiff? They need to both have identical "s1" base states anyway. Code comments: * Comment in DistCpOptions says that forward diff is referred to as "Fdiff" but it's still just "diff" in multiple places. Maybe change to say "referred to as diff or Fdiff"? * Unused Preconditions import in DistCpOptions DistCpSync: * Extra "//" line on diffMap. I'd also prefer if we made this a javadoc block comment ({{/** */}}). * {{preSyncCheck}} could use some explanatory comments about how we flip the diff for rdiff. I don't know what the "c" in "cfs" stands for also, so could use a comment as well. Also, is there a way to additionally dedupe the rdiff/diff checking on the modtime? If not, cfs/cdir aren't saving us much and might as well just put them in the corresponding part of the if/else block. * We lost this comment in sync: {noformat} // TODO: since we have tmp directory, we can support "undo" with failures // set the source path using the snapshot path {noformat} * getRenameAndDeleteDiffsRdiff, I think we mean "reversal" rather than "reversion", and "Reversed" rather than "Reverted" in {{renameDiffsListReverted}} * TestDistCpSync: what happened to testFallback? New syncAndFail also isn't used. I didn't go through all the new tests, will get to that later. > Improve distcp to support efficient restore to an earlier snapshot > ------------------------------------------------------------------ > > Key: HDFS-9820 > URL: https://issues.apache.org/jira/browse/HDFS-9820 > Project: Hadoop HDFS > Issue Type: New Feature > Components: distcp > Reporter: Yongjun Zhang > Assignee: Yongjun Zhang > Attachments: HDFS-9820.001.patch, HDFS-9820.002.patch, > HDFS-9820.003.patch, HDFS-9820.004.patch, HDFS-9820.005.patch > > > A common use scenario (scenaio 1): > # create snapshot sx in clusterX, > # do some experiemnts in clusterX, which creates some files. > # throw away the files changed and go back to sx. > Another scenario (scenario 2) is, there is a production cluster and a backup > cluster, we periodically sync up the data from production cluster to the > backup cluster with distcp. > The cluster in scenario 1 could be the backup cluster in scenario 2. > For scenario 1: > HDFS-4167 intends to restore HDFS to the most recent snapshot, and there are > some complexity and challenges. Before that jira is implemented, we count on > distcp to copy from snapshot to the current state. However, the performance > of this operation could be very bad because we have to go through all files > even if we only changed a few files. > For scenario 2: > HDFS-7535 improved distcp performance by avoiding copying files that changed > name since last backup. > On top of HDFS-7535, HDFS-8828 improved distcp performance when copying data > from source to target cluster, by only copying changed files since last > backup. The way it works is use snapshot diff to find out all files changed, > and copy the changed files only. > See > https://blog.cloudera.com/blog/2015/12/distcp-performance-improvements-in-apache-hadoop/ > This jira is to propose a variation of HDFS-8828, to find out the files > changed in target cluster since last snapshot sx, and copy these from > snapshot sx of either the source or the target cluster, to restore target > cluster's current state to sx. > Specifically, > If a file/dir is > - renamed, rename it back > - created in target cluster, delete it > - modified, put it to the copy list > - run distcp with the copy list, copy from the source cluster's corresponding > snapshot > This could be a new command line switch -rdiff in distcp. > As a native restore feature, HDFS-4167 would still be ideal to have. However, > HDFS-9820 would hopefully be easier to implement, before HDFS-4167 is in > place. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org