[jira] [Commented] (HDFS-11082) Erasure Coding : Provide replicated EC policy to just replicating the files
[ https://issues.apache.org/jira/browse/HDFS-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129764#comment-16129764 ] SammiChen commented on HDFS-11082: -- Double checked the failed UTs, not relevant. > Erasure Coding : Provide replicated EC policy to just replicating the files > --- > > Key: HDFS-11082 > URL: https://issues.apache.org/jira/browse/HDFS-11082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Rakesh R >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11082.001.patch, HDFS-11082.002.patch, > HDFS-11082.003.patch, HDFS-11082.004.patch > > > The idea of this jira is to provide a new {{replicated EC policy}} so that we > can override the EC policy on a parent directory and go back to just > replicating the files based on replication factors. > Thanks [~andrew.wang] for the > [discussions|https://issues.apache.org/jira/browse/HDFS-11072?focusedCommentId=15620743&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15620743]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11082) Erasure Coding : Provide replicated EC policy to just replicating the files
[ https://issues.apache.org/jira/browse/HDFS-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128475#comment-16128475 ] Hadoop QA commented on HDFS-11082: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 34s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 9s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 59s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 2s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 5s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 29s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 28s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 24s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}152m 13s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.net.TestDNS | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.hdfs.web.TestWebHdfsTimeouts | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 | | | hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-11082 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12882083/HDFS-11082.004.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux 2e011f342c6a 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 588c190 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apach
[jira] [Commented] (HDFS-11082) Erasure Coding : Provide replicated EC policy to just replicating the files
[ https://issues.apache.org/jira/browse/HDFS-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128174#comment-16128174 ] Andrew Wang commented on HDFS-11082: Thanks for revving Sammi, only a single nit, looks good otherwise: * We should update the help to make it clear that -policy and -replicate are optional arguments, e.g. {noformat} [-setPolicy -path [-policy ] [-replicate]] {noformat} Please file and link the follow-on JIRA for here for tracking too. Great work! > Erasure Coding : Provide replicated EC policy to just replicating the files > --- > > Key: HDFS-11082 > URL: https://issues.apache.org/jira/browse/HDFS-11082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Rakesh R >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11082.001.patch, HDFS-11082.002.patch, > HDFS-11082.003.patch > > > The idea of this jira is to provide a new {{replicated EC policy}} so that we > can override the EC policy on a parent directory and go back to just > replicating the files based on replication factors. > Thanks [~andrew.wang] for the > [discussions|https://issues.apache.org/jira/browse/HDFS-11072?focusedCommentId=15620743&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15620743]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11082) Erasure Coding : Provide replicated EC policy to just replicating the files
[ https://issues.apache.org/jira/browse/HDFS-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125506#comment-16125506 ] SammiChen commented on HDFS-11082: -- The failed UT seem not relative. > Erasure Coding : Provide replicated EC policy to just replicating the files > --- > > Key: HDFS-11082 > URL: https://issues.apache.org/jira/browse/HDFS-11082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Rakesh R >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11082.001.patch, HDFS-11082.002.patch, > HDFS-11082.003.patch > > > The idea of this jira is to provide a new {{replicated EC policy}} so that we > can override the EC policy on a parent directory and go back to just > replicating the files based on replication factors. > Thanks [~andrew.wang] for the > [discussions|https://issues.apache.org/jira/browse/HDFS-11072?focusedCommentId=15620743&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15620743]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11082) Erasure Coding : Provide replicated EC policy to just replicating the files
[ https://issues.apache.org/jira/browse/HDFS-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125484#comment-16125484 ] Hadoop QA commented on HDFS-11082: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 3s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 28s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 19s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 14s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 18s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 13s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 29s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 51s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 31s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}158m 1s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.fs.sftp.TestSFTPFileSystem | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 | | Timed out junit tests | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-11082 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881691/HDFS-11082.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux 2ff0fc751a43 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptc
[jira] [Commented] (HDFS-11082) Erasure Coding : Provide replicated EC policy to just replicating the files
[ https://issues.apache.org/jira/browse/HDFS-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125322#comment-16125322 ] SammiChen commented on HDFS-11082: -- Hi [~andrew.wang], thanks for the review. I have uploaded a new patch with all your suggests are addressed. With 003.patch, 1. user can set replication policy by `ec -setpolicy -path path -replicate` or DistributedFileSystem.setErasureCodingPolicy() with replication policy name "REPLICATION" 2. replication policy will not returned either 'ec -listPolicies` nor DistributedFileSystem.getAllErasureCodingPolicies(), or DistributedFileSystem.getErasureCodingPolicy(), or `ec -getPolicy -path path` 3. replication policy is maintained in namenode side only. It's always enabled regardless any hdfs property configuration. > Erasure Coding : Provide replicated EC policy to just replicating the files > --- > > Key: HDFS-11082 > URL: https://issues.apache.org/jira/browse/HDFS-11082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Rakesh R >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11082.001.patch, HDFS-11082.002.patch, > HDFS-11082.003.patch > > > The idea of this jira is to provide a new {{replicated EC policy}} so that we > can override the EC policy on a parent directory and go back to just > replicating the files based on replication factors. > Thanks [~andrew.wang] for the > [discussions|https://issues.apache.org/jira/browse/HDFS-11072?focusedCommentId=15620743&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15620743]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11082) Erasure Coding : Provide replicated EC policy to just replicating the files
[ https://issues.apache.org/jira/browse/HDFS-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122677#comment-16122677 ] Andrew Wang commented on HDFS-11082: Hi Sammi, thanks for working on this, sorry for the delay reviewing, * I think we should have {{getErasureCodingPolicy}} return the effective EC policy (null) for both files and directories, for ease of use. Yes, let's file a follow-on JIRA for {{getActualErasureCodingPolicy}}. * Grammar/typo in doc: "enfore the direcotry" -> "force the directory" * How hard is it to further hide the special replication policy name, e.g. just call it "REPLICATION" or better, set it with a command like "hdfs ec setPolicy -replicate" and don't expose it as a new policy? I think Rakesh recommended this before, and though worse from a code perspective, it's better from a user perspective. * If we can't do the above, it'd make the suggestion of suing some more fake looking parameters like "REPLICATION-0-0-1K" to make it clear that these parameters are meaningless. > Erasure Coding : Provide replicated EC policy to just replicating the files > --- > > Key: HDFS-11082 > URL: https://issues.apache.org/jira/browse/HDFS-11082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Rakesh R >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11082.001.patch, HDFS-11082.002.patch > > > The idea of this jira is to provide a new {{replicated EC policy}} so that we > can override the EC policy on a parent directory and go back to just > replicating the files based on replication factors. > Thanks [~andrew.wang] for the > [discussions|https://issues.apache.org/jira/browse/HDFS-11072?focusedCommentId=15620743&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15620743]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11082) Erasure Coding : Provide replicated EC policy to just replicating the files
[ https://issues.apache.org/jira/browse/HDFS-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122011#comment-16122011 ] Hadoop QA commented on HDFS-11082: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color} | {color:red} HDFS-11082 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HDFS-11082 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12880598/HDFS-11082.002.patch | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20637/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Erasure Coding : Provide replicated EC policy to just replicating the files > --- > > Key: HDFS-11082 > URL: https://issues.apache.org/jira/browse/HDFS-11082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Rakesh R >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11082.001.patch, HDFS-11082.002.patch > > > The idea of this jira is to provide a new {{replicated EC policy}} so that we > can override the EC policy on a parent directory and go back to just > replicating the files based on replication factors. > Thanks [~andrew.wang] for the > [discussions|https://issues.apache.org/jira/browse/HDFS-11072?focusedCommentId=15620743&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15620743]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11082) Erasure Coding : Provide replicated EC policy to just replicating the files
[ https://issues.apache.org/jira/browse/HDFS-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121374#comment-16121374 ] SammiChen commented on HDFS-11082: -- I uploaded the 002.patch 3 days ago, while the build still not come out so far. Ping [~drankye] and [~andrew.wang], could you help to check it? > Erasure Coding : Provide replicated EC policy to just replicating the files > --- > > Key: HDFS-11082 > URL: https://issues.apache.org/jira/browse/HDFS-11082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Rakesh R >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11082.001.patch, HDFS-11082.002.patch > > > The idea of this jira is to provide a new {{replicated EC policy}} so that we > can override the EC policy on a parent directory and go back to just > replicating the files based on replication factors. > Thanks [~andrew.wang] for the > [discussions|https://issues.apache.org/jira/browse/HDFS-11072?focusedCommentId=15620743&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15620743]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11082) Erasure Coding : Provide replicated EC policy to just replicating the files
[ https://issues.apache.org/jira/browse/HDFS-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116155#comment-16116155 ] SammiChen commented on HDFS-11082: -- Hi [~andrew.wang], I have rethought your suggest about add another API, such as "getActualECPolicy" or "getEffectiveErasureCodingPolicy", to return "null" for both directory & file which has replication EC policy set, would it be helpful? > Erasure Coding : Provide replicated EC policy to just replicating the files > --- > > Key: HDFS-11082 > URL: https://issues.apache.org/jira/browse/HDFS-11082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Rakesh R >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11082.001.patch, HDFS-11082.002.patch > > > The idea of this jira is to provide a new {{replicated EC policy}} so that we > can override the EC policy on a parent directory and go back to just > replicating the files based on replication factors. > Thanks [~andrew.wang] for the > [discussions|https://issues.apache.org/jira/browse/HDFS-11072?focusedCommentId=15620743&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15620743]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11082) Erasure Coding : Provide replicated EC policy to just replicating the files
[ https://issues.apache.org/jira/browse/HDFS-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113908#comment-16113908 ] Hadoop QA commented on HDFS-11082: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 25s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 3s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 33s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 46s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 4s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 32s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 32s{color} | {color:red} root generated 2 new + 1418 unchanged - 0 fixed = 1420 total (was 1418) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 50s{color} | {color:orange} root: The patch generated 1 new + 40 unchanged - 0 fixed = 41 total (was 40) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 3s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 17s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 26s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 56s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 29s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}147m 45s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.net.TestDNS | | | hadoop.hdfs.TestErasureCodingPolicies | | | hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy | | | hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy | | | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy | | | hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.hdfs.web.TestWebHdfsTimeouts | | | hadoop.hdfs.TestClientProtocolForPipelineRecovery | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:
[jira] [Commented] (HDFS-11082) Erasure Coding : Provide replicated EC policy to just replicating the files
[ https://issues.apache.org/jira/browse/HDFS-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113829#comment-16113829 ] SammiChen commented on HDFS-11082: -- Thanks [~andrew.wang] for the quick review! I just realized that document is not updated, will update it later. {quote} Also need to think about the behavior of getErasureCodingPolicy. Right now it returns "null" to mean replication. With this patch, a user would have to check both for "null" and "replication-1-2-64K" to know if it's replicated. It'd be good to choose one or the other to make it simpler for downstreams. "null" would be more compatible, and it'd hide the special replicated EC policy from non-admin users which I like. {quote} Currently, replication policy can only be set on directory, not the file. Because currently in file header format, replication factor and ec policy ID share the same bits. So a file can be either traditional replication or effective EC, cannot have replication EC policy. For getErasureCodingPolicy on directory, return "null" or "replication-1-2-64k", both have pros and cons. If return "null" for replication EC policy, Pros: 1. It's easy for downstream applications to check it is effectively EC or replication Cons: 1. after set replication EC policy on directory, it cannot be get back, so there is no way to unset the policy or aware of the policy from user's point of view. User cannot distinguish a traditional replication directory and an replication EC policy directory. If return "replication-1-2-64k", the pros and cons are reversed. So it's a style choice, one is give all information to user and let them decide, another is handle it internally on behalf of user. I'm prone to give all information to user. But I'm OK to go "null" solution if it's for sure will add more benefit to users. I think you have more experience on this. You make the call. {quote} This is not directly related (and I think we discussed this a bit on another JIRA) but I'm not happy with our getECPolicy API right now. Right now it returns the effective EC policy. Without being able to query the actual EC policy, the behavior when setting/unsetting is kind of tricky. Should we add an "getActualECPolicy" API? Can be a follow-on JIRA. {quote} Do you refer to {{getErasureCodingPolicy}} when you say {{getECPolicy}}? I'm kind of forget when we have discussed this issue. Can you give more hints? The suggestions in all other comments will be addressed in next patch. > Erasure Coding : Provide replicated EC policy to just replicating the files > --- > > Key: HDFS-11082 > URL: https://issues.apache.org/jira/browse/HDFS-11082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Rakesh R >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11082.001.patch > > > The idea of this jira is to provide a new {{replicated EC policy}} so that we > can override the EC policy on a parent directory and go back to just > replicating the files based on replication factors. > Thanks [~andrew.wang] for the > [discussions|https://issues.apache.org/jira/browse/HDFS-11072?focusedCommentId=15620743&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15620743]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11082) Erasure Coding : Provide replicated EC policy to just replicating the files
[ https://issues.apache.org/jira/browse/HDFS-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113552#comment-16113552 ] Andrew Wang commented on HDFS-11082: Hi Sammi, this looks good overall, thanks for working on this! A few review comments: * We should add documentation and javadocs describing this new special policy so users and admins can be aware * Also need to think about the behavior of {{getErasureCodingPolicy}}. Right now it returns "null" to mean replication. With this patch, a user would have to check both for "null" and "replication-1-2-64K" to know if it's replicated. It'd be good to choose one or the other to make it simpler for downstreams. "null" would be more compatible, and it'd hide the special replicated EC policy from non-admin users which I like. * Please add messages to the asserts in the tests to help with later debugging * Is this policy enabled by default? I think it should be if not. * Would be nice to rename the paths in the test cases to be more descriptive. As an example, right now we have: {code} 723 final Path rootPath = new Path("/striped"); 724 final Path childPath = new Path(rootPath, "replica"); 725 final Path subChildPath = new Path(childPath, "replica"); 726 final Path filePath = new Path(childPath, "file"); 727 final Path filePath2 = new Path(subChildPath, "file"); {code} Instead, perhaps something more like: {code} 723 final Path rootPath = new Path("/striped"); 724 final Path replicaPath = new Path(rootPath, "replica"); 725 final Path subReplicaPath = new Path(replicaPath, "subreplica"); 726 final Path replicaFilePath = new Path(replicaPath, "file"); 727 final Path subReplicaFilePath = new Path(subReplicaPath, "file"); {code} This is not directly related (and I think we discussed this a bit on another JIRA) but I'm not happy with our getECPolicy API right now. Right now it returns the effective EC policy. Without being able to query the actual EC policy, the behavior when setting/unsetting is kind of tricky. Should we add an "getActualECPolicy" API? Can be a follow-on JIRA. If you don't mind, one immediate improvement we could make is documenting in the {{getECPolicy}} javadoc that it returns the effective EC policy. > Erasure Coding : Provide replicated EC policy to just replicating the files > --- > > Key: HDFS-11082 > URL: https://issues.apache.org/jira/browse/HDFS-11082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Rakesh R >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11082.001.patch > > > The idea of this jira is to provide a new {{replicated EC policy}} so that we > can override the EC policy on a parent directory and go back to just > replicating the files based on replication factors. > Thanks [~andrew.wang] for the > [discussions|https://issues.apache.org/jira/browse/HDFS-11072?focusedCommentId=15620743&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15620743]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11082) Erasure Coding : Provide replicated EC policy to just replicating the files
[ https://issues.apache.org/jira/browse/HDFS-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111801#comment-16111801 ] Andrew Wang commented on HDFS-11082: Seems similar to what we discussed earlier, SGTM! > Erasure Coding : Provide replicated EC policy to just replicating the files > --- > > Key: HDFS-11082 > URL: https://issues.apache.org/jira/browse/HDFS-11082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Rakesh R >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > > The idea of this jira is to provide a new {{replicated EC policy}} so that we > can override the EC policy on a parent directory and go back to just > replicating the files based on replication factors. > Thanks [~andrew.wang] for the > [discussions|https://issues.apache.org/jira/browse/HDFS-11072?focusedCommentId=15620743&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15620743]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11082) Erasure Coding : Provide replicated EC policy to just replicating the files
[ https://issues.apache.org/jira/browse/HDFS-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110201#comment-16110201 ] SammiChen commented on HDFS-11082: -- Hi [~andrew.wang], I'm preparing the patch. For how to provide replicated EC policy, here is my overall thoughts, 1. replicated EC policy is one of system built-in policy, like other built-in policies, such as RS-6-3-64k, it can be listed, enabled, disabled, but cannot be removed. 2. replicated EC policy will have codec name "replication" with (1,1) parameter combination 3. replicated EC policy can be set, unset, get from directory & file 4. replicated EC policy should be treated specially when allocate block and DFSClient side data read/write I'd like to hear your opinions. > Erasure Coding : Provide replicated EC policy to just replicating the files > --- > > Key: HDFS-11082 > URL: https://issues.apache.org/jira/browse/HDFS-11082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Rakesh R >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > > The idea of this jira is to provide a new {{replicated EC policy}} so that we > can override the EC policy on a parent directory and go back to just > replicating the files based on replication factors. > Thanks [~andrew.wang] for the > [discussions|https://issues.apache.org/jira/browse/HDFS-11072?focusedCommentId=15620743&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15620743]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11082) Erasure Coding : Provide replicated EC policy to just replicating the files
[ https://issues.apache.org/jira/browse/HDFS-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108275#comment-16108275 ] Andrew Wang commented on HDFS-11082: Hi [~Sammi] any progress on this one? > Erasure Coding : Provide replicated EC policy to just replicating the files > --- > > Key: HDFS-11082 > URL: https://issues.apache.org/jira/browse/HDFS-11082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Rakesh R >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > > The idea of this jira is to provide a new {{replicated EC policy}} so that we > can override the EC policy on a parent directory and go back to just > replicating the files based on replication factors. > Thanks [~andrew.wang] for the > [discussions|https://issues.apache.org/jira/browse/HDFS-11072?focusedCommentId=15620743&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15620743]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org