[jira] [Commented] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849097#comment-16849097 ] Bharat Viswanadham commented on HDDS-700: - This has caused some UT failures. Reported HDDS-1559 to fix this. I think our Jenkins run is not properly running UT's for all modules. I feel it is better to use the PR model for HDDS jiras. As for PR's we have another CI which run's UT's for all modules and also smoke tests. > Support rack awared node placement policy based on network topology > --- > > Key: HDDS-700 > URL: https://issues.apache.org/jira/browse/HDDS-700 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Fix For: 0.4.1 > > Attachments: HDDS-700.01.patch, HDDS-700.02.patch, HDDS-700.03.patch > > > Implement a new container placement policy implementation based datanode's > network topology. It follows the same rule as HDFS. > By default with 3 replica, two replica will be on the same rack, the third > replica and all the remaining replicas will be on different racks. > > {color:#808080} {color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847244#comment-16847244 ] Supratim Deka commented on HDDS-700: looks like the checkstyle issues reported in patch 03 slipped by and made it into the commit. > Support rack awared node placement policy based on network topology > --- > > Key: HDDS-700 > URL: https://issues.apache.org/jira/browse/HDDS-700 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Fix For: 0.4.1 > > Attachments: HDDS-700.01.patch, HDDS-700.02.patch, HDDS-700.03.patch > > > Implement a new container placement policy implementation based datanode's > network topology. It follows the same rule as HDFS. > By default with 3 replica, two replica will be on the same rack, the third > replica and all the remaining replicas will be on different racks. > > {color:#808080} {color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847188#comment-16847188 ] Sammi Chen commented on HDDS-700: - bq. Thanks [~xyao] for helping review and commit the patch. {quote}DatanodeDetails.java Line 357: if the network topology has additional layers above RACK, should we consider a more generic default network location?{quote} I thought the same. It would better to get the default network location from NetworkTopologyInstance. But that means NetworkTopologyInstance need to passed in as a parameter when instantiating a DatanodeDetails object. Will think about how to refactor this part in follow JIRAs. > Support rack awared node placement policy based on network topology > --- > > Key: HDDS-700 > URL: https://issues.apache.org/jira/browse/HDDS-700 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Fix For: 0.4.1 > > Attachments: HDDS-700.01.patch, HDDS-700.02.patch, HDDS-700.03.patch > > > Implement a new container placement policy implementation based datanode's > network topology. It follows the same rule as HDFS. > By default with 3 replica, two replica will be on the same rack, the third > replica and all the remaining replicas will be on different racks. > > {color:#808080} {color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846931#comment-16846931 ] Hudson commented on HDDS-700: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16595 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16595/]) HDDS-700. Support rack awared node placement policy based on network (xyao: rev 20a4ec351c51da3459423852abea1d6c0e3097e3) * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementCapacity.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestReplicationManager.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementRandom.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java * (add) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/TestSCMContainerPlacementRackAware.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/TestUtils.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMCommonPolicy.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/TestSCMContainerPlacementCapacity.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/TestSCMContainerPlacementRandom.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/ozone/container/placement/TestContainerPlacement.java * (add) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementRackAware.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ReplicationManager.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/ContainerPlacementPolicy.java > Support rack awared node placement policy based on network topology > --- > > Key: HDDS-700 > URL: https://issues.apache.org/jira/browse/HDDS-700 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-700.01.patch, HDDS-700.02.patch, HDDS-700.03.patch > > > Implement a new container placement policy implementation based datanode's > network topology. It follows the same rule as HDFS. > By default with 3 replica, two replica will be on the same rack, the third > replica and all the remaining replicas will be on different racks. > > {color:#808080} {color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846863#comment-16846863 ] Xiaoyu Yao commented on HDDS-700: - Thanks [~Sammi] for the update. +1 for v3 patch. I will commit it shortly. I also have a minor suggestion that we could address in follow up JIRA. DatanodeDetails.java Line 357: if the network topology has additional layers above RACK, should we consider a more generic default network location? > Support rack awared node placement policy based on network topology > --- > > Key: HDDS-700 > URL: https://issues.apache.org/jira/browse/HDDS-700 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-700.01.patch, HDDS-700.02.patch, HDDS-700.03.patch > > > Implement a new container placement policy implementation based datanode's > network topology. It follows the same rule as HDFS. > By default with 3 replica, two replica will be on the same rack, the third > replica and all the remaining replicas will be on different racks. > > {color:#808080} {color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846782#comment-16846782 ] Hadoop QA commented on HDDS-700: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 37s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 1s{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 6 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 53s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 2s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 44s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 4m 55s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m 2s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 54s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 49s{color} | {color:orange} hadoop-hdds: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 33s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m 22s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 51s{color} | {color:red} hadoop-hdds in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 23s{color} | {color:red} hadoop-ozone in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 48s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}104m 2s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.ozone.client.rpc.TestWatchForCommit | | | hadoop.ozone.container.common.impl.TestContainerPersistence | | | hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/PreCommit-HDDS-Build/2706/artifact/out/Dockerfile | | JIRA Issue | HDDS-700 | | JIRA Patch URL |
[jira] [Commented] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846721#comment-16846721 ] Sammi Chen commented on HDDS-700: - 03.patch, correct check styles. > Support rack awared node placement policy based on network topology > --- > > Key: HDDS-700 > URL: https://issues.apache.org/jira/browse/HDDS-700 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-700.01.patch, HDDS-700.02.patch, HDDS-700.03.patch > > > Implement a new container placement policy implementation based datanode's > network topology. It follows the same rule as HDFS. > By default with 3 replica, two replica will be on the same rack, the third > replica and all the remaining replicas will be on different racks. > > {color:#808080} {color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844677#comment-16844677 ] Hadoop QA commented on HDDS-700: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 32s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 1s{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 6 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 35s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 46s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 31s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 4m 53s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m 5s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 24s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 40s{color} | {color:orange} hadoop-hdds: The patch generated 17 new + 0 unchanged - 0 fixed = 17 total (was 0) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 33s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m 14s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 37s{color} | {color:red} hadoop-hdds in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 54s{color} | {color:red} hadoop-ozone in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 56s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 97m 51s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdds.scm.pipeline.TestRatisPipelineProvider | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/PreCommit-HDDS-Build/2701/artifact/out/Dockerfile | | JIRA Issue | HDDS-700 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12969237/HDDS-700.02.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | |
[jira] [Commented] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844609#comment-16844609 ] Sammi Chen commented on HDDS-700: - 02.patch, here is the change list, 1. add logic to avoid infinite loop when choosing data 2. refine ContainerPlacementPolicy interface, add favoredNodes to chooseNode API. User can specify referred nodes in this parameter. It's a hint to the allocator. Nodes in favoredNodes list might not be chosen if they don't meet the requirement. 3. add more test cases 4. add "boolean fallback" parameter in class constructor to support both pipeline placement(no fallback) and closed container placement(fallback). 5. addressed all other [~xyao]'s comments. > Support rack awared node placement policy based on network topology > --- > > Key: HDDS-700 > URL: https://issues.apache.org/jira/browse/HDDS-700 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-700.01.patch, HDDS-700.02.patch > > > Implement a new container placement policy implementation based datanode's > network topology. It follows the same rule as HDFS. > By default with 3 replica, two replica will be on the same rack, the third > replica and all the remaining replicas will be on different racks. > > {color:#808080} {color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844527#comment-16844527 ] Siddharth Wagle commented on HDDS-700: -- Thanks [~Sammi], that does make sense. But can a simple change in the current patch to take in a configuration for the ancestor gen jump, would actually enable multiple topologies without new implementations? > Support rack awared node placement policy based on network topology > --- > > Key: HDDS-700 > URL: https://issues.apache.org/jira/browse/HDDS-700 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-700.01.patch > > > Implement a new container placement policy implementation based datanode's > network topology. It follows the same rule as HDFS. > By default with 3 replica, two replica will be on the same rack, the third > replica and all the remaining replicas will be on different racks. > > {color:#808080} {color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16843742#comment-16843742 ] Sammi Chen commented on HDDS-700: - [~swagle], the short answer is "separate NG". My initial target here is to provide a HDFS behavior compatible placement policy. So the network topology likes "/r1/n". Or even If the topology likes "/d1/switch1/r1/n1", this implementation also works. Essentially, this implementation provides the capability to allocate datanode with only leaf's parent involved. In current customizable network topology, admin can use topology with any hierarchy levels. For your example "/d1/r1/ng/n", I think the desired placement policy will be put the first and second replica on different ng on same rack, and the third replica on a different ng on a different rack. So if the requirement is to consider crossing 2 ancestor levels(leaf's parent and grandparent) , there should be a new placement policy implementation here. Likewise, if the requirement is to cross 3 ancestor levels, then another new placement policy implementation is needed. For this implementation, I will add a a topology layer check during initialization. It applies to 3 layers network topology "/r1/n". > Support rack awared node placement policy based on network topology > --- > > Key: HDDS-700 > URL: https://issues.apache.org/jira/browse/HDDS-700 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-700.01.patch > > > Implement a new container placement policy implementation based datanode's > network topology. It follows the same rule as HDFS. > By default with 3 replica, two replica will be on the same rack, the third > replica and all the remaining replicas will be on different racks. > > {color:#808080} {color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842560#comment-16842560 ] Siddharth Wagle commented on HDDS-700: -- [~Sammi] If the topology is modeled as "*/d1/r1/ng/n*", will the cross rack logic still work for replcas 3 and above? The current logic will choose the second replica on the same NG but will the thrid replica correctly land on a separate rack and not just NG? > Support rack awared node placement policy based on network topology > --- > > Key: HDDS-700 > URL: https://issues.apache.org/jira/browse/HDDS-700 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-700.01.patch > > > Implement a new container placement policy implementation based datanode's > network topology. It follows the same rule as HDFS. > By default with 3 replica, two replica will be on the same rack, the third > replica and all the remaining replicas will be on different racks. > > {color:#808080} {color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839942#comment-16839942 ] Hadoop QA commented on HDDS-700: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} yetus {color} | {color:red} 0m 7s{color} | {color:red} Unprocessed flag(s): --jenkins --skip-dir {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/PreCommit-HDDS-Build/2688/artifact/out/Dockerfile | | JIRA Issue | HDDS-700 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12967903/HDDS-700.01.patch | | Console output | https://builds.apache.org/job/PreCommit-HDDS-Build/2688/console | | versions | git=2.7.4 | | Powered by | Apache Yetus 0.11.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Support rack awared node placement policy based on network topology > --- > > Key: HDDS-700 > URL: https://issues.apache.org/jira/browse/HDDS-700 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-700.01.patch > > > Implement a new container placement policy implementation based datanode's > network topology. It follows the same rule as HDFS. > By default with 3 replica, two replica will be on the same rack, the third > replica and all the remaining replicas will be on different racks. > > {color:#808080} {color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839049#comment-16839049 ] Sammi Chen commented on HDDS-700: - Thanks [~xyao] for the comments. bq. Line 353: if the caller does not set the location in the case of no topology defined, should we have a default location like/default-location or /default-rack. Currently I will use /default-rack as the default location to be compatible with current HDFS implementation. In future, we can refine this to get the default location from network topology class. {quote} SCMContainerPlacementRackAware.java Line 77: should we consider redefine the interface to cover case where we want include the affinity node in the first node selection as well? For example, client inside the cluster may prefer to write first copy to local container if possible.{quote} Good point. Will add a new parameter "List favoredNodes" to cover this case, and other cases. {quote} Line 84: should we consider some fallback strategy in certain cases where we allow allocation succeeds but Recover later when the placement if possible later?{quote} Reasonable. But I think this should be related with the mininal acceptable replication factor. For example, if the replication is 3 and minimal replication is 1, then choose datanode should not fail if the mininal replication can be satisfied. Currently there is no minimal replication property in Ozone, also I am not clear if Ratis will still work when the pipeline includes less than 3 nodes. Do you have this information? {quote}Line 101-122: some of the logic can be consolidated.{quote} Need more hint here. All other comments will be addressed in next patch. > Support rack awared node placement policy based on network topology > --- > > Key: HDDS-700 > URL: https://issues.apache.org/jira/browse/HDDS-700 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-700.01.patch > > > Implement a new container placement policy implementation based datanode's > network topology. It follows the same rule as HDFS. > By default with 3 replica, two replica will be on the same rack, the third > replica and all the remaining replicas will be on different racks. > > {color:#808080} {color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836863#comment-16836863 ] Xiaoyu Yao commented on HDDS-700: - Thanks [~Sammi] for the patch. It LGTM overall. Here are a few comments: DatanodeDetails.java Line 266: can we update the toString to include the network location information? Line 353: if the caller does not set the location in the case of no topology defined, should we have a default location like/default-location or /default-rack. SCMContainerPlacementRackAware.java Line 77: should we consider redefine the interface to cover case where we want include the affinity node in the first node selection as well? For example, client inside the cluster may prefer to write first copy to local container if possible. Line 84: should we consider some fallback strategy in certain cases where we allow allocation succeeds but Recover later when the placement if possible later? Line 88: NIT: can we rename it to chosenNodes? Line 101-122: some of the logic can be consolidated. Line 160: can we need to add some logic to avoid infinite loop in node selection? > Support rack awared node placement policy based on network topology > --- > > Key: HDDS-700 > URL: https://issues.apache.org/jira/browse/HDDS-700 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-700.01.patch > > > Implement a new container placement policy implementation based datanode's > network topology. It follows the same rule as HDFS. > By default with 3 replica, two replica will be on the same rack, the third > replica and all the remaining replicas will be on different racks. > > {color:#808080} {color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org