[jira] [Comment Edited] (HDDS-3952) Make container could be reopened
[ https://issues.apache.org/jira/browse/HDDS-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17159636#comment-17159636 ] runzhiwang edited comment on HDDS-3952 at 7/17/20, 3:02 AM: [~msingh] Hi, Thanks for review. The backgroud of this jira is our production cluster has about 7000 small containers which are not full but closed, as the image shows, because ratis pipeline is not stable. So we want to reopen and write to the closed but not full container. The basic idea: 1. SCM build a map with entry <3 datanodes, set of closed but not full containers on the 3 datanodes> 2. SCM check whether any open pipeline locate on the 3 datanodes from the map.entrySet, if exists such open pipeline, we get the closed but not full containers by map.get(pipeline.datanodes()), put them on the pipeline and reopen them. 3. When SCM create new pipeline, we first select from the map which 3 datanodes has the most closed but not full containers, and create pipeline on this 3 datanodes. Then put the containers of map.get(pipeline.datanodes()) on the pipeline and reopen them. [~msingh] What do you think ? !screenshot-1.png! was (Author: yjxxtd): [~msingh] Hi, Thanks for review. The backgroud of this jira is our production cluster has about 7000 small containers which are not full but closed, as the image shows, because ratis pipeline is not stable. So we want to reopen and write to the closed but not full container. The basic idea: 1. SCM build a map with entry <3 datanodes, set of closed but not full containers on the 3 datanodes> 2. SCM check whether any open pipeline locate on the 3 datanodes from the map.entrySet, if exists such open pipeline, we get the closed but not full containers by map.get(pipeline.datanodes()), put them on the pipeline and reopen them. 3. When SCM create new pipeline, we first select from the map which 3 datanodes has the most closed but not full containers, and create pipeline on this 3 datanodes. [~msingh] What do you think ? !screenshot-1.png! > Make container could be reopened > > > Key: HDDS-3952 > URL: https://issues.apache.org/jira/browse/HDDS-3952 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode, SCM >Affects Versions: 0.7.0 >Reporter: maobaolong >Assignee: runzhiwang >Priority: Major > Attachments: screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-3952) Make container could be reopened
[ https://issues.apache.org/jira/browse/HDDS-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17159636#comment-17159636 ] runzhiwang edited comment on HDDS-3952 at 7/17/20, 3:01 AM: [~msingh] Hi, Thanks for review. The backgroud of this jira is our production cluster has about 7000 small containers which are not full but closed, as the image shows, because ratis pipeline is not stable. So we want to reopen and write to the closed but not full container. The basic idea: 1. SCM build a map with entry <3 datanodes, set of closed but not full containers on the 3 datanodes> 2. SCM check whether any open pipeline locate on the 3 datanodes from the map.entrySet, if exists such open pipeline, we get the closed but not full containers by map.get(pipeline.datanodes()), put them on the pipeline and reopen them. 3. When SCM create new pipeline, we first select from the map which 3 datanodes has the most closed but not full containers, and create pipeline on this 3 datanodes. [~msingh] What do you think ? !screenshot-1.png! was (Author: yjxxtd): [~msingh] Hi, Thanks for review. The backgroud of this jira is our production cluster has about 7000 small containers which are not full but closed, as the image shows, because ratis pipeline is not stable. So We want to reopen and write to the closed but not full container. The basic idea: 1. SCM build a map with entry <3 datanodes, set of closed but not full containers on the 3 datanodes> 2. SCM check whether any open pipeline locate on the 3 datanodes from the map.entrySet, if exists such open pipeline, we get the closed but not full containers by map.get(pipeline.datanodes()), put them on the pipeline and reopen them. 3. When SCM create new pipeline, we first select from the map which 3 datanodes has the most closed but not full containers, and create pipeline on this 3 datanodes. [~msingh] What do you think ? !screenshot-1.png! > Make container could be reopened > > > Key: HDDS-3952 > URL: https://issues.apache.org/jira/browse/HDDS-3952 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode, SCM >Affects Versions: 0.7.0 >Reporter: maobaolong >Assignee: runzhiwang >Priority: Major > Attachments: screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-3952) Make container could be reopened
[ https://issues.apache.org/jira/browse/HDDS-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17159636#comment-17159636 ] runzhiwang edited comment on HDDS-3952 at 7/17/20, 3:00 AM: [~msingh] Hi, Thanks for review. The backgroud of this jira is our production cluster has about 7000 small containers which are not full but closed, as the image shows, because ratis pipeline is not stable. So We want to reopen and write to the closed but not full container. The basic idea: 1. SCM build a map with entry <3 datanodes, set of closed but not full containers on the 3 datanodes> 2. SCM check whether any open pipeline locate on the 3 datanodes from the map.entrySet, if exists such open pipeline, we get the closed but not full containers by map.get(pipeline.datanodes()), put them on the pipeline and reopen them. 3. When SCM create new pipeline, we first select from the map which 3 datanodes has the most closed but not full containers, and create pipeline on this 3 datanodes. [~msingh] What do you think ? !screenshot-1.png! was (Author: yjxxtd): [~msingh] Hi, Thanks for review. The backgroud of this jira is our production cluster has about 7000 small container which is not full but closed, as the image shows, because ratis pipeline is not stable. So We want to reopen and write to the closed but not full container. The basic idea: 1. SCM build a map with entry <3 datanodes, set of closed but not full containers on the 3 datanodes> 2. SCM check whether any open pipeline locate on the 3 datanodes from the map.entrySet, if exists such open pipeline, we get the closed but not full containers by map.get(pipeline.datanodes()), put them on the pipeline and reopen them. 3. When SCM create new pipeline, we first select from the map which 3 datanodes has the most closed but not full containers, and create pipeline on this 3 datanodes. [~msingh] What do you think ? !screenshot-1.png! > Make container could be reopened > > > Key: HDDS-3952 > URL: https://issues.apache.org/jira/browse/HDDS-3952 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode, SCM >Affects Versions: 0.7.0 >Reporter: maobaolong >Assignee: runzhiwang >Priority: Major > Attachments: screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
