[jira] [Updated] (HDFS-14069) Better debuggability for datanode decomissioning
[ https://issues.apache.org/jira/browse/HDFS-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Becker updated HDFS-14069: Description: Currently, we only provide "minLiveReplicas" per DN that is being decommissioned, this is not enough info because it is difficult to determine which blocks are on their last replica. We have two design options: # Add it to the existing report, on top of minLiveReplicas ** Advantages: *** Initial debugging information would be more thorough ** Disadvantages: *** Add load to normal NN operation by checking every time a DN is decommissioned # Create a new api for querying more detailed info about one DN ** Advantages: *** We wouldnt be adding more load to the NN in normal operation *** Much easier to extend in the future with more info ** Disadvantages: *** Getting the info on demand for this case will be much more expensive actually, cause we will have to find all the blocks on that DN, and then go through all the blocks again and count how many replicas we have etc. was: Currently, we only provide "minLiveReplicas" per DN that is being decommissioned, this is not enough info because it is difficult to determine which blocks are on their last replica. We have two design options: # Add it to the existing report, on top of minLiveReplicas ** Advantages: *** Initial debugging information would be more thorough ** Disadvantages: *** # Create a new api for querying more detailed info about one DN ** Advantages: *** We wouldnt be adding more load to the NN in normal operation *** Much easier to extend in the future with more info ** Disadvantages: *** Getting the info on demand for this case will be much more expensive actually, cause we will have to find all the blocks on that DN, and then go through all the blocks again and count how many replicas we have etc. > Better debuggability for datanode decomissioning > > > Key: HDFS-14069 > URL: https://issues.apache.org/jira/browse/HDFS-14069 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, hdfs, namenode >Reporter: Danny Becker >Priority: Major > Attachments: HDFS-14069.000.patch > > > Currently, we only provide "minLiveReplicas" per DN that is being > decommissioned, this is not enough info because it is difficult to determine > which blocks are on their last replica. We have two design options: > # Add it to the existing report, on top of minLiveReplicas > ** Advantages: > *** Initial debugging information would be more thorough > ** Disadvantages: > *** Add load to normal NN operation by checking every time a DN is > decommissioned > # Create a new api for querying more detailed info about one DN > ** Advantages: > *** We wouldnt be adding more load to the NN in normal operation > *** Much easier to extend in the future with more info > ** Disadvantages: > *** Getting the info on demand for this case will be much more expensive > actually, cause we will have to find all the blocks on that DN, and then go > through all the blocks again and count how many replicas we have etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14069) Better debuggability for datanode decomissioning
[ https://issues.apache.org/jira/browse/HDFS-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Becker updated HDFS-14069: Description: Currently, we only provide "minLiveReplicas" per DN that is being decommissioned, this is not enough info because it is difficult to determine which blocks are on their last replica. We have two design options: # Add it to the existing report, on top of minLiveReplicas ** Advantages: *** Initial debugging information would be more thorough ** Disadvantages: *** # Create a new api for querying more detailed info about one DN ** Advantages: *** We wouldnt be adding more load to the NN in normal operation *** Much easier to extend in the future with more info ** Disadvantages: *** Getting the info on demand for this case will be much more expensive actually, cause we will have to find all the blocks on that DN, and then go through all the blocks again and count how many replicas we have etc. was: Currently, we only provide "minLiveReplicas" per DN that is being decommissioned, this is not enough info because it is difficult to determine which blocks are on their last replica. We have two design options: # Add it to the existing report, on top of minLiveReplicas ** Advantages: *** ** Disadvantages: *** # Create a new api for querying more detailed info about one DN ** Advantages: *** We wouldnt be adding more load to the NN in normal operation *** Much easier to extend in the future with more info ** Disadvantages: *** Getting the info on demand for this case will be much more expensive actually, cause we will have to find all the blocks on that DN, and then go through all the blocks again and count how many replicas we have etc. > Better debuggability for datanode decomissioning > > > Key: HDFS-14069 > URL: https://issues.apache.org/jira/browse/HDFS-14069 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, hdfs, namenode >Reporter: Danny Becker >Priority: Major > Attachments: HDFS-14069.000.patch > > > Currently, we only provide "minLiveReplicas" per DN that is being > decommissioned, this is not enough info because it is difficult to determine > which blocks are on their last replica. We have two design options: > # Add it to the existing report, on top of minLiveReplicas > ** Advantages: > *** Initial debugging information would be more thorough > ** Disadvantages: > *** > # Create a new api for querying more detailed info about one DN > ** Advantages: > *** We wouldnt be adding more load to the NN in normal operation > *** Much easier to extend in the future with more info > ** Disadvantages: > *** Getting the info on demand for this case will be much more expensive > actually, cause we will have to find all the blocks on that DN, and then go > through all the blocks again and count how many replicas we have etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14069) Better debuggability for datanode decomissioning
[ https://issues.apache.org/jira/browse/HDFS-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Becker updated HDFS-14069: Description: Currently, we only provide "minLiveReplicas" per DN that is being decommissioned, this is not enough info because it is difficult to determine which blocks are on their last replica. We have two design options: # Add it to the existing report, on top of minLiveReplicas ** Advantages: *** ** Disadvantages: *** # Create a new api for querying more detailed info about one DN ** Advantages: *** We wouldnt be adding more load to the NN in normal operation *** Much easier to extend in the future with more info ** Disadvantages: *** Getting the info on demand for this case will be much more expensive actually, cause we will have to find all the blocks on that DN, and then go through all the blocks again and count how many replicas we have etc. was: Currently, we only provide "minLiveReplicas" per DN that is being decommissioned, this is not enough info because it is difficult to determine which blocks are on their last replica. We have two design options: # Add it to the existing report, on top of minLiveReplicas ** Advantages: *** ** Disadvantages: *** # Create a new api for querying more detailed info about one DN ** Advantages: *** We wouldnt be adding more load to the NN in normal operation *** Much easier to extend in the future with more info ** Disadvantages: *** Getting the info on demand for this case will be much more expensive actually, cause we will have to find all the blocks on that DN, and then go through all the blocks again and count how many replicas we have etc. > Better debuggability for datanode decomissioning > > > Key: HDFS-14069 > URL: https://issues.apache.org/jira/browse/HDFS-14069 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, hdfs, namenode >Reporter: Danny Becker >Priority: Major > Attachments: HDFS-14069.000.patch > > > Currently, we only provide "minLiveReplicas" per DN that is being > decommissioned, this is not enough info because it is difficult to determine > which blocks are on their last replica. We have two design options: > # Add it to the existing report, on top of minLiveReplicas > ** Advantages: > *** > ** Disadvantages: > *** > # Create a new api for querying more detailed info about one DN > ** Advantages: > *** We wouldnt be adding more load to the NN in normal operation > *** Much easier to extend in the future with more info > ** Disadvantages: > *** Getting the info on demand for this case will be much more expensive > actually, cause we will have to find all the blocks on that DN, and then go > through all the blocks again and count how many replicas we have etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14069) Better debuggability for datanode decomissioning
[ https://issues.apache.org/jira/browse/HDFS-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Becker updated HDFS-14069: Description: Currently, we only provide "minLiveReplicas" per DN that is being decommissioned, this is not enough info because it is difficult to determine which blocks are on their last replica. We have two design options: # Add it to the existing report, on top of minLiveReplicas # Create a new api for querying more detailed info about one DN ** Advantages: *** We wouldnt be adding more load to the NN in normal operation *** Much easier to extend in the future with more info ** Disadvantages: *** Getting the info on demand for this case will be much more expensive actually, cause we will have to find all the blocks on that DN, and then go through all the blocks again and count how many replicas we have etc. was: Currently, we only provide "minLiveReplicas" per DN that is being decommissioned, this is not enough info because it is difficult to determine which blocks are on their last replica. We have two design options: # Add it to the existing report, on top of minLiveReplicas # Create a new api for querying more detailed info about one DN ** Advantages: *** we wouldnt be adding more load to the NN in normal operation *** much easier to extend in the future with more info ** Disadvantages: > Better debuggability for datanode decomissioning > > > Key: HDFS-14069 > URL: https://issues.apache.org/jira/browse/HDFS-14069 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, hdfs, namenode >Reporter: Danny Becker >Priority: Major > Attachments: HDFS-14069.000.patch > > > Currently, we only provide "minLiveReplicas" per DN that is being > decommissioned, this is not enough info because it is difficult to determine > which blocks are on their last replica. We have two design options: > # Add it to the existing report, on top of minLiveReplicas > # Create a new api for querying more detailed info about one DN > ** Advantages: > *** We wouldnt be adding more load to the NN in normal operation > *** Much easier to extend in the future with more info > ** Disadvantages: > *** Getting the info on demand for this case will be much more expensive > actually, cause we will have to find all the blocks on that DN, and then go > through all the blocks again and count how many replicas we have etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14069) Better debuggability for datanode decomissioning
[jira] [Updated] (HDFS-14069) Better debuggability for datanode decomissioning
[ https://issues.apache.org/jira/browse/HDFS-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Becker updated HDFS-14069: Summary: Better debuggability for datanode decomissioning (was: Better debuggability for datanode decommissioning) > Better debuggability for datanode decomissioning > > > Key: HDFS-14069 > URL: https://issues.apache.org/jira/browse/HDFS-14069 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, hdfs, namenode >Reporter: Danny Becker >Priority: Major > Attachments: HDFS-14069.000.patch > > > Currently, we only provide "minLiveReplicas" per DN that is being decomission > Add totalAccessibleBlocks to NumberReplicas > Add logic to track blocks that have less than the maxReplicasTracked > Add Map of low replica blockids to DatanodeDescriptor -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14069) Better debuggability for datanode decomissioning
[ https://issues.apache.org/jira/browse/HDFS-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Becker updated HDFS-14069: Description: Currently, we only provide "minLiveReplicas" per DN that is being decommissioned, this is not enough info because it is difficult to determine which blocks are on their last replica. We have two design options: # Add it to the existing report, on top of minLiveReplicas # Create a new api for querying more detailed info about one DN ** Advantages: *** we wouldnt be adding more load to the NN in normal operation *** much easier to extend in the future with more info ** Disadvantages: was: Currently, we only provide "minLiveReplicas" per DN that is being decommissioned, this is not enough info because it is difficult to determine which blocks are on their last replica. We have two design options: \t > Better debuggability for datanode decomissioning > > > Key: HDFS-14069 > URL: https://issues.apache.org/jira/browse/HDFS-14069 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, hdfs, namenode >Reporter: Danny Becker >Priority: Major > Attachments: HDFS-14069.000.patch > > > Currently, we only provide "minLiveReplicas" per DN that is being > decommissioned, this is not enough info because it is difficult to determine > which blocks are on their last replica. We have two design options: > # Add it to the existing report, on top of minLiveReplicas > # Create a new api for querying more detailed info about one DN > ** Advantages: > *** we wouldnt be adding more load to the NN in normal operation > *** much easier to extend in the future with more info > ** Disadvantages: -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14069) Better debuggability for datanode decomissioning
[ https://issues.apache.org/jira/browse/HDFS-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Becker updated HDFS-14069: Description: Currently, we only provide "minLiveReplicas" per DN that is being decommissioned, this is not enough info because it is difficult to determine which blocks are on their last replica. We have two design options: \t was: Currently, we only provide "minLiveReplicas" per DN that is being decomission Add totalAccessibleBlocks to NumberReplicas Add logic to track blocks that have less than the maxReplicasTracked Add Map of low replica blockids to DatanodeDescriptor > Better debuggability for datanode decomissioning > > > Key: HDFS-14069 > URL: https://issues.apache.org/jira/browse/HDFS-14069 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, hdfs, namenode >Reporter: Danny Becker >Priority: Major > Attachments: HDFS-14069.000.patch > > > Currently, we only provide "minLiveReplicas" per DN that is being > decommissioned, this is not enough info because it is difficult to determine > which blocks are on their last replica. We have two design options: > \t -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
