[jira] [Commented] (HDFS-14678) Allow triggerBlockReport to a specific namenode
[ https://issues.apache.org/jira/browse/HDFS-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943259#comment-16943259 ] Wei-Chiu Chuang commented on HDFS-14678: Cherry picking the commit into branch-3.2 and branch-3.1. There's just a trivial conflict in the test code due to HADOOP-14178. > Allow triggerBlockReport to a specific namenode > --- > > Key: HDFS-14678 > URL: https://issues.apache.org/jira/browse/HDFS-14678 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.8.2 >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Major > Fix For: 3.3.0 > > > In our largest prod cluster (running 2.8.2) we have >3k hosts. Every time > when rolling restarting NNs we will need to wait for block report which takes > >2.5 hours for each NN. > One way to make it faster is to manually trigger a full block report from all > datanodes. [HDFS-7278|https://issues.apache.org/jira/browse/HDFS-7278]. > However, the current triggerBlockReport command will trigger a block report > on all NNs which will flood the active NN as well. > A quick solution will be adding an option to specify a NN that the manually > triggered block report will go to, something like: > *_hdfs dfsadmin [-triggerBlockReport [-incremental] ] > [-namenode] _* > So when doing a restart of standby NN or observer NN we can trigger an > aggressive block report to a specific NN to exit safemode faster without > risking active NN performance. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14678) Allow triggerBlockReport to a specific namenode
[ https://issues.apache.org/jira/browse/HDFS-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910767#comment-16910767 ] Leon Gao commented on HDFS-14678: - Yes we need this in branch 2, I will create another PR for this. > Allow triggerBlockReport to a specific namenode > --- > > Key: HDFS-14678 > URL: https://issues.apache.org/jira/browse/HDFS-14678 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.8.2 >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Major > Fix For: 3.3.0 > > > In our largest prod cluster (running 2.8.2) we have >3k hosts. Every time > when rolling restarting NNs we will need to wait for block report which takes > >2.5 hours for each NN. > One way to make it faster is to manually trigger a full block report from all > datanodes. [HDFS-7278|https://issues.apache.org/jira/browse/HDFS-7278]. > However, the current triggerBlockReport command will trigger a block report > on all NNs which will flood the active NN as well. > A quick solution will be adding an option to specify a NN that the manually > triggered block report will go to, something like: > *_hdfs dfsadmin [-triggerBlockReport [-incremental] ] > [-namenode] _* > So when doing a restart of standby NN or observer NN we can trigger an > aggressive block report to a specific NN to exit safemode faster without > risking active NN performance. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14678) Allow triggerBlockReport to a specific namenode
[ https://issues.apache.org/jira/browse/HDFS-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909403#comment-16909403 ] Hudson commented on HDFS-14678: --- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17137 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17137/]) HDFS-14678. Allow triggerBlockReport to a specific namenode. (#1252). (weichiu: rev 9a1d8cfaf50ec29ffb2d8522ba2f4bc6605d8b8b) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSAdmin.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientDatanodeProtocol.proto * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/BlockReportOptions.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java > Allow triggerBlockReport to a specific namenode > --- > > Key: HDFS-14678 > URL: https://issues.apache.org/jira/browse/HDFS-14678 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.8.2 >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Major > Fix For: 3.3.0 > > > In our largest prod cluster (running 2.8.2) we have >3k hosts. Every time > when rolling restarting NNs we will need to wait for block report which takes > >2.5 hours for each NN. > One way to make it faster is to manually trigger a full block report from all > datanodes. [HDFS-7278|https://issues.apache.org/jira/browse/HDFS-7278]. > However, the current triggerBlockReport command will trigger a block report > on all NNs which will flood the active NN as well. > A quick solution will be adding an option to specify a NN that the manually > triggered block report will go to, something like: > *_hdfs dfsadmin [-triggerBlockReport [-incremental] ] > [-namenode] _* > So when doing a restart of standby NN or observer NN we can trigger an > aggressive block report to a specific NN to exit safemode faster without > risking active NN performance. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14678) Allow triggerBlockReport to a specific namenode
[ https://issues.apache.org/jira/browse/HDFS-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895754#comment-16895754 ] CR Hota commented on HDFS-14678: [~LeonG] This is a very important. Thanks for creating the ticket. Looking forward to the fix. > Allow triggerBlockReport to a specific namenode > --- > > Key: HDFS-14678 > URL: https://issues.apache.org/jira/browse/HDFS-14678 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.8.2 >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Minor > > In our largest prod cluster (running 2.8.2) we have >3k hosts. Every time > when rolling restarting NNs we will need to wait for block report which takes > >2.5 hours for each NN. > One way to make it faster is to manually trigger a full block report from all > datanodes. [HDFS-7278|https://issues.apache.org/jira/browse/HDFS-7278]. > However, the current triggerBlockReport command will trigger a block report > on all NNs which will flood the active NN as well. > A quick solution will be adding an option to specify a NN that the manually > triggered block report will go to, something like: > *_hdfs dfsadmin [-triggerBlockReport [-incremental] ] > [-namenode] _* > So when doing a restart of standby NN or observer NN we can trigger an > aggressive block report to a specific NN to exit safemode faster without > risking active NN performance. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14678) Allow triggerBlockReport to a specific namenode
[ https://issues.apache.org/jira/browse/HDFS-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895727#comment-16895727 ] Ayush Saxena commented on HDFS-14678: - Thanx [~LeonG] & [~csun], Makes sense to me. Assigned this to [~LeonG] > Allow triggerBlockReport to a specific namenode > --- > > Key: HDFS-14678 > URL: https://issues.apache.org/jira/browse/HDFS-14678 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.8.2 >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Minor > > In our largest prod cluster (running 2.8.2) we have >3k hosts. Every time > when rolling restarting NNs we will need to wait for block report which takes > >2.5 hours for each NN. > One way to make it faster is to manually trigger a full block report from all > datanodes. [HDFS-7278|https://issues.apache.org/jira/browse/HDFS-7278]. > However, the current triggerBlockReport command will trigger a block report > on all NNs which will flood the active NN as well. > A quick solution will be adding an option to specify a NN that the manually > triggered block report will go to, something like: > *_hdfs dfsadmin [-triggerBlockReport [-incremental] ] > [-namenode] _* > So when doing a restart of standby NN or observer NN we can trigger an > aggressive block report to a specific NN to exit safemode faster without > risking active NN performance. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14678) Allow triggerBlockReport to a specific namenode
[ https://issues.apache.org/jira/browse/HDFS-14678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895664#comment-16895664 ] Chao Sun commented on HDFS-14678: - cc [~jojochuang]: could you add [~LeonG] as contributor and assign this JIRA to him? also it would be great if you know whether this issue has been raised before or not :) - it is one of the items we want to improve in H2. > Allow triggerBlockReport to a specific namenode > --- > > Key: HDFS-14678 > URL: https://issues.apache.org/jira/browse/HDFS-14678 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.8.2 >Reporter: Leon Gao >Priority: Minor > > In our largest prod cluster (running 2.8.2) we have >3k hosts. Every time > when rolling restarting NNs we will need to wait for block report which takes > >2.5 hours for each NN. > One way to make it faster is to manually trigger a full block report from all > datanodes. [HDFS-7278|https://issues.apache.org/jira/browse/HDFS-7278]. > However, the current triggerBlockReport command will trigger a block report > on all NNs which will flood the active NN as well. > A quick solution will be adding an option to specify a NN that the manually > triggered block report will go to, something like: > *_hdfs dfsadmin [-triggerBlockReport [-incremental] ] > [-namenode] _* > So when doing a restart of standby NN or observer NN we can trigger an > aggressive block report to a specific NN to exit safemode faster without > risking active NN performance. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
