[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14696376#comment-14696376 ] Zhihua Deng commented on HDFS-7980: --- Recently, we encountered the same problem in

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534504#comment-14534504 ] Hudson commented on HDFS-7980: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #921 (See

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534698#comment-14534698 ] Hudson commented on HDFS-7980: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2119 (See

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534931#comment-14534931 ] Hudson commented on HDFS-7980: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2137 (See

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533167#comment-14533167 ] Hudson commented on HDFS-7980: -- FAILURE: Integrated in Hadoop-trunk-Commit #7760 (See

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533639#comment-14533639 ] Hudson commented on HDFS-7980: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #178 (See

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533399#comment-14533399 ] Hudson commented on HDFS-7980: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #188 (See

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533585#comment-14533585 ] Hudson commented on HDFS-7980: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #188 (See

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530441#comment-14530441 ] Walter Su commented on HDFS-7980: - reposted 004 patch. Incremental BlockReport will

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14531148#comment-14531148 ] Tsz Wo Nicholas Sze commented on HDFS-7980: --- The patch looks good. Just a

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14531086#comment-14531086 ] Hadoop QA commented on HDFS-7980: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ ||

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14531859#comment-14531859 ] Walter Su commented on HDFS-7980: - Fsimage saves Files, and block Ids belonging to some

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14529737#comment-14529737 ] Tsz Wo Nicholas Sze commented on HDFS-7980: --- The patch could not be applied.

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14529045#comment-14529045 ] Hadoop QA commented on HDFS-7980: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ ||

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526132#comment-14526132 ] Hadoop QA commented on HDFS-7980: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ ||

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525766#comment-14525766 ] Hadoop QA commented on HDFS-7980: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ ||

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524649#comment-14524649 ] Hadoop QA commented on HDFS-7980: - (!) The patch artifact directory has been removed! This

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522654#comment-14522654 ] Hadoop QA commented on HDFS-7980: - (!) The patch artifact directory has been removed! This

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518628#comment-14518628 ] Walter Su commented on HDFS-7980: - 004 patch fixes failed test. By the way: 1. If a block

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14517984#comment-14517984 ] Tsz Wo Nicholas Sze commented on HDFS-7980: --- The failure of

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14516592#comment-14516592 ] Hadoop QA commented on HDFS-7980: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ ||

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14516284#comment-14516284 ] Walter Su commented on HDFS-7980: - 003 patch is ready for review. Incremental BlockReport

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482692#comment-14482692 ] Yongjun Zhang commented on HDFS-7980: - Thanks [~walter.k.su]. Since an incremental

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482658#comment-14482658 ] Walter Su commented on HDFS-7980: - When I said even when storageInfo.numBlocks() 0 I mean

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482706#comment-14482706 ] Walter Su commented on HDFS-7980: - The incremental block report is processed by

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483764#comment-14483764 ] Tsz Wo Nicholas Sze commented on HDFS-7980: --- I don't think we need to add a test

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484375#comment-14484375 ] Yongjun Zhang commented on HDFS-7980: - Hi [~walter.k.su], Thanks for clarifying. I'd

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484480#comment-14484480 ] Colin Patrick McCabe commented on HDFS-7980: +1 for adding a unit test.

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484564#comment-14484564 ] Walter Su commented on HDFS-7980: - bq. It needs a test for checking the correctness. It

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482513#comment-14482513 ] Walter Su commented on HDFS-7980: - I don't think we need to add a test case. It's too

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482590#comment-14482590 ] Yongjun Zhang commented on HDFS-7980: - Hi Guys, Thanks for reporting and working on

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14393542#comment-14393542 ] Tsz Wo Nicholas Sze commented on HDFS-7980: --- ... But Mockito doesn't support

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14385190#comment-14385190 ] Walter Su commented on HDFS-7980: - I trying to write a test covering the if statement.

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14385003#comment-14385003 ] Tsz Wo Nicholas Sze commented on HDFS-7980: --- Sounds good. Could you also add a

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381699#comment-14381699 ] Walter Su commented on HDFS-7980: - bq. NN treats the incremental block report as the first

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381800#comment-14381800 ] Walter Su commented on HDFS-7980: - Sorry what I said before was not clear. I mean: If we

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381848#comment-14381848 ] Tsz Wo Nicholas Sze commented on HDFS-7980: --- Missing delHint seems a minor issue.

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383287#comment-14383287 ] Walter Su commented on HDFS-7980: - {quote} NN could safely ignore the incremental when

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379484#comment-14379484 ] Tsz Wo Nicholas Sze commented on HDFS-7980: --- As mentioned by [~huizane] in the

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377730#comment-14377730 ] Walter Su commented on HDFS-7980: - Hi, [~huizane]. It looks like a blockreport storm, which

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377756#comment-14377756 ] Hui Zheng commented on HDFS-7980: - Hi [~walter.k.su] dfs.blockreport.initailDelay is 600

[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377689#comment-14377689 ] Hui Zheng commented on HDFS-7980: - In our environment there are over 3000 datanodes, 100