[GitHub] [hadoop] Neilxzn commented on pull request #3320: HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneo
Neilxzn commented on pull request #3320: URL: https://github.com/apache/hadoop/pull/3320#issuecomment-910066888 cc @jojochuang . Failed junit tests seem unrelated. Can you review this patch again? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Neilxzn commented on pull request #3320: HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneo
Neilxzn commented on pull request #3320: URL: https://github.com/apache/hadoop/pull/3320#issuecomment-910066888 cc @jojochuang . Failed junit tests seem unrelated. Can you review this patch again? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Neilxzn commented on pull request #3320: HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneo
Neilxzn commented on pull request #3320: URL: https://github.com/apache/hadoop/pull/3320#issuecomment-906450340 > @Neilxzn Please fix checkstyle and check failed unit tests first. Thanks @Hexiaoqiao Thank you for your review. 1. About BlockPlacementPolicyDefault$chooseTarget, ParameterNumber checkstyle warning is not generated by this patch and it is hard to fix. Maybe we ignore it this patch? 2. I checked these failed tests agian and these tests run pass locally. And these tests seem unrelated. Please review these tests again. Thanks. > hadoop.hdfs.server.namenode.ha.TestEditLogTailer > hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped > hadoop.hdfs.TestHDFSFileSystemContract -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Neilxzn commented on pull request #3320: HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneo
Neilxzn commented on pull request #3320: URL: https://github.com/apache/hadoop/pull/3320#issuecomment-906027161 Failed junit tests seem unrelated. And I try these tests in IDEA locally, these tests run pass. - hadoop.hdfs.server.namenode.ha.TestEditLogTailer - hadoop.hdfs.TestHDFSFileSystemContract - hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Neilxzn commented on pull request #3320: HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneo
Neilxzn commented on pull request #3320: URL: https://github.com/apache/hadoop/pull/3320#issuecomment-905714180 > Thanks @Neilxzn for your works. Great catch here! LGTM. +1 once leaved nits comments fixed. I think it is a bug just when fallback storage policy happens. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Neilxzn commented on pull request #3320: HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneo
Neilxzn commented on pull request #3320: URL: https://github.com/apache/hadoop/pull/3320#issuecomment-905710927 Just wonder if this is common bug for every chooseTarget invokes when assign requiredStorageTypes.size() to numOfReplicas here? or just for set storagePolicy? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Neilxzn commented on pull request #3320: HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneo
Neilxzn commented on pull request #3320: URL: https://github.com/apache/hadoop/pull/3320#issuecomment-904304994 Agree it. I think we should fix it. In my cluster, we use BlockPlacementPolicyDefault to choose dn and the number of SSD DN is much less than DISK DN. It may cause to some block that should be placed to SSD DNs fallback to place DISK DNs when SSD DNs are too busy or no enough place. Consider the following scenario. 1. Create empty file /foo_file 2. Set its storagepolicy to All_SSD 3. Put data to /foo_file 4. /foo_file gets 3 DISK dns for pipeline because SSD dns are too busy at the beginning. 5. When it transfers data in pipeline, one of 3 DISK dns shut down. 6. The client need to get one new dn for existing pipeline in DataStreamer$addDatanode2ExistingPipeline. 7. If SSD dns are available at the moment, namenode will choose the 3 SSD dns and return it to the client. However, the client just need one new dn, namenode returns 3 new SSD dn and the client threw exception in DataStreamer$findNewDatanode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
