[jira] [Commented] (HDFS-5970) callers of NetworkTopology's chooseRandom method to expect null return value
[ https://issues.apache.org/jira/browse/HDFS-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584198#comment-16584198 ] Henrique Barros commented on HDFS-5970: --- I just reproduced it returning null. See the issue I created please: https://issues.apache.org/jira/browse/HDFS-13833 > callers of NetworkTopology's chooseRandom method to expect null return value > > > Key: HDFS-5970 > URL: https://issues.apache.org/jira/browse/HDFS-5970 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Yongjun Zhang >Priority: Minor > > Class NetworkTopology's method >public Node chooseRandom(String scope) > calls >private Node chooseRandom(String scope, String excludedScope) > which may return null value. > Callers of this method such as BlockPlacementPolicyDefault etc need to be > aware that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5970) callers of NetworkTopology's chooseRandom method to expect null return value
[ https://issues.apache.org/jira/browse/HDFS-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021654#comment-16021654 ] zhangyubiao commented on HDFS-5970: --- [~olegd], what action you do to reproduced ? > callers of NetworkTopology's chooseRandom method to expect null return value > > > Key: HDFS-5970 > URL: https://issues.apache.org/jira/browse/HDFS-5970 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Yongjun Zhang >Priority: Minor > > Class NetworkTopology's method >public Node chooseRandom(String scope) > calls >private Node chooseRandom(String scope, String excludedScope) > which may return null value. > Callers of this method such as BlockPlacementPolicyDefault etc need to be > aware that. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5970) callers of NetworkTopology's chooseRandom method to expect null return value
[ https://issues.apache.org/jira/browse/HDFS-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15150485#comment-15150485 ] Oleg Danilov commented on HDFS-5970: We just "reproduced" this issue accidentally using Hadoop 2.3.0: ... 2016-02-16 11:21:37,217 INFO org.apache.hadoop.net.NetworkTopology: Removing a node: /default-rack/10.5.68.40:1004 2016-02-16 11:21:37,217 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* removeDeadDatanode: lost heartbeat from 10.5.68.45:1004 2016-02-16 11:21:37,217 INFO org.apache.hadoop.net.NetworkTopology: Removing a node: /default-rack/10.5.68.45:1004 2016-02-16 11:21:37,218 FATAL org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: ReplicationMonitor thread received Runtime exception. java.lang.NullPointerException at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:507) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRemoteRack(BlockPlacementPolicyDefault.java:455) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:278) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:212) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:117) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationWork.chooseTargets(BlockManager.java:3309) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationWork.access$200(BlockManager.java:3277) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWorkForBlocks(BlockManager.java:1283) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWork(BlockManager.java:1190) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3250) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3204) at java.lang.Thread.run(Thread.java:745) 2016-02-16 11:21:37,246 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1 2016-02-16 11:21:37,260 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: Unfortunately it causes the namenode shutdown. > callers of NetworkTopology's chooseRandom method to expect null return value > > > Key: HDFS-5970 > URL: https://issues.apache.org/jira/browse/HDFS-5970 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0 >Reporter: Yongjun Zhang >Priority: Minor > > Class NetworkTopology's method >public Node chooseRandom(String scope) > calls >private Node chooseRandom(String scope, String excludedScope) > which may return null value. > Callers of this method such as BlockPlacementPolicyDefault etc need to be > aware that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-5970) callers of NetworkTopology's chooseRandom method to expect null return value
[ https://issues.apache.org/jira/browse/HDFS-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13906813#comment-13906813 ] Junping Du commented on HDFS-5970: -- The plan sounds reasonable. I agree NPE happens here is still theoretically, so we can come back to this when real case happen. Move its priority to Minor but leave it open until we seriously check it won't happen. callers of NetworkTopology's chooseRandom method to expect null return value Key: HDFS-5970 URL: https://issues.apache.org/jira/browse/HDFS-5970 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Yongjun Zhang Priority: Minor Class NetworkTopology's method public Node chooseRandom(String scope) calls private Node chooseRandom(String scope, String excludedScope) which may return null value. Callers of this method such as BlockPlacementPolicyDefault etc need to be aware that. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5970) callers of NetworkTopology's chooseRandom method to expect null return value
[ https://issues.apache.org/jira/browse/HDFS-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13907105#comment-13907105 ] Yongjun Zhang commented on HDFS-5970: - Thanks for making the change. Indeed it's minor. callers of NetworkTopology's chooseRandom method to expect null return value Key: HDFS-5970 URL: https://issues.apache.org/jira/browse/HDFS-5970 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Yongjun Zhang Priority: Minor Class NetworkTopology's method public Node chooseRandom(String scope) calls private Node chooseRandom(String scope, String excludedScope) which may return null value. Callers of this method such as BlockPlacementPolicyDefault etc need to be aware that. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5970) callers of NetworkTopology's chooseRandom method to expect null return value
[ https://issues.apache.org/jira/browse/HDFS-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13905627#comment-13905627 ] Yongjun Zhang commented on HDFS-5970: - Thanks Junping. callers of NetworkTopology's chooseRandom method to expect null return value Key: HDFS-5970 URL: https://issues.apache.org/jira/browse/HDFS-5970 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Yongjun Zhang Class NetworkTopology's method public Node chooseRandom(String scope) calls private Node chooseRandom(String scope, String excludedScope) which may return null value. Callers of this method such as BlockPlacementPolicyDefault etc need to be aware that. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5970) callers of NetworkTopology's chooseRandom method to expect null return value
[ https://issues.apache.org/jira/browse/HDFS-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13906529#comment-13906529 ] Junping Du commented on HDFS-5970: -- Nice catch, Yongjun! Would be great if you can deliver a patch with unit test. callers of NetworkTopology's chooseRandom method to expect null return value Key: HDFS-5970 URL: https://issues.apache.org/jira/browse/HDFS-5970 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Yongjun Zhang Class NetworkTopology's method public Node chooseRandom(String scope) calls private Node chooseRandom(String scope, String excludedScope) which may return null value. Callers of this method such as BlockPlacementPolicyDefault etc need to be aware that. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5970) callers of NetworkTopology's chooseRandom method to expect null return value
[ https://issues.apache.org/jira/browse/HDFS-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13906672#comment-13906672 ] Yongjun Zhang commented on HDFS-5970: - Hi Iunping, Thanks for your encouraging words! I need to mention that this bug is related to HDFS-5939 that I'm working on. I tried to address HDFS-5939 by adding return null from within the chooseRandom mehtod. Thanks to [~wheat9] who suggested that we need to file this JIRA for callers of this method to be aware of that the method could return null when reviewing my HDFS-5939 fix. While we are still discussing the best solution for HDFS-5939, it's likely we will replace the return null I added with throwing an exception. Prior to my change, the method chooseRandom already has a piece of code that returns null under certain condition, So this JIRA is still valid in that sense. However, we haven't got any report of null value returned from that piece of code yet, probably we can keep this JIRA open until we have a real case. Thanks. callers of NetworkTopology's chooseRandom method to expect null return value Key: HDFS-5970 URL: https://issues.apache.org/jira/browse/HDFS-5970 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Yongjun Zhang Class NetworkTopology's method public Node chooseRandom(String scope) calls private Node chooseRandom(String scope, String excludedScope) which may return null value. Callers of this method such as BlockPlacementPolicyDefault etc need to be aware that. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
