[
https://issues.apache.org/jira/browse/HDFS-12638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200469#comment-16200469
]
Daryn Sharp commented on HDFS-12638:
------------------------------------
[~yangjiandan] do you have any further info you can share? Did this involve
snapshots? Do you know if the problematic block belonged to a deleted file, or
a truncated file, abandoned block, etc?
This case case appears to mirror ours (except we didn't crash). The block has
a "valid" (not the sentinel no collection id) block collection id for an inode
not in the inode map. The block also isn't in the blocksmap. These are big
problems.
Either the replication monitor is working with copies of stored blocks, or
values are being cleared in the wrong order. Delete tries to unlink the blocks
from the collection before removing from the blocks map, then clears the
collection's block list before removing from the inode map. Thus there should
never be a block referencing a missing collection...
Moving the block's bc from a reference to an id is highly likely to have
exposed latent bugs if copies of stored blocks are not re-resolved via the
blocks map. Having a reference hid logic issues.
> NameNode exits due to ReplicationMonitor thread received Runtime exception in
> ReplicationWork#chooseTargets
> -----------------------------------------------------------------------------------------------------------
>
> Key: HDFS-12638
> URL: https://issues.apache.org/jira/browse/HDFS-12638
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 2.8.2
> Reporter: Jiandan Yang
>
> Active NamNode exit due to NPE, I can confirm that the BlockCollection passed
> in when creating ReplicationWork is null, but I do not know why
> BlockCollection is null, By view history I found
> [HDFS-9754|https://issues.apache.org/jira/browse/HDFS-9754] remove judging
> whether BlockCollection is null.
> NN logs are as following:
> {code:java}
> 2017-10-11 16:29:06,161 ERROR [ReplicationMonitor]
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> ReplicationMonitor thread received Runtime exception.
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.ReplicationWork.chooseTargets(ReplicationWork.java:55)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWorkForBlocks(BlockManager.java:1532)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWork(BlockManager.java:1491)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3792)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3744)
> at java.lang.Thread.run(Thread.java:834)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
