[
https://issues.apache.org/jira/browse/HDFS-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852104#comment-13852104
]
Buddy commented on HDFS-5483:
-----------------------------
We are hitting this running against trunk with well formed block reports when
the block is reported on the same storage by multiple data nodes.
This is the shared storage case where the block is read/write on one data node
and read/only on others for the same storage.
2013-12-17 23:00:10,329 WARN [IPC Server handler 3 on 8020] WARN
org.apache.hadoop.ipc.Server: IPC Server handler 3 on 8020, call
org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol.blockReport from
10.63.174.149:37148 Call#5919 Retry#0: error: java.lang.AssertionError: Index
is out of bound
java.lang.AssertionError: Index is out of bound
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.setNext(BlockInfo.java:152)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.moveBlockToHead(BlockInfo.java:351)
at
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo.moveBlockToHead(DatanodeStorageInfo.java:237)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.reportDiff(BlockManager.java:1855)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1724)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1652)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReport(NameNodeRpcServer.java:985)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.blockReport(DatanodeProtocolServerSideTranslatorPB.java:145)
at
org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:27220)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:605)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1515)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
> Make reportDiff resilient to malformed block reports
> ----------------------------------------------------
>
> Key: HDFS-5483
> URL: https://issues.apache.org/jira/browse/HDFS-5483
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
> Affects Versions: Heterogeneous Storage (HDFS-2832)
> Reporter: Arpit Agarwal
>
> {{BlockManager#reportDiff}} can cause an assertion failure in
> {{BlockInfo#moveBlockToHead}} if the block report shows the same block as
> belonging to more than one storage.
> The issue is that {{moveBlockToHead}} assumes it will find the
> DatanodeStorageInfo for the given block.
> Exception details:
> {code}
> java.lang.AssertionError: Index is out of bound
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.setNext(BlockInfo.java:152)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.moveBlockToHead(BlockInfo.java:351)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo.moveBlockToHead(DatanodeStorageInfo.java:243)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.reportDiff(BlockManager.java:1841)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1709)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1637)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReport(NameNodeRpcServer.java:984)
> at
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure.testVolumeFailure(TestDataNodeVolumeFailure.java:165)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
