[
https://issues.apache.org/jira/browse/HDFS-9334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14979748#comment-14979748
]
He Xiaoqiao commented on HDFS-9334:
-----------------------------------
hi [~walter.k.su], the case happens at SBN when do editlog tailer. Stack trace:
{code:xml}
2015-10-28 23:14:36,737 ERROR
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered exception
on operation DeleteOp [length=0, path=$PATH, timestamp=1446044851382,
RpcClientId=42f62f2c-2d6a-4bfd-8988-e1ae9fe15b34, RpcCallId=53195,
opCode=OP_DELETE, txid=26131111424]
java.lang.ArrayIndexOutOfBoundsException: -2
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.setPrevious(BlockInfo.java:140)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.listRemove(BlockInfo.java:333)
at
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo.removeBlock(DatanodeStorageInfo.java:227)
at
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor.removeBlock(DatanodeDescriptor.java:288)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.removeBlock(BlocksMap.java:125)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.removeBlock(BlockManager.java:3212)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeBlocksAndUpdateSafemodeTotal(FSNamesystem.java:3473)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removePathAndBlocks(FSNamesystem.java:3445)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedDelete(FSDirectory.java:1418)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:502)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:224)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:786)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:230)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:324)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:282)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:299)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1536)
at
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:413)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:295)
2015-10-28 23:14:36,739 FATAL
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unknown error
encountered while tailing edits. Shutting down standby NN.
java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: -2
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:234)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:786)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:230)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:324)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:282)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:299)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1536)
at
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:413)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:295)
Caused by: java.lang.ArrayIndexOutOfBoundsException: -2
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.setPrevious(BlockInfo.java:140)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.listRemove(BlockInfo.java:333)
at
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo.removeBlock(DatanodeStorageInfo.java:227)
at
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor.removeBlock(DatanodeDescriptor.java:288)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.removeBlock(BlocksMap.java:125)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.removeBlock(BlockManager.java:3212)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeBlocksAndUpdateSafemodeTotal(FSNamesystem.java:3473)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removePathAndBlocks(FSNamesystem.java:3445)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedDelete(FSDirectory.java:1418)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:502)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:224)
... 12 more
{code}
> NameNode crash caused by ArrayIndexOutOfBoundsException
> -------------------------------------------------------
>
> Key: HDFS-9334
> URL: https://issues.apache.org/jira/browse/HDFS-9334
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 2.4.1
> Reporter: He Xiaoqiao
> Assignee: He Xiaoqiao
>
> when remove block from list of blocks by method {{listRemove}} in
> {{BlockInfo.java}}, and {{findStorageInfo(storage)}} return -1 means not
> found, {{prev.setNext(prev.findStorageInfo(storage), next)}} or
> {{next.setPrevious(next.findStorageInfo(storage), prev)}} could throw
> ArrayIndexOutOfBoundsException cause NameNode Process exit.
> {code:title=BlockInfo.java}
> BlockInfo listRemove(BlockInfo head, DatanodeStorageInfo storage) {
> if(head == null)
> return null;
> int dnIndex = this.findStorageInfo(storage);
> if(dnIndex < 0) // this block is not on the data-node list
> return head;
> BlockInfo next = this.getNext(dnIndex);
> BlockInfo prev = this.getPrevious(dnIndex);
> this.setNext(dnIndex, null);
> this.setPrevious(dnIndex, null);
> if(prev != null)
> prev.setNext(prev.findStorageInfo(storage), next);
> if(next != null)
> next.setPrevious(next.findStorageInfo(storage), prev);
> if(this == head) // removing the head
> head = next;
> return head;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
