[ https://issues.apache.org/jira/browse/HDFS-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14041456#comment-14041456 ]
Siqi Li commented on HDFS-6527: ------------------------------- When running unit tests in this patch v5, I get the following errors 2014-06-23 13:36:09,516 ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(873)) - Failed to close file /testDeleteAddBlockRace org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /testDeleteAddBlockRace: File does not exist. Holder DFSClient_NONMAPREDUCE_1652233532_1 does not have any open files. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2941) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2762) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2706) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:585) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:394) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1547) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2008) at org.apache.hadoop.ipc.Client.call(Client.java:1410) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy17.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:188) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103) at com.sun.proxy.$Proxy17.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1443) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1265) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:529) > Edit log corruption due to defered INode removal > ------------------------------------------------ > > Key: HDFS-6527 > URL: https://issues.apache.org/jira/browse/HDFS-6527 > Project: Hadoop HDFS > Issue Type: Bug > Affects Versions: 2.4.0 > Reporter: Kihwal Lee > Assignee: Kihwal Lee > Priority: Blocker > Fix For: 2.4.1 > > Attachments: HDFS-6527.branch-2.4.patch, HDFS-6527.trunk.patch, > HDFS-6527.v2.patch, HDFS-6527.v3.patch, HDFS-6527.v4.patch, HDFS-6527.v5.patch > > > We have seen a SBN crashing with the following error: > {panel} > \[Edit log tailer\] ERROR namenode.FSEditLogLoader: > Encountered exception on operation AddBlockOp > [path=/xxx, > penultimateBlock=NULL, lastBlock=blk_111_111, RpcClientId=, > RpcCallId=-2] > java.io.FileNotFoundException: File does not exist: /xxx > {panel} > This was caused by the deferred removal of deleted inodes from the inode map. > Since getAdditionalBlock() acquires FSN read lock and then write lock, a > deletion can happen in between. Because of deferred inode removal outside FSN > write lock, getAdditionalBlock() can get the deleted inode from the inode map > with FSN write lock held. This allow addition of a block to a deleted file. > As a result, the edit log will contain OP_ADD, OP_DELETE, followed by > OP_ADD_BLOCK. This cannot be replayed by NN, so NN doesn't start up or SBN > crashes. -- This message was sent by Atlassian JIRA (v6.2#6252)