[
https://issues.apache.org/jira/browse/HDFS-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14245908#comment-14245908
]
Brahma Reddy Battula commented on HDFS-7414:
--------------------------------------------
Got the cause:
OP_CLOSE has been added after the OP_DELETE of the parent node, This causes the
exception.
So ignore the node file doesnt exists exception as the file is already deleted
from the Namenode so it can suppress this exception.
> Namenode got shutdown and can't recover where edit update might be missed
> -------------------------------------------------------------------------
>
> Key: HDFS-7414
> URL: https://issues.apache.org/jira/browse/HDFS-7414
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.4.1, 2.5.1
> Reporter: Brahma Reddy Battula
> Assignee: Brahma Reddy Battula
> Priority: Blocker
>
> Scenario:
> ========
> Was running mapreduce job.
> CPU usage crossed 190% for Datanode and machine became slow..
> and seen the following exception ..
> *Did not get the exact root cause, but as cpu usage more edit log updation
> might be missed...Need dig to more...anyone have any thoughts.*
> {noformat}
> 2014-11-20 05:01:18,430 | ERROR | main | Encountered exception on operation
> CloseOp [length=0, inodeId=0,
> path=/outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_000025_1/part-m-00025,
> replication=2, mtime=1416409309023, atime=1416409290816, blockSize=67108864,
> blocks=[blk_1073766144_25321, blk_1073766154_25331, blk_1073766160_25337],
> permissions=mapred:supergroup:rw-r--r--, aclEntries=null, clientName=,
> clientMachine=, opCode=OP_CLOSE, txid=162982] |
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232)
> java.io.FileNotFoundException: File does not exist:
> /outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_000025_1/part-m-00025
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:409)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:224)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:893)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:640)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:519)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:575)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:741)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1387)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1459)
> 2014-11-20 05:01:18,654 | WARN | main | Encountered exception loading
> fsimage |
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:642)
> java.io.FileNotFoundException: File does not exist:
> /outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_000025_1/part-m-00025
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:409)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:224)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:893)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:640)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:519)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:575)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:741)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1387)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1459)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
