Brahma Reddy Battula created HDFS-7414:
------------------------------------------
Summary: Namenode got shutdown and can't recover where edit update
might be missed
Key: HDFS-7414
URL: https://issues.apache.org/jira/browse/HDFS-7414
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.1, 2.4.1
Reporter: Brahma Reddy Battula
Priority: Blocker
Scenario:
========
Was running mapreduce job.
CPU usage crossed 190% for Datanode and machine became slow..
and seen the following exception ..
*Did not get the exact root cause, but as cpu usage more edit log updation
might be missed...Need dig to more...anyone any thoughts.*
{noformat}
2014-11-20 05:01:18,430 | ERROR | main | Encountered exception on operation
CloseOp [length=0, inodeId=0,
path=/outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_000025_1/part-m-00025,
replication=2, mtime=1416409309023, atime=1416409290816, blockSize=67108864,
blocks=[blk_1073766144_25321, blk_1073766154_25331, blk_1073766160_25337],
permissions=mapred:supergroup:rw-r--r--, aclEntries=null, clientName=,
clientMachine=, opCode=OP_CLOSE, txid=162982] |
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232)
java.io.FileNotFoundException: File does not exist:
/outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_000025_1/part-m-00025
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:409)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:224)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:893)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:640)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:519)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:575)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:741)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1387)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1459)
2014-11-20 05:01:18,654 | WARN | main | Encountered exception loading fsimage
|
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:642)
java.io.FileNotFoundException: File does not exist:
/outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_000025_1/part-m-00025
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:409)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:224)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:893)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:640)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:519)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:575)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:741)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1387)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1459)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
