[
https://issues.apache.org/jira/browse/HDFS-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14226141#comment-14226141
]
Brahma Reddy Battula commented on HDFS-7414:
--------------------------------------------
Seen following error from namenode log when it's initially came..
{noformat}
2014-11-20 05:01:18,430 | ERROR | main | Encountered exception on operation
CloseOp [length=0, inodeId=0,
path=/outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_000025_1/part-m-00025,
replication=2, mtime=1416409309023, atime=1416409290816, blockSize=67108864,
blocks=[blk_1073766144_25321, blk_1073766154_25331, blk_1073766160_25337],
permissions=mapred:supergroup:rw-r--r--, aclEntries=null, clientName=,
clientMachine=, opCode=OP_CLOSE, txid=162982] |
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232
{noformat}
and let me describe full scenario..
Deleting DIr (which is having 100 files) and running mapreduce job..
I think, while gettingLastInodeInpath it may get null, since I am deleting the
dir where it edits will not synced..
{code}
final INodesInPath iip = fsDir.getLastINodeInPath(path);
final INodeFile file = INodeFile.valueOf(iip.getINode(0), path);
{code}
Anyone have Anythoughts on this..?
> Namenode got shutdown and can't recover where edit update might be missed
> -------------------------------------------------------------------------
>
> Key: HDFS-7414
> URL: https://issues.apache.org/jira/browse/HDFS-7414
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.4.1, 2.5.1
> Reporter: Brahma Reddy Battula
> Assignee: Brahma Reddy Battula
> Priority: Blocker
>
> Scenario:
> ========
> Was running mapreduce job.
> CPU usage crossed 190% for Datanode and machine became slow..
> and seen the following exception ..
> *Did not get the exact root cause, but as cpu usage more edit log updation
> might be missed...Need dig to more...anyone have any thoughts.*
> {noformat}
> 2014-11-20 05:01:18,430 | ERROR | main | Encountered exception on operation
> CloseOp [length=0, inodeId=0,
> path=/outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_000025_1/part-m-00025,
> replication=2, mtime=1416409309023, atime=1416409290816, blockSize=67108864,
> blocks=[blk_1073766144_25321, blk_1073766154_25331, blk_1073766160_25337],
> permissions=mapred:supergroup:rw-r--r--, aclEntries=null, clientName=,
> clientMachine=, opCode=OP_CLOSE, txid=162982] |
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232)
> java.io.FileNotFoundException: File does not exist:
> /outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_000025_1/part-m-00025
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:409)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:224)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:893)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:640)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:519)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:575)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:741)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1387)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1459)
> 2014-11-20 05:01:18,654 | WARN | main | Encountered exception loading
> fsimage |
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:642)
> java.io.FileNotFoundException: File does not exist:
> /outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_000025_1/part-m-00025
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:409)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:224)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:893)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:640)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:519)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:575)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:741)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1387)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1459)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
