[
https://issues.apache.org/jira/browse/HDFS-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13970218#comment-13970218
]
Lohit Vijayarenu commented on HDFS-6248:
----------------------------------------
Here is stack trace of SNN before crash.
{noformat}
2014-03-29 18:07:35,380 ERROR
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered exception
on operation AddOp [length=0, path=/user/foo/bar.txt, replication=3,
mtime=1396116335071, atime=1396116335071, blockSize=536870912, blocks=[],
permissions=foo:supergroup:rw-r--r--,
clientName=DFSClient_attempt_1395346107078_146938_m_000041_1_1098354233_1,
clientMachine=10.10.10.10, opCode=OP_ADD, txid=487688396]
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:281)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:171)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:90)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:708)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:227)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:321)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
at
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
2014-03-29 18:07:35,622 WARN org.apache.hadoop.hdfs.server.namenode.NameNode:
Quota violation in image for //user/foo (Namespace quota : 1445052 consumed :
1304943) (Diskspace quota : 2199023255552000 consumed : 2199023483200164).
2014-03-29 18:07:36,429 FATAL
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unknown error
encountered while tailing edits. Shutting down standby NN.
java.io.IOException: Failed to apply edit log operation AddOp [length=0,
path=/user/foo/bar.txt, replication=3, mtime=1396116335071,
atime=1396116335071, blockSize=536870912, blocks=[],
permissions=foo:supergroup:rw-r--r--,
clientName=DFSClient_attempt_1395346107078_146938_m_000041_1_1098354233_1,
clientMachine=10.10.10.10, opCode=OP_ADD, txid=487688396]: error null
at
org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:94)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:174)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:90)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:708)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:227)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:321)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
at
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
2014-03-29 18:07:36,431 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
2014-03-29 18:07:36,433 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
SHUTDOWN_MSG:
{noformat}
> SNN crash during replay of FSEditLog of files inside directories having
> QuotaExceeded directories
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-6248
> URL: https://issues.apache.org/jira/browse/HDFS-6248
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.6-alpha, 2.4.0
> Environment: NameNode HA setup with Active/Standby using QJM
> Reporter: Lohit Vijayarenu
>
> We are seeing cases when Secondary NameNode crashes without recovery when it
> tries to replay edit log of files which are part of directories which have
> exceeded Quota. While debugging we got stack trace but we are still trying to
> reproduce this and wanted to note this to see if anyone else had seen this
> issue already.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
