[
https://issues.apache.org/jira/browse/HDFS-7046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130072#comment-14130072
]
Daryn Sharp commented on HDFS-7046:
-----------------------------------
Stack trace from an internal 2.5.1-ish build. Line numbers may be off, but
demonstrates a delete adjusted safe mode block totals, triggering exit of safe
mode, which called {{startSecretManagerIfNecessary}} which only starts if
{{getEditLog().isOpenForWrite()}} is true. The edit log stream is actually
null.
{noformat}
[IPC Server handler 84 on 8020] ERROR namenode.FSEditLogLoader: Encountered
exception on operation DeleteOp [length=0, path=..., timestamp=1410390000919,
RpcClientId=..., RpcCallId=..., opCode=OP_DELETE, txid=...]
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.logEdit(FSEditLog.java:410)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.logUpdateMasterKey(FSEditLog.java:954)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.logUpdateMasterKey(FSNamesystem.java:6842)
at
org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSecretManager.logUpdateMasterKey(DelegationTokenSecretManager.java:378)
at
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.updateCurrentKey(AbstractDelegationTokenSecretManager.java:232)
at
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.startThreads(AbstractDelegationTokenSecretManager.java:116)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startSecretManager(FSNamesystem.java:990)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startSecretManagerIfNecessary(FSNamesystem.java:1004)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.access$1200(FSNamesystem.java:294)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.leave(FSNamesystem.java:5208)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.checkMode(FSNamesystem.java:5283)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.setBlockTotal(FSNamesystem.java:5320)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.adjustBlockTotals(FSNamesystem.java:5509)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo.access$2100(FSNamesystem.java:5054)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.adjustSafeModeBlockTotals(FSNamesystem.java:5654)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeBlocksAndUpdateSafemodeTotal(FSNamesystem.java:3646)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removePathAndBlocks(FSNamesystem.java:3610)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedDelete(FSDirectory.java:1196)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:506)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:228)
{noformat}
> HA NN can NPE upon transition to active
> ---------------------------------------
>
> Key: HDFS-7046
> URL: https://issues.apache.org/jira/browse/HDFS-7046
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 3.0.0, 2.5.0
> Reporter: Daryn Sharp
> Priority: Critical
>
> While processing edits, the NN may decide after adjusting block totals to
> leave safe mode - in the middle of the edit. Going active starts the secret
> manager which generates a new secret key, which in turn generates an edit,
> which NPEs because the edit log is not open.
> # Transitions should _not_ occur in the middle of an edit.
> # The edit log appears to claim it's open for write when the stream isn't
> even open
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
