[
https://issues.apache.org/jira/browse/HDFS-11112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677124#comment-15677124
]
Wei-Chiu Chuang commented on HDFS-11112:
----------------------------------------
Hi [~arpitagarwal]
a while back I also studied the behavior of JN in this situation (with minor
deviation) after finished HDFS-10360 to deal with the similar issue in
DataNodes. Here's what I saw. Could you comment if this is also what you saw?
Maybe the situation is different, and from what I saw JN didn't automatically
format the directory.
{quote}
if current/VERSION is missing, it will start with no problem like nothing has
happened. No auto-format. But when NN tries to send edits to the JN, the JN
will respond with JournalNotFormattedException. Cloudera Manager dashboard will
see JN and NN in bad health.
2016-06-07 11:14:18,656 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
on 8485, call org.apache.hadoop.hdfs.qjournal.protocol.QJournal
Protocol.getJournalState from 172.26.10.80:36663 Call#7 Retry#0
org.apache.hadoop.hdfs.qjournal.protocol.JournalNotFormattedException: Journal
Storage Directory /data/jn/nameservice1 not formatted
at
org.apache.hadoop.hdfs.qjournal.server.Journal.checkFormatted(Journal.java:472)
at
org.apache.hadoop.hdfs.qjournal.server.Journal.getLastPromisedEpoch(Journal.java:245)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.getJournalState(JournalNodeRpcServer.java:124)
at
org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.getJournalState(QJournalProtocolServerSideTranslat
orPB.java:118)
at
org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.
java:25415)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
If I remove current/VERSION from all three JNs,
NN will shutdown because it's unabel to get a quorum.
If I put the file back to an active JN, it will still complain about it. But if
I restart the JN with the file in place, the JN will function normally.
So in the case of JN, there is no permanent harm and the damage can be reverted.
{quote}
[~linyiqun] I think the patch looks good to me overall. Do you know the
behavior of JN after your patch? Does JN simply ignores format? Does JN
continues to function normally?
> Journal Nodes should refuse to format non-empty directories
> -----------------------------------------------------------
>
> Key: HDFS-11112
> URL: https://issues.apache.org/jira/browse/HDFS-11112
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Arpit Agarwal
> Assignee: Yiqun Lin
> Attachments: HDFS-11112.001.patch, HDFS-11112.002.patch
>
>
> Journal Nodes should reject the {{format}} RPC request if a storage directory
> is non-empty. The relevant code is in {{JNStorage#format}}.
> {code}
> void format(NamespaceInfo nsInfo) throws IOException {
> setStorageInfo(nsInfo);
> ...
> unlockAll();
> sd.clearDirectory();
> writeProperties(sd);
> createPaxosDir();
> analyzeStorage();
> {code}
> This would make the behavior similar to {{namenode -format -nonInteractive}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
