[
https://issues.apache.org/jira/browse/HDDS-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299644#comment-17299644
]
Janus Chow commented on HDDS-4703:
----------------------------------
In om.log, there is a log showing the subdirectory was there before the first
time OM was started.
{code:java}
2021-03-08 15:28:22 INFO RaftServerConfigKeys:44 -
raft.server.leaderelection.leader.step-down.wait-time = 10s (default)
2021-03-08 15:28:22 INFO RaftServerConfigKeys:44 - raft.server.storage.dir =
[/data/metadata/ratis] (custom)
2021-03-08 15:28:22 INFO RaftServer:235 - om1: found a subdirectory
/data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e
2021-03-08 15:28:22 INFO RaftServer:95 - om1: addNew
group-C5BA1605619E:[om1|rpc:ozone-om-0.ozone-om:9872|priority:0] returns
group-C5BA1605619E:java.util.concurrent.CompletableFuture@b0a1231[Not completed]
2021-03-08 15:28:22 INFO OzoneManager:1336 - OzoneManager Ratis server
initialized at port 9872
{code}
Since the directory of
"/data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e" should be created
when OM runs the first time. The normal log should be as follows:
{code:java}
2021-03-11 23:24:12,709 [Socket Reader #1 for port 9862] INFO
org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 9862
2021-03-11 23:24:12,712 [pool-17-thread-1] INFO
org.apache.ratis.server.RaftServerConfigKeys: raft.server.log.corruption.policy
= EXCEPTION (default)
2021-03-11 23:24:12,713 [pool-17-thread-1] INFO
org.apache.ratis.server.storage.RaftStorageDirectory: The storage directory
/data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e does not exist.
Creating ...
2021-03-11 23:24:12,717 [pool-17-thread-1] INFO
org.apache.ratis.server.storage.RaftStorageDirectory: Lock on
/data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e/in_use.lock acquired
by nodename [email protected]
2021-03-11 23:24:12,725 [pool-17-thread-1] INFO
org.apache.ratis.server.storage.RaftStorage: Storage directory
/data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e has been successfully
formatted.
{code}
A reasonable explanation would be the
"/data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e" was created by a
earlier operation which failed to run the format operation, which is to create
the metafile for Ratis.
Have uploaded a normal OM start log for comparison.
> New OM couldn't be started due to NOT_FORMATTED Ratis dir
> ----------------------------------------------------------
>
> Key: HDDS-4703
> URL: https://issues.apache.org/jira/browse/HDDS-4703
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Marton Elek
> Priority: Blocker
> Attachments: om.init.log, om.log
>
>
> Used the scripts as before, the OM couldn't be started anymore:
> Version:
> {code}
> Source code repository [email protected]:apache/ozone.git -r
> 159b0c61c3264c9c3c3e1e6e94ef853e31138557
> {code}
> The ozone init was successfull:
> {code}
> ************************************************************/
> 2021-01-14 16:01:41 INFO OzoneManagerStarter:90 - registered UNIX signal
> handlers for [TERM, HUP, INT]
> 2021-01-14 16:01:41 INFO OMHANodeDetails:104 - ozone.om.internal.service.id
> is not defined, falling back to ozone.om.service.ids to find serviceID for
> OzoneManager if it is HA enabled cluster
> 2021-01-14 16:01:41 INFO OMHANodeDetails:210 - Configuration either no
> ozone.om.address set. Falling back to the default OM address
> ozone-om-0.ozone-om:9862
> 2021-01-14 16:01:41 INFO OMHANodeDetails:238 - OM Service ID is not set.
> Setting it to the default ID: omServiceIdDefault
> 2021-01-14 16:01:41 WARN ServerUtils:225 - ozone.om.db.dirs is not
> configured. We recommend adding this setting. Falling back to
> ozone.metadata.dirs instead.
> 2021-01-14 16:01:41 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> OM initialization succeeded.Current cluster id for
> sd=/data/metadata/om;cid=CID-4b397366-5296-4cfd-addd-e7cf94ceb846;layoutVersion=0
> 2021-01-14 16:01:41 INFO OzoneManagerStarter:124 - SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down OzoneManager at
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.3.3
> ************************************************************/
> {code}
> But om failed to start:
> {code}
> 2021-01-14 16:11:26 ERROR OzoneManagerStarter:69 - OM start failed with
> exception
> java.io.IOException: Cannot load Storage Directory
> /data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e. Its state:
> NOT_FORMATTED
> at
> org.apache.ratis.server.storage.RaftStorageImpl.<init>(RaftStorageImpl.java:68)
> at
> org.apache.ratis.server.storage.RaftStorageImpl.<init>(RaftStorageImpl.java:51)
> at
> org.apache.ratis.server.impl.ServerState.<init>(ServerState.java:108)
> at
> org.apache.ratis.server.impl.RaftServerImpl.<init>(RaftServerImpl.java:192)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$newRaftServerImpl$4(RaftServerProxy.java:266)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Cannot load Storage Directory
> /data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e. Its state:
> NOT_FORMATTED
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
