[
https://issues.apache.org/jira/browse/HDDS-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17722843#comment-17722843
]
Uma Maheswara Rao G commented on HDDS-5641:
-------------------------------------------
[~Sammi] is this fixed now? Could you please confirm?
> Empty CURRENT file of rocksDB after DN restart
> ----------------------------------------------
>
> Key: HDDS-5641
> URL: https://issues.apache.org/jira/browse/HDDS-5641
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Sammi Chen
> Priority: Critical
>
> When restart a DN, found following LOGs,
> 2021-08-19 15:41:56,893 [Thread-6] ERROR
> org.apache.hadoop.ozone.container.common.utils.ContainerCache: Error opening
> DB. Container:54469
> ContainerPath:/data11/ozone/hdds/hdds/CID-97a03be3-7554-46dc-ae39-58d600bfb107/current/containerDir106/54469/metadata/54469-dn-container.db
> java.io.IOException: Failed init RocksDB, db path :
> /data11/ozone/hdds/hdds/CID-97a03be3-7554-46dc-ae39-58d600bfb107/current/containerDir106/54469/metadata/54469-dn-container.db,
> exception :org.rocksdb.RocksDBException CURRENT file does not end with
> newline; status : Corruption; message : CURRENT file does not end with newline
> at
> org.apache.hadoop.hdds.utils.HddsServerUtil.toIOException(HddsServerUtil.java:545)
> at org.apache.hadoop.hdds.utils.db.RDBStore.<init>(RDBStore.java:163)
> at
> org.apache.hadoop.hdds.utils.db.DBStoreBuilder.build(DBStoreBuilder.java:191)
> at
> org.apache.hadoop.ozone.container.metadata.AbstractDatanodeStore.start(AbstractDatanodeStore.java:131)
> at
> org.apache.hadoop.ozone.container.metadata.AbstractDatanodeStore.<init>(AbstractDatanodeStore.java:106)
> at
> org.apache.hadoop.ozone.container.metadata.DatanodeStoreSchemaTwoImpl.<init>(DatanodeStoreSchemaTwoImpl.java:48)
> at
> org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils.getUncachedDatanodeStore(BlockUtils.java:75)
> at
> org.apache.hadoop.ozone.container.common.utils.ContainerCache.getDB(ContainerCache.java:159)
> at
> org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils.getDB(BlockUtils.java:121)
> at
> org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.parseKVContainerData(KeyValueContainerUtil.java:201)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.upgradeContainerLayout(ContainerReader.java:246)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.verifyAndFixupContainerData(ContainerReader.java:271)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.verifyContainerFile(ContainerReader.java:215)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.readVolume(ContainerReader.java:159)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.run(ContainerReader.java:109)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.rocksdb.RocksDBException: CURRENT file does not end with
> newline
> at org.rocksdb.RocksDB.open(Native Method)
> at org.rocksdb.RocksDB.open(RocksDB.java:306)
> at org.apache.hadoop.hdds.utils.db.RDBStore.<init>(RDBStore.java:118)
> ... 14 more
> 2021-08-19 15:41:56,895 [Thread-6] ERROR
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader: Failed to parse
> ContainerFile for ContainerID: 54469
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> Error opening DB. Container:54469
> ContainerPath:/data11/ozone/hdds/hdds/CID-97a03be3-7554-46dc-ae39-58d600bfb107/current/containerDir106/54469/metadata/54469-dn-container.db
> at
> org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils.getDB(BlockUtils.java:129)
> at
> org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.parseKVContainerData(KeyValueContainerUtil.java:201)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.upgradeContainerLayout(ContainerReader.java:246)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.verifyAndFixupContainerData(ContainerReader.java:271)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.verifyContainerFile(ContainerReader.java:215)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.readVolume(ContainerReader.java:159)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.run(ContainerReader.java:109)
> at java.lang.Thread.run(Thread.java:748)
>
> A futher investigation shows that the CURRENT file is empty.
> Here is the content of 54469.container,
> !<KeyValueContainerData>
> checksum: 4abf4aeeb494c96d47b8c5b0da3a7e8de8c18fe418efd150bc88531cb813fe5a
> chunksPath:
> /data11/ozone/hdds/hdds/CID-97a03be3-7554-46dc-ae39-58d600bfb107/current/containerDir106/54469/chunks
> containerDBType: RocksDB
> containerID: 54469
> containerType: KeyValueContainer
> layOutVersion: 2
> maxSize: 5368709120
> metadata: {}
> metadataPath:
> /data11/ozone/hdds/hdds/CID-97a03be3-7554-46dc-ae39-58d600bfb107/current/containerDir106/54469/metadata
> originNodeId: 634a1474-5580-48ac-b652-618ba2a7a27e
> originPipelineId: e397a2d3-0ee3-4d57-bad1-5914fb0fa3fa
> schemaVersion: '2'
> state: CLOSED
> And here is the output of container info CLI
> {
> "state" : "CLOSED",
> "replicationFactor" : "THREE",
> "replicationType" : "RATIS",
> "usedBytes" : 4026743214,
> "numberOfKeys" : 31,
> "lastUsed" : "2021-08-19T07:52:59.758Z",
> "stateEnterTime" : "2021-05-14T06:39:18.675Z",
> "owner" : "BP-374564002-11.26.14.197-1614743795885",
> "containerID" : 54469,
> "deleteTransactionId" : 0,
> "sequenceId" : 3595566,
> "open" : false
> }
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
