Jyotirmoy Sinha created HDDS-8071:
-------------------------------------
Summary: [snapshot] OM Process exits with
'java.lang.RuntimeException: Can't find SST file'
Key: HDDS-8071
URL: https://issues.apache.org/jira/browse/HDDS-8071
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Manager
Reporter: Jyotirmoy Sinha
OM Process exits with 'java.lang.RuntimeException: Can't find SST file'
Cluster contains more than 1400 snapshot across many volumes and buckets.
ozone-om.log stacktrace -
{code:java}
2023-03-02 14:12:33,537 WARN
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Compaction log exists:
/var/lib/hadoop-ozone/om/data/compaction-log/0000000000000015569.log. Will
append
2023-03-02 14:12:33,569 ERROR org.apache.hadoop.ozone.om.OzoneManagerStarter:
OM start failed with exception
java.lang.RuntimeException: Can't find SST file: 000076.sst
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getAbsoluteSstFilePath(RocksDBCheckpointDiffer.java:560)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTFileSummary(RocksDBCheckpointDiffer.java:540)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.addNodeToDAG(RocksDBCheckpointDiffer.java:1001)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.lambda$populateCompactionDAG$2(RocksDBCheckpointDiffer.java:1030)
at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.populateCompactionDAG(RocksDBCheckpointDiffer.java:1029)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.processCompactionLogLine(RocksDBCheckpointDiffer.java:676)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at
java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at
java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.readCompactionLogToDAG(RocksDBCheckpointDiffer.java:690)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.loadAllCompactionLogs(RocksDBCheckpointDiffer.java:711)
at org.apache.hadoop.hdds.utils.db.RDBStore.<init>(RDBStore.java:166)
at
org.apache.hadoop.hdds.utils.db.DBStoreBuilder.build(DBStoreBuilder.java:219)
at
org.apache.hadoop.ozone.om.OmMetadataManagerImpl.loadDB(OmMetadataManagerImpl.java:481)
at
org.apache.hadoop.ozone.om.OmMetadataManagerImpl.loadDB(OmMetadataManagerImpl.java:465)
at
org.apache.hadoop.ozone.om.OmMetadataManagerImpl.start(OmMetadataManagerImpl.java:457)
at
org.apache.hadoop.ozone.om.OmMetadataManagerImpl.<init>(OmMetadataManagerImpl.java:295)
at
org.apache.hadoop.ozone.om.OzoneManager.instantiateServices(OzoneManager.java:743)
at org.apache.hadoop.ozone.om.OzoneManager.<init>(OzoneManager.java:623)
at
org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:708)
at
org.apache.hadoop.ozone.om.OzoneManagerStarter$OMStarterHelper.start(OzoneManagerStarter.java:189)
at
org.apache.hadoop.ozone.om.OzoneManagerStarter.startOm(OzoneManagerStarter.java:86)
at
org.apache.hadoop.ozone.om.OzoneManagerStarter.call(OzoneManagerStarter.java:74)
at org.apache.hadoop.hdds.cli.GenericCli.call(GenericCli.java:38)
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
at picocli.CommandLine.access$1300(CommandLine.java:145)
at
picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
at
picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
at picocli.CommandLine.execute(CommandLine.java:2078)
at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100)
at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91)
at
org.apache.hadoop.ozone.om.OzoneManagerStarter.main(OzoneManagerStarter.java:58)
2023-03-02 14:12:33,573 INFO org.apache.hadoop.ozone.om.OzoneManagerStarter:
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down OzoneManager at
jssnap01-2.jssnap01.root.hwx.site/172.27.31.201
************************************************************/ {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
