Jyotirmoy Sinha created HDDS-13173:
--------------------------------------
Summary: [snapshot] OM start failed with exception
java.io.IOException
Key: HDDS-13173
URL: https://issues.apache.org/jira/browse/HDDS-13173
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Manager
Reporter: Jyotirmoy Sinha
Steps :
* Execute snapshot load workloads
* Workload executed for 3+ days
* Workload details -
** Volume - 100
** Buckets - 400 (4 per volume)
*** Ratis-fso
*** Ratis-obs
*** Ec-fso
*** Ec-obs
** Keys -
*** 500-1000 per snapshot
*** Size = 10kb
* Configs -
** ozone.compaction.service.enabled = True
** ozone.om.compaction.service.run.interval = 1m
** ozone.om.snapshot.compaction.dag.prune.daemon.run.interval = 5m
** ozone.scm.block.size = 1KB
OM Error stacktrace -
{code:java}
2025-06-04 05:09:28,028 INFO
[main]-org.apache.hadoop.hdds.utils.NativeLibraryLoader: Loading Library:
ozone_rocksdb_tools
2025-06-04 05:09:28,312 INFO
[main]-org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Shutting down
CompactionDagPruningService.
2025-06-04 05:09:28,313 ERROR
[main]-org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with
exception
java.io.IOException: Failed to create RDBStore from
/var/lib/hadoop-ozone/om/data/om.db
at org.apache.hadoop.hdds.utils.db.RDBStore.<init>(RDBStore.java:186)
at
org.apache.hadoop.hdds.utils.db.DBStoreBuilder.build(DBStoreBuilder.java:236)
at
org.apache.hadoop.ozone.om.OmMetadataManagerImpl.loadDB(OmMetadataManagerImpl.java:603)
at
org.apache.hadoop.ozone.om.OmMetadataManagerImpl.loadDB(OmMetadataManagerImpl.java:564)
at
org.apache.hadoop.ozone.om.OmMetadataManagerImpl.start(OmMetadataManagerImpl.java:554)
at
org.apache.hadoop.ozone.om.OmMetadataManagerImpl.<init>(OmMetadataManagerImpl.java:338)
at
org.apache.hadoop.ozone.om.OzoneManager.instantiateServices(OzoneManager.java:909)
at org.apache.hadoop.ozone.om.OzoneManager.<init>(OzoneManager.java:685)
at
org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:874)
at
org.apache.hadoop.ozone.om.OzoneManagerStarter$OMStarterHelper.start(OzoneManagerStarter.java:189)
at
org.apache.hadoop.ozone.om.OzoneManagerStarter.startOm(OzoneManagerStarter.java:86)
at
org.apache.hadoop.ozone.om.OzoneManagerStarter.call(OzoneManagerStarter.java:74)
at org.apache.hadoop.hdds.cli.GenericCli.call(GenericCli.java:38)
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
at picocli.CommandLine.access$1300(CommandLine.java:145)
at
picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
at
picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
at picocli.CommandLine.execute(CommandLine.java:2078)
at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:103)
at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:94)
at
org.apache.hadoop.ozone.om.OzoneManagerStarter.main(OzoneManagerStarter.java:58)
Caused by: org.apache.hadoop.hdds.utils.db.RocksDatabaseException:
IOError(NoSpace): class org.apache.hadoop.hdds.utils.db.RocksDatabase: Failed
to open /var/lib/hadoop-ozone/om/data/om.db
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.toRocksDatabaseException(RocksDatabase.java:93)
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.open(RocksDatabase.java:170)
at org.apache.hadoop.hdds.utils.db.RDBStore.<init>(RDBStore.java:118)
... 22 more
Caused by: org.rocksdb.RocksDBException: While appending to file:
/var/lib/hadoop-ozone/om/data/om.db/062092.dbtmp: No space left on device
at org.rocksdb.RocksDB.open(Native Method)
at org.rocksdb.RocksDB.open(RocksDB.java:307)
at
org.apache.hadoop.hdds.utils.db.managed.ManagedRocksDB.open(ManagedRocksDB.java:84)
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.open(RocksDatabase.java:156)
... 23 more
2025-06-04 05:09:28,318 INFO
[shutdown-hook-0]-org.apache.hadoop.ozone.om.OzoneManagerStarter: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down OzoneManager at
ccycloud-6.quasar-vckttz.root.comops.site/10.140.147.64
************************************************************/ {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
