Jyotirmoy Sinha created HDDS-10706:
--------------------------------------
Summary: OMDoubleBuffer error while creating snapshot
Key: HDDS-10706
URL: https://issues.apache.org/jira/browse/HDDS-10706
Project: Apache Ozone
Issue Type: Bug
Components: OM, Snapshot
Reporter: Jyotirmoy Sinha
Scenario -
* Generate data over parallel threads over various volume/buckets
* Perform parallel snapshot create/delete/list operations over above buckets
* Perform parallel snapdiff operations over each bucket
* Perform parallel read operations of snapshot contents
* Introduce OM and cluster restarts in between along with DN decommissioning
and balancer restarts.
OM Leader error stacktrace -
{code:java}
2024-04-16 22:51:39,385 ERROR
[OMDoubleBufferFlushThread]-org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer:
Terminating with exit status 1: During flush to DB encountered error in
OMDoubleBuffer flush thread OMDoubleBufferFlushThread when handling OMRequest:
cmdType: CreateSnapshot
traceID: ""
success: true
status: OK
CreateSnapshotResponse {
snapshotInfo {
snapshotID {
mostSigBits: -8880365051952676788
leastSigBits: -7918193529622336024
}
name: "snap339"
volumeName: "voltest21713265900"
bucketName: "buck1"
snapshotStatus: SNAPSHOT_ACTIVE
creationTime: 1713333097858
deletionTime: 18446744073709551615
pathPreviousSnapshotID {
mostSigBits: -3527873859313710896
leastSigBits: -6772710914652776790
}
globalPreviousSnapshotID {
mostSigBits: -8910008813791655688
leastSigBits: -8609017308136858185
}
snapshotPath: "voltest21713265900/buck1"
checkpointDir: "-84c29b09-1291-404c-921c-ecefec106de8"
dbTxSequenceNumber: 59906687
deepClean: true
sstFiltered: false
}
}
java.io.IOException: Rocks Database is closed
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.acquire(RocksDatabase.java:439)
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.newIterator(RocksDatabase.java:777)
at org.apache.hadoop.hdds.utils.db.RDBTable.iterator(RDBTable.java:232)
at
org.apache.hadoop.hdds.utils.db.TypedTable.iterator(TypedTable.java:418)
at
org.apache.hadoop.hdds.utils.db.TypedTable.iterator(TypedTable.java:55)
at
org.apache.hadoop.ozone.om.OmSnapshotManager.deleteKeysFromDelKeyTableInSnapshotScope(OmSnapshotManager.java:573)
at
org.apache.hadoop.ozone.om.OmSnapshotManager.createOmSnapshotCheckpoint(OmSnapshotManager.java:444)
at
org.apache.hadoop.ozone.om.response.snapshot.OMSnapshotCreateResponse.addToDBBatch(OMSnapshotCreateResponse.java:81)
at
org.apache.hadoop.ozone.om.response.OMClientResponse.checkAndUpdateDB(OMClientResponse.java:73)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$5(OzoneManagerDoubleBuffer.java:382)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatchWithTrace(OzoneManagerDoubleBuffer.java:220)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatch(OzoneManagerDoubleBuffer.java:381)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushBatch(OzoneManagerDoubleBuffer.java:324)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushCurrentBuffer(OzoneManagerDoubleBuffer.java:297)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:262)
at java.lang.Thread.run(Thread.java:748) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
