Jyotirmoy Sinha created HDDS-9714:
-------------------------------------
Summary: [snapshot] OM shutdown due to block checksum mismatch
while creating snapshot
Key: HDDS-9714
URL: https://issues.apache.org/jira/browse/HDDS-9714
Project: Apache Ozone
Issue Type: Bug
Components: Snapshot
Reporter: Jyotirmoy Sinha
OM shutdown due to block checksum mismatch while creating snapshot
OM error log stacktrace -
{code:java}
2023-11-17 01:36:32,449 INFO [OM StateMachine ApplyTransaction Thread -
0]-org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest: Created
snapshot: 'snap-ck8h9' with snapshotId: 'f6f0dc88-5937-4722-98cc-cd6e1afc0558'
under path 'voljitog/bucketjitog'
2023-11-17 01:36:33,531 ERROR
[OMDoubleBufferFlushThread]-org.apache.hadoop.hdds.utils.db.RDBCheckpointManager:
Unable to create RocksDB Snapshot.
java.io.IOException: RocksDatabase[/var/lib/hadoop-ozone/om/data167990/om.db]:
Failed to flush; status : Corruption; message : block checksum mismatch: stored
= 3301695847, computed = 750921363, type = 1 in
/var/lib/hadoop-ozone/om/data167990/om.db/000701.sst offset 0 size 317
at
org.apache.hadoop.hdds.utils.HddsServerUtil.toIOException(HddsServerUtil.java:667)
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.toIOException(RocksDatabase.java:90)
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.flush(RocksDatabase.java:504)
at
org.apache.hadoop.hdds.utils.db.RDBCheckpointManager.createCheckpoint(RDBCheckpointManager.java:81)
at
org.apache.hadoop.hdds.utils.db.RDBStore.getSnapshot(RDBStore.java:329)
at
org.apache.hadoop.ozone.om.OmSnapshotManager.createOmSnapshotCheckpoint(OmSnapshotManager.java:437)
at
org.apache.hadoop.ozone.om.response.snapshot.OMSnapshotCreateResponse.addToDBBatch(OMSnapshotCreateResponse.java:81)
at
org.apache.hadoop.ozone.om.response.OMClientResponse.checkAndUpdateDB(OMClientResponse.java:73)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$5(OzoneManagerDoubleBuffer.java:409)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatchWithTrace(OzoneManagerDoubleBuffer.java:237)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatch(OzoneManagerDoubleBuffer.java:408)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushBatch(OzoneManagerDoubleBuffer.java:335)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushCurrentBuffer(OzoneManagerDoubleBuffer.java:314)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:279)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored =
3301695847, computed = 750921363, type = 1 in
/var/lib/hadoop-ozone/om/data167990/om.db/000701.sst offset 0 size 317
at org.rocksdb.RocksDB.flush(Native Method)
at org.rocksdb.RocksDB.flush(RocksDB.java:3785)
at org.rocksdb.RocksDB.flush(RocksDB.java:3763)
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.flush(RocksDatabase.java:500)
... 12 more
2023-11-17 01:36:33,584 ERROR
[OMDoubleBufferFlushThread]-org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer:
Terminating with exit status 1: During flush to DB encountered error in
OMDoubleBuffer flush thread OMDoubleBufferFlushThread when handling OMRequest:
cmdType: CreateSnapshot
traceID: ""
success: true
status: OK
CreateSnapshotResponse {
snapshotInfo {
snapshotID {
mostSigBits: -652779467798264030
leastSigBits: -7436343011912710824
}
name: "snap-ck8h9"
volumeName: "voljitog"
bucketName: "bucketjitog"
snapshotStatus: SNAPSHOT_ACTIVE
creationTime: 1700184992446
deletionTime: 18446744073709551615
globalPreviousSnapshotID {
mostSigBits: 6265401439073879226
leastSigBits: -4917962699912174853
}
snapshotPath: "voljitog/bucketjitog"
checkpointDir: "-f6f0dc88-5937-4722-98cc-cd6e1afc0558"
dbTxSequenceNumber: 8220
deepClean: true
sstFiltered: false
}
} java.io.IOException: Rocks Database is closed
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.assertClose(RocksDatabase.java:444)
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.newIterator(RocksDatabase.java:856)
at org.apache.hadoop.hdds.utils.db.RDBTable.iterator(RDBTable.java:232)
at
org.apache.hadoop.hdds.utils.db.TypedTable.iterator(TypedTable.java:417)
at
org.apache.hadoop.hdds.utils.db.TypedTable.iterator(TypedTable.java:409)
at
org.apache.hadoop.hdds.utils.db.TypedTable.iterator(TypedTable.java:55)
at
org.apache.hadoop.ozone.om.OmSnapshotManager.deleteKeysFromDelKeyTableInSnapshotScope(OmSnapshotManager.java:637)
at
org.apache.hadoop.ozone.om.OmSnapshotManager.createOmSnapshotCheckpoint(OmSnapshotManager.java:442)
at
org.apache.hadoop.ozone.om.response.snapshot.OMSnapshotCreateResponse.addToDBBatch(OMSnapshotCreateResponse.java:81)
at
org.apache.hadoop.ozone.om.response.OMClientResponse.checkAndUpdateDB(OMClientResponse.java:73)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$5(OzoneManagerDoubleBuffer.java:409)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatchWithTrace(OzoneManagerDoubleBuffer.java:237)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatch(OzoneManagerDoubleBuffer.java:408)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushBatch(OzoneManagerDoubleBuffer.java:335)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushCurrentBuffer(OzoneManagerDoubleBuffer.java:314)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:279)
at java.base/java.lang.Thread.run(Thread.java:833) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
