[
https://issues.apache.org/jira/browse/HDDS-11152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876092#comment-17876092
]
Hemant Kumar commented on HDDS-11152:
-------------------------------------
Based on the logs below, it happens snapshot gets purged before other
transactions on the same snapshot are not flushed by DoubleBuffer. Because in
{*}OMSnapshotPurgeRequest{*}, the snapshot is removed from the
snapshotInfoTable cache and its rocksDB instance is removed from SnapshotCache.
Snapshot created:
{code:java}
2024-08-22 06:45:28,515 INFO [om113-OMStateMachineApplyTransactionThread -
0]-org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest: Created
snapshot: 'snap1724234145' with snapshotId:
'f7f9fad6-82ed-405a-83bd-ae0e64b223c5' under path 'testvol/buckecobs'{code}
Snapshot deep cleaned:
{code:java}
2024-08-22 06:45:28,523 INFO [om113-OMStateMachineApplyTransactionThread -
0]-org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotSetPropertyRequest:
Successfully executed snapshotSetPropertyRequest: {snapshotKey:
"/testvol/buckecobs/snap1724234145"
deepCleanedDeletedKey: true
}.{code}
Snapshot deleted:
{code:java}
2024-08-22 06:45:28,649 INFO [om113-OMStateMachineApplyTransactionThread -
0]-org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotDeleteRequest: Deleted
snapshot 'snap1724234145' under path 'testvol/buckecobs'{code}
Snapshot purged:
{code:java}
2024-08-22 06:45:28,724 INFO [om113-OMStateMachineApplyTransactionThread -
0]-org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotPurgeRequest:
Successfully executed snapshotPurgeRequest: {snapshotDBKeys:
"/testvol/buckecobs/snap1724234145"
} along with updating deep clean flags for snapshots: [] and global and
previous for snapshots:[/voltest11724230537/buck4/snap97].{code}
OM crashed because it tried to open a deleted snapshot:
{code:java}
2024-08-22 06:48:44,726 ERROR
[om113-OMDoubleBufferFlushThread]-org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer:
Terminating with exit status 1: During flush to DB encountered error in
OMDoubleBuffer flush thread om113-OMDoubleBufferFlushThread when handling
OMRequest: cmdType: PurgeKeys
traceID: ""
success: true
status: OK
KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Snapshot
'/testvol/buckecobs/snap1724234145' is not found.
at
org.apache.hadoop.ozone.om.snapshot.SnapshotUtils.getSnapshotInfo(SnapshotUtils.java:84)
at
org.apache.hadoop.ozone.om.OmSnapshotManager.getSnapshot(OmSnapshotManager.java:666)
at
org.apache.hadoop.ozone.om.OmSnapshotManager.getSnapshot(OmSnapshotManager.java:661)
at
org.apache.hadoop.ozone.om.OmSnapshotManager.getSnapshot(OmSnapshotManager.java:644)
at
org.apache.hadoop.ozone.om.response.key.OMKeyPurgeResponse.addToDBBatch(OMKeyPurgeResponse.java:82)
at
org.apache.hadoop.ozone.om.response.OMClientResponse.checkAndUpdateDB(OMClientResponse.java:66)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$8(OzoneManagerDoubleBuffer.java:408)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatchWithTrace(OzoneManagerDoubleBuffer.java:253)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatch(OzoneManagerDoubleBuffer.java:407)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushBatch(OzoneManagerDoubleBuffer.java:353)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushCurrentBuffer(OzoneManagerDoubleBuffer.java:328)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:295)
at java.lang.Thread.run(Thread.java:748){code}
> OMDoubleBuffer error when handling OMRequest: cmdType: SnapshotMoveDeletedKeys
> ------------------------------------------------------------------------------
>
> Key: HDDS-11152
> URL: https://issues.apache.org/jira/browse/HDDS-11152
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM, Snapshot
> Reporter: Jyotirmoy Sinha
> Assignee: Hemant Kumar
> Priority: Major
> Labels: ozone-snapshot
>
> OMDoubleBuffer error when handling OMRequest: cmdType: SnapshotMoveDeletedKeys
> Error stacktrace -
> {code:java}
> 2024-07-11 16:39:41,419 ERROR
> [om136-OMDoubleBufferFlushThread]-org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer:
> Terminating with exit status 1: During flush to DB encountered error in
> OMDoubleBuffer flush thread om136-OMDoubleBufferFlushThread when handling
> OMRequest: cmdType: SnapshotMoveDeletedKeys
> traceID: ""
> success: true
> status: OKjava.io.IOException: Rocks Database is closed
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.acquire(RocksDatabase.java:439)
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.batchWrite(RocksDatabase.java:776)
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.batchWrite(RocksDatabase.java:785)
> at
> org.apache.hadoop.hdds.utils.db.RDBBatchOperation.commit(RDBBatchOperation.java:348)
> at
> org.apache.hadoop.hdds.utils.db.RDBStore.commitBatchOperation(RDBStore.java:285)
> at
> org.apache.hadoop.ozone.om.response.snapshot.OMSnapshotMoveDeletedKeysResponse.addToDBBatch(OMSnapshotMoveDeletedKeysResponse.java:136)
> at
> org.apache.hadoop.ozone.om.response.OMClientResponse.checkAndUpdateDB(OMClientResponse.java:66)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$8(OzoneManagerDoubleBuffer.java:408)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatchWithTrace(OzoneManagerDoubleBuffer.java:253)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatch(OzoneManagerDoubleBuffer.java:407)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushBatch(OzoneManagerDoubleBuffer.java:353)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushCurrentBuffer(OzoneManagerDoubleBuffer.java:328)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:295)
> at java.base/java.lang.Thread.run(Thread.java:834) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
