[
https://issues.apache.org/jira/browse/HDDS-10548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hemant Kumar resolved HDDS-10548.
---------------------------------
Resolution: Duplicate
> [snapshot-LR] OM is getting shutdown due to Snapshot chain corruption in an
> LR setup
> ------------------------------------------------------------------------------------
>
> Key: HDDS-10548
> URL: https://issues.apache.org/jira/browse/HDDS-10548
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager, Snapshot
> Reporter: Jyotirmoy Sinha
> Priority: Major
> Labels: ozone-snapshot
>
> OM is getting shutdown due to Snapshot chain corruption in an LR setup
> Scenario :
> * Generate data over parallel threads over various volume/buckets
> * Perform parallel snapshot create/delete/list operations over above buckets
> * Perform parallel snapdiff operations over each bucket
> * Perform parallel read operations of snapshot contents
> * Introduce OM and cluster restarts in between along with DN decommissioning
> and balancer restarts.
> OM Leader error stacktrace -
> {code:java}
> 2024-03-14 04:07:13,525 INFO
> [JvmPauseMonitor0]-org.apache.ratis.util.JvmPauseMonitor:
> JvmPauseMonitor-om232: Started
> 2024-03-14 04:07:13,534 INFO [main]-org.apache.hadoop.ozone.om.OzoneManager:
> Starting secret key client.
> 2024-03-14 04:07:13,615 ERROR [OM StateMachine ApplyTransaction Thread -
> 0]-org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine: Terminating
> with exit status 1: OM Ratis Server has received unrecoverable error, to
> avoid further DB corruption, terminating OM. Error Response received
> is:cmdType: CreateSnapshot
> traceID: ""
> success: false
> message: "java.io.IOException: Snapshot chain is corrupted.\n\tat
> org.apache.hadoop.ozone.om.SnapshotChainManager.validateSnapshotChain(SnapshotChainManager.java:550)\n\tat
>
> org.apache.hadoop.ozone.om.SnapshotChainManager.getLatestPathSnapshotId(SnapshotChainManager.java:378)\n\tat
>
> org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest.addSnapshotInfoToSnapshotChainAndCache(OMSnapshotCreateRequest.java:232)\n\tat
>
> org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest.validateAndUpdateCache(OMSnapshotCreateRequest.java:162)\n\tat
>
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:378)\n\tat
>
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:568)\n\tat
>
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$1(OzoneManagerStateMachine.java:363)\n\tat
>
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)\n\tat
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat
> java.lang.Thread.run(Thread.java:748)\n"
> status: INTERNAL_ERRORINTERNAL_ERROR
> org.apache.hadoop.ozone.om.exceptions.OMException: java.io.IOException:
> Snapshot chain is corrupted.
> at
> org.apache.hadoop.ozone.om.SnapshotChainManager.validateSnapshotChain(SnapshotChainManager.java:550)
> at
> org.apache.hadoop.ozone.om.SnapshotChainManager.getLatestPathSnapshotId(SnapshotChainManager.java:378)
> at
> org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest.addSnapshotInfoToSnapshotChainAndCache(OMSnapshotCreateRequest.java:232)
> at
> org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest.validateAndUpdateCache(OMSnapshotCreateRequest.java:162)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:378)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:568)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$1(OzoneManagerStateMachine.java:363)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.terminate(OzoneManagerStateMachine.java:404)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$2(OzoneManagerStateMachine.java:379)
> at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1609)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2024-03-14 04:07:13,620 INFO
> [shutdown-hook-0]-org.apache.ranger.audit.provider.AuditProviderFactory: ==>
> JVMShutdownHook.run()
> 2024-03-14 04:07:13,621 INFO
> [shutdown-hook-0]-org.apache.ranger.audit.provider.AuditProviderFactory:
> JVMShutdownHook: Signalling async audit cleanup to start. {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
