[
https://issues.apache.org/jira/browse/HDDS-8832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732758#comment-17732758
]
Hemant Kumar edited comment on HDDS-8832 at 6/14/23 8:53 PM:
-------------------------------------------------------------
I suspect that *SnapshotChain* gets corrupted because
[get|https://github.infra.cloudera.com/CDH/ozone/blob/CDH-7.1.9.x/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java#L163-L167]
and
[add|https://github.com/apache/ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java#LL175C35-L175C42]
are not atomic operation. *get* and *add* should be single atomic operation.
Let's assume two (cloud be more) thread TD-1 and TD-2 creating snapshots on
different buckets. They arrived at the same time at OM and can be executed in
parallel because snapshots are for different buckets and lock is acquired at
bucket and snapshot level.
Scenario-1:
Time-1: Bucket and snapshot locks are acquired by TD-1 and TD-1 gets the global
previous and path level previous snapshotIds. Lets assume it is first snapshot
even snapshot hence both would be null.
Time-2: TD-1 adds the new snapshot (SnapshotId: *SnapId-1*) to SnapshotChain.
Now GlobalPrevious is *SnapId-1* and global chain is *SnapId-1*.
Time-3: Bucket and snapshot locks are acquired by TD-2 and TD-2 gets the global
previous and path level previous snapshotIds. Since TD-1 added SnapId-1 to the
chain, global previous would be SnapId-1 and local previous would be null.
Time-4: TD-2 adds the new snapshot (SnapshotId: *SnapId-2*) to SnapshotChain.
Now GlobalPrevious is *SnapId-2* and global chain is *SnapId-1* -> *SnapId-2*.
Time-5: TD-1 fails to update cache/DB. And should be removed from the chain.
(It won't be removed from the *SnapshotChain* because we have [this
check|https://github.com/apache/ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java#L197]).
Scenario-2:
Time-1: Bucket and snapshot locks are acquired by TD-1 and TD-1 gets the global
previous and path level previous snapshotIds. Lets assume it is first snapshot
even snapshot hence both would be null.
Time-2: Bucket and snapshot locks are acquired by TD-2 and TD-2 gets the global
previous and path level previous snapshotIds. It will get the same result as
TD-1.
Time-3: TD-1 adds the new snapshot (SnapshotId: *SnapId-1*) to SnapshotChain.
Now GlobalPrevious is *SnapId-1* and global chain is *SnapId-1*. And previous
of *SnapId-1* is null.
Time-4: TD-2 adds the new snapshot (SnapshotId: *SnapId-2*) to SnapshotChain.
Now GlobalPrevious is *SnapId-2* and global chain is *SnapId-1* -> *SnapId-2*.
And previous of *SnapId-2* is also null.
In scenario-1, *SnapId-2*'s previous is pointing to *SnapId-1* which doesn't
exist. In scenario-1, both *SnapId-1* and *SnapId-2*'s previous are pointing
to null.
Both scenarios are caused because *get* and *add* are separate and non-atomic
operation.
was (Author: JIRAUSER297350):
I suspect that *SnapshotChain* gets corrupted because
[get|https://github.infra.cloudera.com/CDH/ozone/blob/CDH-7.1.9.x/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java#L163-L167]
and
[add|https://github.com/apache/ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java#LL175C35-L175C42]
are not atomic operation. *get* and *add* should be single atomic operation.
Let's assume two (cloud be more) thread TD-1 and TD-2 creating snapshots on
different buckets. They arrived at the same time at OM and can be executed in
parallel because snapshots are for different buckets and lock is acquired at
bucket and snapshot level.
Scenario-1:
Time-1: Bucket and snapshot locks are acquired by TD-1 and TD-1 gets the global
previous and path level previous snapshotIds. Lets assume it is first snapshot
even snapshot hence both would be null.
Time-2: TD-1 adds the new snapshot (SnapshotId: *SnapId-1*) to SnapshotChain.
Now GlobalPrevious is *SnapId-1* and global chain is *SnapId-1*.
Time-3: Bucket and snapshot locks are acquired by TD-2 and TD-2 gets the global
previous and path level previous snapshotIds. Since TD-1 added SnapId-1 to the
chain, global previous would be SnapId-1 and local previous would be null.
Time-4: TD-2 adds the new snapshot (SnapshotId: *SnapId-2*) to SnapshotChain.
Now GlobalPrevious is *SnapId-2* and global chain is *SnapId-1* -> *SnapId-2*.
Time-5: TD-1 fails to update cache/DB. And should be removed from the chain.
(It won't be removed from the *SnapshotChain* because we have [this
check|https://github.com/apache/ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java#L197]).
Scenario-2:
Time-1: Bucket and snapshot locks are acquired by TD-1 and TD-1 gets the global
previous and path level previous snapshotIds. Lets assume it is first snapshot
even snapshot hence both would be null.
Time-2: Bucket and snapshot locks are acquired by TD-2 and TD-2 gets the global
previous and path level previous snapshotIds. It will get the same result as
TD-1.
Time-3: TD-1 adds the new snapshot (SnapshotId: *SnapId-1*) to SnapshotChain.
Now GlobalPrevious is *SnapId-1* and global chain is *SnapId-1*. And previous
of *SnapId-1* is null.
Time-4: TD-2 adds the new snapshot (SnapshotId: *SnapId-2*) to SnapshotChain.
Now GlobalPrevious is *SnapId-2* and global chain is *SnapId-1* -> *SnapId-2*.
And previous of *SnapId-2* is also null.
In scenario-1, *SnapId-2*'s previous is pointing to *SnapId-1* which doesn't
exist. In scenario-1, both *SnapId-1* and *SnapId-2*'s previous are pointing
to null.
Both scenario are caused because *get* and *add* are separate and non-atomic
operation.
> [snapshot] OM start failed with exception "Snapshot chain corruption"
> ---------------------------------------------------------------------
>
> Key: HDDS-8832
> URL: https://issues.apache.org/jira/browse/HDDS-8832
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Jyotirmoy Sinha
> Assignee: Hemant Kumar
> Priority: Major
> Labels: ozone-snapshot
>
> Created 6000+ snapshots, and then restarted OM.
> OM error stacktrace -
> {code:java}
> 2023-06-13 05:44:39,448 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '023153'
> 2023-06-13 05:44:39,457 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '023184'
> 2023-06-13 05:44:39,459 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '023200'
> 2023-06-13 05:44:39,462 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '023213'
> 2023-06-13 05:44:39,463 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '023226'
> 2023-06-13 05:44:39,465 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '023239'
> 2023-06-13 05:44:39,467 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '023252'
> 2023-06-13 05:44:39,469 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '023265'
> 2023-06-13 05:44:39,471 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '023278'
> 2023-06-13 05:44:39,473 [main] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Can't find SST '023291'
> 2023-06-13 05:44:39,571 [Thread-8385] INFO
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Skipped the compaction
> entry. Compaction input files:
> [/var/lib/hadoop-ozone/om/data/om.db/023299.sst] and output files:
> [/var/lib/hadoop-ozone/om/data/om.db/023299.sst] are same.
> 2023-06-13 05:44:40,681 [main] ERROR
> org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with exception
> java.io.IOException: Snapshot chain corruption. Previous snapshotId:
> 115d7882-8831-477d-8497-cdef7235d811 is set for snapshotId:
> 69d40951-6008-495d-8a0e-90ef9c4ac54a but no associated snapshot found in
> snapshot chain.
> at
> org.apache.hadoop.ozone.om.SnapshotChainManager.addSnapshotGlobal(SnapshotChainManager.java:87)
> at
> org.apache.hadoop.ozone.om.SnapshotChainManager.addSnapshot(SnapshotChainManager.java:283)
> at
> org.apache.hadoop.ozone.om.SnapshotChainManager.loadFromSnapshotInfoTable(SnapshotChainManager.java:273)
> at
> org.apache.hadoop.ozone.om.SnapshotChainManager.<init>(SnapshotChainManager.java:64)
> at
> org.apache.hadoop.ozone.om.OmMetadataManagerImpl.start(OmMetadataManagerImpl.java:519)
> at
> org.apache.hadoop.ozone.om.OmMetadataManagerImpl.<init>(OmMetadataManagerImpl.java:312)
> at
> org.apache.hadoop.ozone.om.OzoneManager.instantiateServices(OzoneManager.java:787)
> at
> org.apache.hadoop.ozone.om.OzoneManager.<init>(OzoneManager.java:667)
> at
> org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:752)
> at
> org.apache.hadoop.ozone.om.OzoneManagerStarter$OMStarterHelper.start(OzoneManagerStarter.java:189)
> at
> org.apache.hadoop.ozone.om.OzoneManagerStarter.startOm(OzoneManagerStarter.java:86)
> at
> org.apache.hadoop.ozone.om.OzoneManagerStarter.call(OzoneManagerStarter.java:74)
> at org.apache.hadoop.hdds.cli.GenericCli.call(GenericCli.java:38)
> at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
> at picocli.CommandLine.access$1300(CommandLine.java:145)
> at
> picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
> at
> picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
> at picocli.CommandLine.execute(CommandLine.java:2078)
> at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100)
> at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91)
> at
> org.apache.hadoop.ozone.om.OzoneManagerStarter.main(OzoneManagerStarter.java:58)
> 2023-06-13 05:44:40,693 [shutdown-hook-0] INFO
> org.apache.hadoop.ozone.om.OzoneManagerStarter: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down OzoneManager at
> quasar-exotla-1.quasar-exotla.root.hwx.site/172.27.89.207
> ************************************************************/ {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
