[
https://issues.apache.org/jira/browse/HDDS-7913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sadanand Shenoy updated HDDS-7913:
----------------------------------
Labels: ozone-snapshot (was: )
> 'Graph traversal level exceeded allowed maximum' when the difference b/w
> snapshots is more than 3.
> --------------------------------------------------------------------------------------------------
>
> Key: HDDS-7913
> URL: https://issues.apache.org/jira/browse/HDDS-7913
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Sadanand Shenoy
> Priority: Major
> Labels: ozone-snapshot
>
> Getting the below error when calling snapdiff b/w snapshots which are more
> than 2-3 snapshots apart.
> {code:java}
> 2023-02-07 10:50:07,464 [IPC Server handler 13 on default port 60192] ERROR
> rocksdiff.RocksDBCheckpointDiffer
> (RocksDBCheckpointDiffer.java:internalGetSSTDiffList(899)) - Graph traversal
> level exceeded allowed maximum (1000000). This could be due to invalid input
> generating a loop in the traversal path. Same SSTs found so far: [000053],
> different SSTs: []
> 2023-02-07 10:50:07,465 [IPC Server handler 13 on default port 60192] WARN
> ipc.Server (Server.java:logException(3035)) - IPC Server handler 13 on
> default port 60192, call Call#102 Retry#159
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest from
> 127.0.0.1:60273
> java.lang.RuntimeException: Graph traversal level exceeded allowed maximum
> (1000000). This could be due to invalid input generating a loop in the
> traversal path. Same SSTs found so far: [000053], different SSTs: []
> at
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.internalGetSSTDiffList(RocksDBCheckpointDiffer.java:904)
> at
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffList(RocksDBCheckpointDiffer.java:788)
> at
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffListWithFullPath(RocksDBCheckpointDiffer.java:756)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFiles(SnapshotDiffManager.java:231)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getSnapshotDiffReport(SnapshotDiffManager.java:134)
> at
> org.apache.hadoop.ozone.om.OmSnapshotManager.getSnapshotDiffReport(OmSnapshotManager.java:250)
> at
> org.apache.hadoop.ozone.om.OzoneManager.snapshotDiff(OzoneManager.java:4325)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.snapshotDiff(OzoneManagerRequestHandler.java:1216)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleReadRequest(OzoneManagerRequestHandler.java:298)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:223)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:177)
> at
> org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:147)
> at
> org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:465)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:578)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976) {code}
> Unit test to repro in *TestOmSnapshot.java*
> {code:java}
> @Test
> public void testSnapDiffWithMultipleSSTs()
> throws IOException, InterruptedException, TimeoutException {
> String volumeName1 = "vol-" + RandomStringUtils.randomNumeric(5);
> String bucketName1 = "buck1";
> String bucketName2 = "bucketz";
> store.createVolume(volumeName1);
> OzoneVolume volume1 = store.getVolume(volumeName1);
> volume1.createBucket(bucketName1);
> volume1.createBucket(bucketName2);
> OzoneBucket bucket1 = volume1.getBucket(bucketName1);
> OzoneBucket bucket2 = volume1.getBucket(bucketName2);
> String keyPrefix = "key-";
> createFileKey(bucket1, keyPrefix);
> String snap1 = "snap" + RandomStringUtils.randomNumeric(5);
> createSnapshot(volumeName1, bucketName1, snap1);
> createFileKey(bucket2, keyPrefix);
> String snap2 = "snap" + RandomStringUtils.randomNumeric(5);
> createSnapshot(volumeName1, bucketName1, snap2);
> createFileKey(bucket2, keyPrefix);
> String snap3 = "snap" + RandomStringUtils.randomNumeric(5);
> createSnapshot(volumeName1, bucketName1, snap3);
> createFileKey(bucket1,keyPrefix);
> String snap4 = "snap" + RandomStringUtils.randomNumeric(5);
> createSnapshot(volumeName1, bucketName1, snap4);
> SnapshotDiffReport diff1 =
> store.snapshotDiff(volumeName1, bucketName1, snap1, snap4);
> }
> {code}
> cc [~smeng]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
