[
https://issues.apache.org/jira/browse/HDDS-7913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sadanand Shenoy updated HDDS-7913:
----------------------------------
Description:
Getting the below error when calling snapdiff b/w snapshots which are more than
2-3 snapshots apart.
{code:java}
2023-02-07 10:50:07,464 [IPC Server handler 13 on default port 60192] ERROR
rocksdiff.RocksDBCheckpointDiffer
(RocksDBCheckpointDiffer.java:internalGetSSTDiffList(899)) - Graph traversal
level exceeded allowed maximum (1000000). This could be due to invalid input
generating a loop in the traversal path. Same SSTs found so far: [000053],
different SSTs: []
2023-02-07 10:50:07,465 [IPC Server handler 13 on default port 60192] WARN
ipc.Server (Server.java:logException(3035)) - IPC Server handler 13 on default
port 60192, call Call#102 Retry#159
org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest from
127.0.0.1:60273
java.lang.RuntimeException: Graph traversal level exceeded allowed maximum
(1000000). This could be due to invalid input generating a loop in the
traversal path. Same SSTs found so far: [000053], different SSTs: []
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.internalGetSSTDiffList(RocksDBCheckpointDiffer.java:904)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffList(RocksDBCheckpointDiffer.java:788)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffListWithFullPath(RocksDBCheckpointDiffer.java:756)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFiles(SnapshotDiffManager.java:231)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getSnapshotDiffReport(SnapshotDiffManager.java:134)
at
org.apache.hadoop.ozone.om.OmSnapshotManager.getSnapshotDiffReport(OmSnapshotManager.java:250)
at
org.apache.hadoop.ozone.om.OzoneManager.snapshotDiff(OzoneManager.java:4325)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.snapshotDiff(OzoneManagerRequestHandler.java:1216)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleReadRequest(OzoneManagerRequestHandler.java:298)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:223)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:177)
at
org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:147)
at
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:465)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:578)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976) {code}
Unit test to repro in *TestOmSnapshot.java*
{code:java}
@Test
public void testSnapDiffWithMultipleSSTs()
throws IOException, InterruptedException, TimeoutException {
String volumeName1 = "vol-" + RandomStringUtils.randomNumeric(5);
String bucketName1 = "buck1";
String bucketName2 = "bucketz";
store.createVolume(volumeName1);
OzoneVolume volume1 = store.getVolume(volumeName1);
volume1.createBucket(bucketName1);
volume1.createBucket(bucketName2);
OzoneBucket bucket1 = volume1.getBucket(bucketName1);
OzoneBucket bucket2 = volume1.getBucket(bucketName2);
String keyPrefix = "key-";
createFileKey(bucket1, keyPrefix);
String snap1 = "snap" + RandomStringUtils.randomNumeric(5);
createSnapshot(volumeName1, bucketName1, snap1);
createFileKey(bucket2, keyPrefix);
String snap2 = "snap" + RandomStringUtils.randomNumeric(5);
createSnapshot(volumeName1, bucketName1, snap2);
createFileKey(bucket2, keyPrefix);
String snap3 = "snap" + RandomStringUtils.randomNumeric(5);
createSnapshot(volumeName1, bucketName1, snap3);
createFileKey(bucket1,keyPrefix);
String snap4 = "snap" + RandomStringUtils.randomNumeric(5);
createSnapshot(volumeName1, bucketName1, snap4);
SnapshotDiffReport diff1 =
store.snapshotDiff(volumeName1, bucketName1, snap1, snap4);
}
{code}
was:
Getting the below error when calling snapdiff b/w snapshots which are more than
2-3 snapshots apart.
{code:java}
2023-02-07 10:50:07,464 [IPC Server handler 13 on default port 60192] ERROR
rocksdiff.RocksDBCheckpointDiffer
(RocksDBCheckpointDiffer.java:internalGetSSTDiffList(899)) - Graph traversal
level exceeded allowed maximum (1000000). This could be due to invalid input
generating a loop in the traversal path. Same SSTs found so far: [000053],
different SSTs: []
2023-02-07 10:50:07,465 [IPC Server handler 13 on default port 60192] WARN
ipc.Server (Server.java:logException(3035)) - IPC Server handler 13 on default
port 60192, call Call#102 Retry#159
org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest from
127.0.0.1:60273
java.lang.RuntimeException: Graph traversal level exceeded allowed maximum
(1000000). This could be due to invalid input generating a loop in the
traversal path. Same SSTs found so far: [000053], different SSTs: []
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.internalGetSSTDiffList(RocksDBCheckpointDiffer.java:904)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffList(RocksDBCheckpointDiffer.java:788)
at
org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffListWithFullPath(RocksDBCheckpointDiffer.java:756)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFiles(SnapshotDiffManager.java:231)
at
org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getSnapshotDiffReport(SnapshotDiffManager.java:134)
at
org.apache.hadoop.ozone.om.OmSnapshotManager.getSnapshotDiffReport(OmSnapshotManager.java:250)
at
org.apache.hadoop.ozone.om.OzoneManager.snapshotDiff(OzoneManager.java:4325)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.snapshotDiff(OzoneManagerRequestHandler.java:1216)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleReadRequest(OzoneManagerRequestHandler.java:298)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:223)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:177)
at
org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:147)
at
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:465)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:578)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976) {code}
Unit test to repro
{code:java}
@Test
public void testSnapDiffWithMultipleSSTs()
throws IOException, InterruptedException, TimeoutException {
String volumeName1 = "vol-" + RandomStringUtils.randomNumeric(5);
String bucketName1 = "buck1";
String bucketName2 = "bucketz";
store.createVolume(volumeName1);
OzoneVolume volume1 = store.getVolume(volumeName1);
volume1.createBucket(bucketName1);
volume1.createBucket(bucketName2);
OzoneBucket bucket1 = volume1.getBucket(bucketName1);
OzoneBucket bucket2 = volume1.getBucket(bucketName2);
String keyPrefix = "key-";
createFileKey(bucket1, keyPrefix);
String snap1 = "snap" + RandomStringUtils.randomNumeric(5);
createSnapshot(volumeName1, bucketName1, snap1); // 1.sst
createFileKey(bucket2, keyPrefix);
String snap2 = "snap" + RandomStringUtils.randomNumeric(5);
createSnapshot(volumeName1, bucketName1, snap2); // 1.sst
createFileKey(bucket2, keyPrefix);
String snap3 = "snap" + RandomStringUtils.randomNumeric(5);
createSnapshot(volumeName1, bucketName1, snap3); // 1.sst
createFileKey(bucket1,keyPrefix);
String snap4 = "snap" + RandomStringUtils.randomNumeric(5);
createSnapshot(volumeName1, bucketName1, snap4); // 1.sst 2.sst 3.sst 4.sst
SnapshotDiffReport diff1 =
store.snapshotDiff(volumeName1, bucketName1, snap1, snap4);
Assert.assertEquals(2, diff1.getDiffList().size());
}
{code}
> 'Graph traversal level exceeded allowed maximum' when the difference b/w
> snapshots is more than 3.
> --------------------------------------------------------------------------------------------------
>
> Key: HDDS-7913
> URL: https://issues.apache.org/jira/browse/HDDS-7913
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Sadanand Shenoy
> Priority: Major
>
> Getting the below error when calling snapdiff b/w snapshots which are more
> than 2-3 snapshots apart.
> {code:java}
> 2023-02-07 10:50:07,464 [IPC Server handler 13 on default port 60192] ERROR
> rocksdiff.RocksDBCheckpointDiffer
> (RocksDBCheckpointDiffer.java:internalGetSSTDiffList(899)) - Graph traversal
> level exceeded allowed maximum (1000000). This could be due to invalid input
> generating a loop in the traversal path. Same SSTs found so far: [000053],
> different SSTs: []
> 2023-02-07 10:50:07,465 [IPC Server handler 13 on default port 60192] WARN
> ipc.Server (Server.java:logException(3035)) - IPC Server handler 13 on
> default port 60192, call Call#102 Retry#159
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest from
> 127.0.0.1:60273
> java.lang.RuntimeException: Graph traversal level exceeded allowed maximum
> (1000000). This could be due to invalid input generating a loop in the
> traversal path. Same SSTs found so far: [000053], different SSTs: []
> at
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.internalGetSSTDiffList(RocksDBCheckpointDiffer.java:904)
> at
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffList(RocksDBCheckpointDiffer.java:788)
> at
> org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffListWithFullPath(RocksDBCheckpointDiffer.java:756)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFiles(SnapshotDiffManager.java:231)
> at
> org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getSnapshotDiffReport(SnapshotDiffManager.java:134)
> at
> org.apache.hadoop.ozone.om.OmSnapshotManager.getSnapshotDiffReport(OmSnapshotManager.java:250)
> at
> org.apache.hadoop.ozone.om.OzoneManager.snapshotDiff(OzoneManager.java:4325)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.snapshotDiff(OzoneManagerRequestHandler.java:1216)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleReadRequest(OzoneManagerRequestHandler.java:298)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:223)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:177)
> at
> org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:147)
> at
> org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:465)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:578)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976) {code}
> Unit test to repro in *TestOmSnapshot.java*
> {code:java}
> @Test
> public void testSnapDiffWithMultipleSSTs()
> throws IOException, InterruptedException, TimeoutException {
> String volumeName1 = "vol-" + RandomStringUtils.randomNumeric(5);
> String bucketName1 = "buck1";
> String bucketName2 = "bucketz";
> store.createVolume(volumeName1);
> OzoneVolume volume1 = store.getVolume(volumeName1);
> volume1.createBucket(bucketName1);
> volume1.createBucket(bucketName2);
> OzoneBucket bucket1 = volume1.getBucket(bucketName1);
> OzoneBucket bucket2 = volume1.getBucket(bucketName2);
> String keyPrefix = "key-";
> createFileKey(bucket1, keyPrefix);
> String snap1 = "snap" + RandomStringUtils.randomNumeric(5);
> createSnapshot(volumeName1, bucketName1, snap1);
> createFileKey(bucket2, keyPrefix);
> String snap2 = "snap" + RandomStringUtils.randomNumeric(5);
> createSnapshot(volumeName1, bucketName1, snap2);
> createFileKey(bucket2, keyPrefix);
> String snap3 = "snap" + RandomStringUtils.randomNumeric(5);
> createSnapshot(volumeName1, bucketName1, snap3);
> createFileKey(bucket1,keyPrefix);
> String snap4 = "snap" + RandomStringUtils.randomNumeric(5);
> createSnapshot(volumeName1, bucketName1, snap4);
> SnapshotDiffReport diff1 =
> store.snapshotDiff(volumeName1, bucketName1, snap1, snap4);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
