ArafatKhan2198 commented on code in PR #6503:
URL: https://github.com/apache/ozone/pull/6503#discussion_r1582361820
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBSAndLegacy.java:
##########
@@ -904,6 +975,101 @@ public void testNormalizePathUptoBucket() {
OmUtils.normalizePathUptoBucket("volume/bucket/key$%#1/./////////key$%#2"));
}
+ @Test
+ public void testListKeysBucketFive() throws Exception {
+ // filter list keys under bucketFive based on RATIS ReplicationConfig and
key creation date
+ // creationDate filter passed 1 minute above of KEY6 creation date, so
listKeys API will return
+ // ZERO keys, as one RATIS keys got created after creationDate filter
value.
+ Response bucketResponse = nsSummaryEndpoint.listKeysWithDu("RATIS",
+ "04-04-2024 12:31:00", 0, BUCKET_FIVE_PATH, 10, false);
+ DUResponse duBucketResponse = (DUResponse) bucketResponse.getEntity();
+ // There are no sub-paths under this OBS bucket.
+ assertEquals(1, duBucketResponse.getCount());
+
+ // creationDate filter and keySize filter both are empty, so listKeys API
should return both KEY6 and KEY7 keys,
+ bucketResponse = nsSummaryEndpoint.listKeysWithDu("RATIS",
+ null, 0, BUCKET_FIVE_PATH, 10, false);
+ duBucketResponse = (DUResponse) bucketResponse.getEntity();
+ // There are no sub-paths under this OBS bucket.
+ assertEquals(2, duBucketResponse.getCount());
+ assertEquals(KEY_SIX, duBucketResponse.getDuData().get(0).getSubpath());
+
+ // creationDate filter passed same as KEY6 creation date, so listKeys API
will return
+ // KEY6 and KEY7 keys, as only 2 RATIS keys created at "04-04-2024
12:30:00".
+ bucketResponse = nsSummaryEndpoint.listKeysWithDu("RATIS",
+ "04-04-2024 12:30:00", 0, BUCKET_FIVE_PATH, 10, false);
+ duBucketResponse = (DUResponse) bucketResponse.getEntity();
+ // There are no sub-paths under this OBS bucket.
+ assertEquals(2, duBucketResponse.getCount());
+ assertEquals(KEY_SIX, duBucketResponse.getDuData().get(0).getSubpath());
+
+ // creationDate filter passed same as KEY6 and KEY7 creation date, but
replicationType filter is EC,
+ // so listKeys API will return zero keys, because no EC key got created at
or after creationDate filter value.
+ bucketResponse = nsSummaryEndpoint.listKeysWithDu("EC",
+ "04-04-2024 12:30:00", 0, BUCKET_FIVE_PATH, 10, false);
+ duBucketResponse = (DUResponse) bucketResponse.getEntity();
+ // There are no sub-paths under this OBS bucket.
+ assertEquals(0, duBucketResponse.getCount());
+
+ // creationDate filter passed same as KEY7 creation date, but
replicationType filter is RATIS,
+ // so listKeys API will return ZERO keys, as only 1 RATIS key got created
at or after creationDate filter value.
+ bucketResponse = nsSummaryEndpoint.listKeysWithDu("RATIS",
+ "04-05-2024 12:30:00", 0, BUCKET_FIVE_PATH, 10, false);
+ duBucketResponse = (DUResponse) bucketResponse.getEntity();
+ // There are no sub-paths under this OBS bucket.
+ assertEquals(1, duBucketResponse.getCount());
+
+ // creationDate filter passed same as KEY6 creation date, and
replicationType filter is RATIS,
+ // so listKeys API will return only KEY6, as only one RATIS key got
created at or after creationDate filter value,
+ // but since keySize filter value is 110 bytes and all RATIS keys created
are of size 100 bytes, so KEY6 will be
+ // filtered out and API will return ZERO keys.
+ bucketResponse = nsSummaryEndpoint.listKeysWithDu("RATIS",
+ "04-04-2024 12:30:00", 110, BUCKET_FIVE_PATH, 10, false);
+ duBucketResponse = (DUResponse) bucketResponse.getEntity();
+ // There are no sub-paths under this OBS bucket.
+ assertEquals(0, duBucketResponse.getCount());
+
+ // creationDate filter passed same as KEY6 creation date, and
replicationType filter is EC,
+ // so listKeys API will return only KEY7, as only one EC key got created
at or after creationDate filter value,
+ // but since keySize filter value is 110 bytes and all EC keys created are
of size 100 bytes, so KEY7 will be
+ // filtered out and API will return ZERO keys.
+ bucketResponse = nsSummaryEndpoint.listKeysWithDu("EC",
+ "04-04-2024 12:30:00", 110, BUCKET_FIVE_PATH, 10, false);
+ duBucketResponse = (DUResponse) bucketResponse.getEntity();
+ // There are no sub-paths under this OBS bucket.
+ assertEquals(0, duBucketResponse.getCount());
+
+ assertEquals(BUCKET_FIVE_DATA_SIZE, duBucketResponse.getSize());
+ }
+
+ @Test
+ public void testListKeysBucketSix() throws Exception {
+ // filter list keys under bucketSix based on RATIS ReplicationConfig and
key creation date
+ Response bucketResponse = nsSummaryEndpoint.listKeysWithDu("RATIS",
+ "04-04-2024 12:20:00", 0, BUCKET_SIX_PATH, 10, false);
+ DUResponse duBucketResponse = (DUResponse) bucketResponse.getEntity();
+ // There are no sub-paths under this LEGACY bucket.
+ assertEquals(2, duBucketResponse.getCount());
Review Comment:
We are mentioning that there are no sub paths under this legacy bucket but
we are asserting that the `subPathCount` returned by
`duBucketResponse.getCount()` returns **2**
Or does the comment is mentioning something else that I have mistaken?
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithLegacy.java:
##########
@@ -134,6 +134,7 @@ public class TestNSSummaryEndpointWithLegacy {
private static final String BUCKET_TWO = "bucket2";
private static final String BUCKET_THREE = "bucket3";
private static final String BUCKET_FOUR = "bucket4";
+ private static final String BUCKET_FIVE = "bucket5";
Review Comment:
Are we asserting anything in the test class
`TestNSSummaryEndpointWithLegacy`? We have created a bunch of new files and
directories but have not asserted anything yet.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/FSOBucketHandler.java:
##########
@@ -146,49 +145,116 @@ public long calculateDUUnderObject(long parentId)
break;
}
OmKeyInfo keyInfo = kv.getValue();
+ if (recursive) {
+ if (stats.getLimit() == -1) {
+ populateDiskUsage(keyInfo, diskUsageList);
+ } else {
+ if (stats.getCurrentCount() < stats.getLimit()) {
+ populateDiskUsage(keyInfo, diskUsageList);
+ stats.setCurrentCount(stats.getCurrentCount() + 1);
+ stats.setLastKey(kv.getKey());
+ }
+ }
+ }
if (keyInfo != null) {
+ stats.setTotalCount(stats.getTotalCount() + 1);
totalDU += keyInfo.getReplicatedSize();
}
}
}
// handle nested keys (DFS)
NSSummary nsSummary = getReconNamespaceSummaryManager()
- .getNSSummary(parentId);
+ .getNSSummary(parentId);
// empty bucket
if (nsSummary == null) {
return 0;
}
Set<Long> subDirIds = nsSummary.getChildDir();
for (long subDirId: subDirIds) {
- totalDU += calculateDUUnderObject(subDirId);
+ totalDU += calculateDUUnderObject(subDirId, recursive, diskUsageList,
stats);
}
return totalDU;
}
+ private void populateDiskUsage(OmKeyInfo keyInfo, List<DUResponse.DiskUsage>
diskUsageList) throws IOException {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ diskUsage.setKey(keyInfo.isFile());
+ diskUsage.setSubpath(constructFullPath(keyInfo,
getReconNamespaceSummaryManager()));
+ diskUsage.setSize(keyInfo.getDataSize());
+ diskUsage.setSizeWithReplica(keyInfo.getReplicatedSize());
+
diskUsage.setReplicationType(keyInfo.getReplicationConfig().getReplicationType().name());
+ diskUsage.setCreationTime(keyInfo.getCreationTime());
+ diskUsage.setModificationTime(keyInfo.getModificationTime());
+
+ diskUsageList.add(diskUsage);
+ }
+
+ /**
+ * Constructs the full path of a key from its OmKeyInfo using a bottom-up
approach, starting from the leaf node.
+ * <p>
+ * The method begins with the leaf node (the key itself) and recursively
prepends parent directory names, fetched
+ * via NSSummary objects, until reaching the parent bucket (parentId is -1).
It effectively builds the path from
+ * bottom to top, finally prepending the volume and bucket names to complete
the full path.
+ *
+ * @param omKeyInfo The OmKeyInfo object for the key
+ * @return The constructed full path of the key as a String.
+ * @throws IOException
+ */
+ public static String constructFullPath(OmKeyInfo omKeyInfo,
Review Comment:
While reviewing this PR I noticed that some of the code from [PR
6492](https://github.com/apache/ozone/pull/6492) is also included in your PR: ,
such as the introduction of parent ID and construction of the full path. I
believe we need these changes for your pull request. We could always rebase it
once [PR 6492](https://github.com/apache/ozone/pull/6492) gets merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
