sumitagrawl commented on code in PR #6503:
URL: https://github.com/apache/ozone/pull/6503#discussion_r1587073910
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/FSOBucketHandler.java:
##########
@@ -146,49 +145,116 @@ public long calculateDUUnderObject(long parentId)
break;
}
OmKeyInfo keyInfo = kv.getValue();
+ if (recursive) {
+ if (stats.getLimit() == -1) {
+ populateDiskUsage(keyInfo, diskUsageList);
+ } else {
+ if (stats.getCurrentCount() < stats.getLimit()) {
+ populateDiskUsage(keyInfo, diskUsageList);
+ stats.setCurrentCount(stats.getCurrentCount() + 1);
+ stats.setLastKey(kv.getKey());
Review Comment:
for list file, LastKey can be one of file, but seems current logic do not
support iteration continue from that point, eg:
- lets list file for a volume /a
- limit reached for one of file /a/b/c/d/1.txt and this is last key
- How this will continue listing further keys and other bucket files if
given as input ?
Shall we use other listing operation as we use from rocks db iterator ?
which is lexicographic ordering ?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NSSummaryEndpoint.java:
##########
@@ -128,8 +139,110 @@ public Response getDiskUsage(@QueryParam("path") String
path,
reconNamespaceSummaryManager,
omMetadataManager, reconSCM, path);
- duResponse = handler.getDuResponse(listFile, withReplica, sortSubpaths);
+ duResponse = handler.getDuResponse(listFile, withReplica, sortSubpaths,
false, new Stats(-1));
+
+ return Response.ok(duResponse).build();
+ }
+
+ /**
+ * This API will list out limited 'count' number of keys after applying
below filters in API parameters:
+ * Default Values of API param filters:
+ * -- replicationType - empty string and filter will not be applied, so
list out all keys irrespective of
+ * replication type.
+ * -- creationTime - empty string and filter will not be applied, so list
out keys irrespective of age.
+ * -- keySize - 0 bytes, which means all keys greater than zero bytes
will be listed, effectively all.
+ * -- startPrefix - /
+ * -- count - 1000
+ *
+ * @param replicationType Filter for RATIS or EC replication keys
+ * @param creationDate Filter for keys created after creationDate in
"MM-dd-yyyy HH:mm:ss" string format.
+ * @param keySize Filter for Keys greater than keySize in bytes.
+ * @param startPrefix Filter for startPrefix path.
+ * @param limit Filter for limited count of keys.
+ * @param recursive listing out keys recursively for FSO buckets.
+ * @return the list of keys in below structured format:
+ * Response For OBS Bucket keys:
+ * ********************************************************
+ * {
+ * "status": "OK",
+ * "path": "/volume1/obs-bucket/",
+ * "size": 73400320,
+ * "sizeWithReplica": 81788928,
+ * "subPathCount": 1,
+ * "totalKeyCount": 7,
+ * "lastKey": "/volume1/obs-bucket/key7",
+ * "subPaths": [
+ * {
+ * "key": true,
+ * "path": "key1",
+ * "size": 10485760,
+ * "sizeWithReplica": 18874368,
+ * "isKey": true,
+ * "replicationType": "RATIS",
+ * "creationTime": 1712321367060,
+ * "modificationTime": 1712321368190
+ * },
+ * {
+ * "key": true,
+ * "path": "key7",
+ * "size": 10485760,
+ * "sizeWithReplica": 18874368,
+ * "isKey": true,
+ * "replicationType": "EC",
+ * "creationTime": 1713261005555,
+ * "modificationTime": 1713261006728
+ * }
+ * ],
+ * "sizeDirectKey": 73400320
+ * }
+ * ********************************************************
+ * @throws IOException
+ */
+ @GET
+ @Path("/listKeys")
+ @SuppressWarnings("methodlength")
+ public Response listKeysWithDu(@QueryParam("replicationType") String
replicationType,
+ @QueryParam("creationDate") String
creationDate,
+ @DefaultValue(DEFAULT_KEY_SIZE)
@QueryParam("keySize") long keySize,
+ @DefaultValue(OM_KEY_PREFIX)
@QueryParam("startPrefix") String startPrefix,
+ @DefaultValue(DEFAULT_FETCH_COUNT)
@QueryParam("count") long limit,
+ @DefaultValue("false")
@QueryParam("recursive") boolean recursive)
+ throws IOException {
+
+ if (startPrefix == null || startPrefix.length() == 0) {
+ return Response.status(Response.Status.BAD_REQUEST).build();
+ }
+ DUResponse duResponse = new DUResponse();
+ if (!isInitializationComplete()) {
+ duResponse.setStatus(ResponseStatus.INITIALIZING);
+ return Response.ok(duResponse).build();
+ }
+ EntityHandler handler = EntityHandler.getEntityHandler(
+ reconNamespaceSummaryManager,
+ omMetadataManager, reconSCM, startPrefix);
+
+ Stats stats = new Stats(limit);
+
+ duResponse = handler.getListKeysResponse(stats, recursive);
Review Comment:
Instead of overloading DuReponse class, we can define new response class for
listFile purpose and return only required information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
