ArafatKhan2198 commented on code in PR #6658:
URL: https://github.com/apache/ozone/pull/6658#discussion_r1605972314
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/OMDBInsightEndpoint.java:
##########
@@ -674,6 +685,720 @@ public Response getDeletedDirectorySummary() {
return Response.ok(dirSummary).build();
}
+ /**
+ * This API will list out limited 'count' number of keys after applying
below filters in API parameters:
+ * Default Values of API param filters:
+ * -- replicationType - empty string and filter will not be applied, so
list out all keys irrespective of
+ * replication type.
+ * -- creationTime - empty string and filter will not be applied, so list
out keys irrespective of age.
+ * -- keySize - 0 bytes, which means all keys greater than zero bytes
will be listed, effectively all.
+ * -- startPrefix - /, API assumes that startPrefix path always starts
with /. E.g. /volume/bucket
+ * -- prevKey - ""
+ * -- limit - 1000
+ *
+ * @param replicationType Filter for RATIS or EC replication keys
+ * @param creationDate Filter for keys created after creationDate in
"MM-dd-yyyy HH:mm:ss" string format.
+ * @param keySize Filter for Keys greater than keySize in bytes.
+ * @param startPrefix Filter for startPrefix path.
+ * @param prevKey rocksDB last key of page requested.
+ * @param limit Filter for limited count of keys.
+ *
+ * @return the list of keys in JSON structured format as per respective
bucket layout.
+ *
+ * Now lets consider, we have following OBS, LEGACY and FSO bucket key/files
namespace tree structure
+ *
+ * For OBS Bucket
+ *
+ * /volume1/obs-bucket/key1
+ * /volume1/obs-bucket/key1/key2
+ * /volume1/obs-bucket/key1/key2/key3
+ * /volume1/obs-bucket/key4
+ * /volume1/obs-bucket/key5
+ * /volume1/obs-bucket/key6
+ * For LEGACY Bucket
+ *
+ * /volume1/legacy-bucket/key
+ * /volume1/legacy-bucket/key1/key2
+ * /volume1/legacy-bucket/key1/key2/key3
+ * /volume1/legacy-bucket/key4
+ * /volume1/legacy-bucket/key5
+ * /volume1/legacy-bucket/key6
+ * For FSO Bucket
+ *
+ * /volume1/fso-bucket/dir1/dir2/dir3
+ * /volume1/fso-bucket/dir1/testfile
+ * /volume1/fso-bucket/dir1/file1
+ * /volume1/fso-bucket/dir1/dir2/testfile
+ * /volume1/fso-bucket/dir1/dir2/file1
+ * /volume1/fso-bucket/dir1/dir2/dir3/testfile
+ * /volume1/fso-bucket/dir1/dir2/dir3/file1
+ * Input Request for OBS bucket:
+ *
+ *
`api/v1/keys/listKeys?startPrefix=/volume1/obs-bucket&limit=2&replicationType=RATIS`
+ * Output Response:
+ *
+ * {
+ * "status": "OK",
+ * "path": "/volume1/obs-bucket",
+ * "replicatedDataSize": 62914560,
+ * "unReplicatedDataSize": 62914560,
+ * "lastKey": "/volume1/obs-bucket/key6",
+ * "keys": [
+ * {
+ * "key": "/volume1/obs-bucket/key1",
+ * "path": "volume1/obs-bucket/key1",
+ * "size": 10485760,
+ * "replicatedSize": 10485760,
+ * "replicationInfo": {
+ * "replicationFactor": "ONE",
+ * "requiredNodes": 1,
+ * "replicationType": "RATIS"
+ * },
+ * "creationTime": 1715781418742,
+ * "modificationTime": 1715781419762,
+ * "isKey": true
+ * },
+ * {
+ * "key": "/volume1/obs-bucket/key1/key2",
+ * "path": "volume1/obs-bucket/key1/key2",
+ * "size": 10485760,
+ * "replicatedSize": 10485760,
+ * "replicationInfo": {
+ * "replicationFactor": "ONE",
+ * "requiredNodes": 1,
+ * "replicationType": "RATIS"
+ * },
+ * "creationTime": 1715781421716,
+ * "modificationTime": 1715781422723,
+ * "isKey": true
+ * },
+ * {
+ * "key": "/volume1/obs-bucket/key1/key2/key3",
+ * "path": "volume1/obs-bucket/key1/key2/key3",
+ * "size": 10485760,
+ * "replicatedSize": 10485760,
+ * "replicationInfo": {
+ * "replicationFactor": "ONE",
+ * "requiredNodes": 1,
+ * "replicationType": "RATIS"
+ * },
+ * "creationTime": 1715781424718,
+ * "modificationTime": 1715781425598,
+ * "isKey": true
+ * },
+ * {
+ * "key": "/volume1/obs-bucket/key4",
+ * "path": "volume1/obs-bucket/key4",
+ * "size": 10485760,
+ * "replicatedSize": 10485760,
+ * "replicationInfo": {
+ * "replicationFactor": "ONE",
+ * "requiredNodes": 1,
+ * "replicationType": "RATIS"
+ * },
+ * "creationTime": 1715781427561,
+ * "modificationTime": 1715781428407,
+ * "isKey": true
+ * },
+ * {
+ * "key": "/volume1/obs-bucket/key5",
+ * "path": "volume1/obs-bucket/key5",
+ * "size": 10485760,
+ * "replicatedSize": 10485760,
+ * "replicationInfo": {
+ * "replicationFactor": "ONE",
+ * "requiredNodes": 1,
+ * "replicationType": "RATIS"
+ * },
+ * "creationTime": 1715781430347,
+ * "modificationTime": 1715781431185,
+ * "isKey": true
+ * },
+ * {
+ * "key": "/volume1/obs-bucket/key6",
+ * "path": "volume1/obs-bucket/key6",
+ * "size": 10485760,
+ * "replicatedSize": 10485760,
+ * "replicationInfo": {
+ * "replicationFactor": "ONE",
+ * "requiredNodes": 1,
+ * "replicationType": "RATIS"
+ * },
+ * "creationTime": 1715781433154,
+ * "modificationTime": 1715781433962,
+ * "isKey": true
+ * }
+ * ]
+ * }
+ * Input Request for FSO bucket:
+ *
+ *
`api/v1/keys/listKeys?startPrefix=/volume1/fso-bucket&limit=2&replicationType=RATIS`
+ * Output Response:
+ *
+ * {
+ * "status": "OK",
+ * "path": "/volume1/fso-bucket",
+ * "replicatedDataSize": 188743680,
+ * "unReplicatedDataSize": 62914560,
+ * "lastKey":
"/-9223372036854775552/-9223372036854774016/-9223372036854773503/testfile",
+ * "keys": [
+ * {
+ * "key":
"/-9223372036854775552/-9223372036854774016/-9223372036854773501/file1",
+ * "path": "volume1/fso-bucket/dir1/dir2/dir3/file1",
+ * "size": 10485760,
+ * "replicatedSize": 31457280,
+ * "replicationInfo": {
+ * "replicationFactor": "THREE",
+ * "requiredNodes": 3,
+ * "replicationType": "RATIS"
+ * },
+ * "creationTime": 1715781411785,
+ * "modificationTime": 1715781415119,
+ * "isKey": true
+ * },
+ * {
+ * "key":
"/-9223372036854775552/-9223372036854774016/-9223372036854773501/testfile",
+ * "path": "volume1/fso-bucket/dir1/dir2/dir3/testfile",
+ * "size": 10485760,
+ * "replicatedSize": 31457280,
+ * "replicationInfo": {
+ * "replicationFactor": "THREE",
+ * "requiredNodes": 3,
+ * "replicationType": "RATIS"
+ * },
+ * "creationTime": 1715781409146,
+ * "modificationTime": 1715781409882,
+ * "isKey": true
+ * },
+ * {
+ * "key":
"/-9223372036854775552/-9223372036854774016/-9223372036854773502/file1",
+ * "path": "volume1/fso-bucket/dir1/dir2/file1",
+ * "size": 10485760,
+ * "replicatedSize": 31457280,
+ * "replicationInfo": {
+ * "replicationFactor": "THREE",
+ * "requiredNodes": 3,
+ * "replicationType": "RATIS"
+ * },
+ * "creationTime": 1715781406333,
+ * "modificationTime": 1715781407140,
+ * "isKey": true
+ * },
+ * {
+ * "key":
"/-9223372036854775552/-9223372036854774016/-9223372036854773502/testfile",
+ * "path": "volume1/fso-bucket/dir1/dir2/testfile",
+ * "size": 10485760,
+ * "replicatedSize": 31457280,
+ * "replicationInfo": {
+ * "replicationFactor": "THREE",
+ * "requiredNodes": 3,
+ * "replicationType": "RATIS"
+ * },
+ * "creationTime": 1715781403655,
+ * "modificationTime": 1715781404460,
+ * "isKey": true
+ * },
+ * {
+ * "key":
"/-9223372036854775552/-9223372036854774016/-9223372036854773503/file1",
+ * "path": "volume1/fso-bucket/dir1/file1",
+ * "size": 10485760,
+ * "replicatedSize": 31457280,
+ * "replicationInfo": {
+ * "replicationFactor": "THREE",
+ * "requiredNodes": 3,
+ * "replicationType": "RATIS"
+ * },
+ * "creationTime": 1715781400980,
+ * "modificationTime": 1715781401768,
+ * "isKey": true
+ * },
+ * {
+ * "key":
"/-9223372036854775552/-9223372036854774016/-9223372036854773503/testfile",
+ * "path": "volume1/fso-bucket/dir1/testfile",
+ * "size": 10485760,
+ * "replicatedSize": 31457280,
+ * "replicationInfo": {
+ * "replicationFactor": "THREE",
+ * "requiredNodes": 3,
+ * "replicationType": "RATIS"
+ * },
+ * "creationTime": 1715781397636,
+ * "modificationTime": 1715781398919,
+ * "isKey": true
+ * }
+ * ]
+ * }
+ *
+ * Input Request for Legacy bucket:
+ *
+ *
`api/v1/keys/listKeys?startPrefix=/volume1/legacy-bucket&limit=2&replicationType=RATIS`
+ * Output Response:
+ *
+ * {
+ * "status": "OK",
+ * "path": "/volume1/legacy-bucket",
+ * "replicatedDataSize": 157286400,
+ * "unReplicatedDataSize": 157286400,
+ * "lastKey": "/volume1/legacy-bucket/key6",
+ * "keys": [
+ * {
+ * "key": "/volume1/legacy-bucket/key1",
+ * "path": "volume1/legacy-bucket/key1",
+ * "size": 10485760,
+ * "replicatedSize": 10485760,
+ * "replicationInfo": {
+ * "replicationFactor": "ONE",
+ * "requiredNodes": 1,
+ * "replicationType": "RATIS"
+ * },
+ * "creationTime": 1715781440620,
+ * "modificationTime": 1715781441448,
+ * "isKey": true
+ * },
+ * {
+ * "key": "/volume1/legacy-bucket/key1/key2",
+ * "path": "volume1/legacy-bucket/key1/key2",
+ * "size": 41943040,
+ * "replicatedSize": 41943040,
+ * "replicationInfo": {
+ * "replicationFactor": "ONE",
+ * "requiredNodes": 1,
+ * "replicationType": "RATIS"
+ * },
+ * "creationTime": 1715781443431,
+ * "modificationTime": 1715781444680,
+ * "isKey": true
+ * },
+ * {
+ * "key": "/volume1/legacy-bucket/key1/key2/key3",
+ * "path": "volume1/legacy-bucket/key1/key2/key3",
+ * "size": 10485760,
+ * "replicatedSize": 10485760,
+ * "replicationInfo": {
+ * "replicationFactor": "ONE",
+ * "requiredNodes": 1,
+ * "replicationType": "RATIS"
+ * },
+ * "creationTime": 1715781446657,
+ * "modificationTime": 1715781447476,
+ * "isKey": true
+ * },
+ * {
+ * "key": "/volume1/legacy-bucket/key4",
+ * "path": "volume1/legacy-bucket/key4",
+ * "size": 41943040,
+ * "replicatedSize": 41943040,
+ * "replicationInfo": {
+ * "replicationFactor": "ONE",
+ * "requiredNodes": 1,
+ * "replicationType": "RATIS"
+ * },
+ * "creationTime": 1715781449445,
+ * "modificationTime": 1715781450464,
+ * "isKey": true
+ * },
+ * {
+ * "key": "/volume1/legacy-bucket/key5",
+ * "path": "volume1/legacy-bucket/key5",
+ * "size": 10485760,
+ * "replicatedSize": 10485760,
+ * "replicationInfo": {
+ * "replicationFactor": "ONE",
+ * "requiredNodes": 1,
+ * "replicationType": "RATIS"
+ * },
+ * "creationTime": 1715781452476,
+ * "modificationTime": 1715781453335,
+ * "isKey": true
+ * },
+ * {
+ * "key": "/volume1/legacy-bucket/key6",
+ * "path": "volume1/legacy-bucket/key6",
+ * "size": 41943040,
+ * "replicatedSize": 41943040,
+ * "replicationInfo": {
+ * "replicationFactor": "ONE",
+ * "requiredNodes": 1,
+ * "replicationType": "RATIS"
+ * },
+ * "creationTime": 1715781477894,
+ * "modificationTime": 1715781479351,
+ * "isKey": true
+ * }
+ * ]
+ * }
+ * ********************************************************
+ * @throws IOException
+ */
+ @GET
+ @Path("/listKeys")
+ @SuppressWarnings("methodlength")
+ public Response listKeys(@QueryParam("replicationType") String
replicationType,
+ @QueryParam("creationDate") String creationDate,
+ @DefaultValue(DEFAULT_KEY_SIZE)
@QueryParam("keySize") long keySize,
+ @DefaultValue(OM_KEY_PREFIX)
@QueryParam("startPrefix") String startPrefix,
+ @DefaultValue(StringUtils.EMPTY)
@QueryParam(RECON_QUERY_PREVKEY) String prevKey,
+ @DefaultValue(DEFAULT_FETCH_COUNT)
@QueryParam("limit") int limit) {
+
+
+ // This API supports startPrefix from bucket level.
+ if (startPrefix == null || startPrefix.length() == 0) {
+ return Response.status(Response.Status.BAD_REQUEST).build();
+ }
+ String[] names = startPrefix.split(OM_KEY_PREFIX);
+ if (names.length < 3) {
+ return Response.status(Response.Status.BAD_REQUEST).build();
+ }
+ ListKeysResponse listKeysResponse = new ListKeysResponse();
+ if (!ReconUtils.isInitializationComplete(omMetadataManager)) {
+ listKeysResponse.setStatus(ResponseStatus.INITIALIZING);
+ return Response.ok(listKeysResponse).build();
+ }
+ ParamInfo paramInfo = new ParamInfo(replicationType, creationDate,
keySize, startPrefix, prevKey,
+ limit, false, "");
+ Response response = getListKeysResponse(paramInfo);
+ if (response.getEntity() instanceof ListKeysResponse) {
+ listKeysResponse = (ListKeysResponse) response.getEntity();
+ }
+
+ List<KeyEntityInfo> keyInfoList = listKeysResponse.getKeys();
+ if (!keyInfoList.isEmpty()) {
+ listKeysResponse.setLastKey(keyInfoList.get(keyInfoList.size() -
1).getKey());
+ }
+ return Response.ok(listKeysResponse).build();
+ }
+
+ private Response getListKeysResponse(ParamInfo paramInfo) {
+ try {
+ paramInfo.setLimit(Math.max(0, paramInfo.getLimit())); // Ensure limit
is non-negative
+ ListKeysResponse listKeysResponse = new ListKeysResponse();
+ listKeysResponse.setPath(paramInfo.getStartPrefix());
+ long replicatedTotal = 0;
+ long unreplicatedTotal = 0;
+ boolean keysFound = false; // Flag to track if any keys are found
+
+ // Search keys from non-FSO layout.
+ Map<String, OmKeyInfo> obsKeys;
+ Table<String, OmKeyInfo> keyTable =
+ omMetadataManager.getKeyTable(BucketLayout.LEGACY);
+ obsKeys = retrieveKeysFromTable(keyTable, paramInfo);
+ for (Map.Entry<String, OmKeyInfo> entry : obsKeys.entrySet()) {
+ keysFound = true;
+ KeyEntityInfo keyEntityInfo =
+ createKeyEntityInfoFromOmKeyInfo(entry.getKey(), entry.getValue());
+
+ listKeysResponse.getKeys().add(keyEntityInfo);
+ replicatedTotal += entry.getValue().getReplicatedSize();
+ unreplicatedTotal += entry.getValue().getDataSize();
+ }
+
+ // Search keys from FSO layout.
+ Map<String, OmKeyInfo> fsoKeys = searchKeysInFSO(paramInfo);
+ for (Map.Entry<String, OmKeyInfo> entry : fsoKeys.entrySet()) {
+ keysFound = true;
+ KeyEntityInfo keyEntityInfo =
+ createKeyEntityInfoFromOmKeyInfo(entry.getKey(), entry.getValue());
+
+ listKeysResponse.getKeys().add(keyEntityInfo);
+ replicatedTotal += entry.getValue().getReplicatedSize();
+ unreplicatedTotal += entry.getValue().getDataSize();
+ }
+
+ // If no keys were found, return a response indicating that no keys
matched
+ if (!keysFound) {
+ return
ReconResponseUtils.noMatchedKeysResponse(paramInfo.getStartPrefix());
+ }
+
+ // Set the aggregated totals in the response
+ listKeysResponse.setReplicatedDataSize(replicatedTotal);
+ listKeysResponse.setUnReplicatedDataSize(unreplicatedTotal);
+
+ return Response.ok(listKeysResponse).build();
+ } catch (IOException e) {
+ return ReconResponseUtils.createInternalServerErrorResponse(
+ "Error searching keys in OM DB: " + e.getMessage());
+ } catch (IllegalArgumentException e) {
+ return ReconResponseUtils.createBadRequestResponse("Invalid startPrefix:
" + e.getMessage());
+ }
+ }
+
+ public Map<String, OmKeyInfo> searchKeysInFSO(ParamInfo paramInfo)
+ throws IOException, IllegalArgumentException {
+ int originalLimit = paramInfo.getLimit();
+ Map<String, OmKeyInfo> matchedKeys = new LinkedHashMap<>();
+ // Convert the search prefix to an object path for FSO buckets
+ String startPrefixObjectPath =
convertStartPrefixPathToObjectIdPath(paramInfo.getStartPrefix());
+ String[] names = parseRequestPath(startPrefixObjectPath);
+ Table<String, OmKeyInfo> fileTable =
+ omMetadataManager.getKeyTable(BucketLayout.FILE_SYSTEM_OPTIMIZED);
+
+ // If names.length > 2, then the search prefix is at the volume or bucket
level hence
+ // no need to find parent or extract id's or find subpaths as the
fileTable is
+ // suitable for volume and bucket level search
+ if (names.length > 2) {
+ // Fetch the parent ID to search for
+ long parentId = Long.parseLong(names[names.length - 1]);
+
+ // Fetch the nameSpaceSummary for the parent ID
+ NSSummary parentSummary =
+ reconNamespaceSummaryManager.getNSSummary(parentId);
+ if (parentSummary == null) {
+ return matchedKeys;
+ }
+ List<String> subPaths = new ArrayList<>();
+ // Add the initial search prefix object path because it can have both
files and subdirectories with files.
+ subPaths.add(startPrefixObjectPath);
+
+ // Recursively gather all subpaths
+ gatherSubPaths(parentId, subPaths, Long.parseLong(names[0]),
Long.parseLong(names[1]));
+ // Iterate over the subpaths and retrieve the files
+ for (String subPath : subPaths) {
+ paramInfo.setStartPrefix(subPath);
+ matchedKeys.putAll(
+ retrieveKeysFromTable(fileTable, paramInfo));
+ paramInfo.setLimit(originalLimit - matchedKeys.size());
+ if (matchedKeys.size() >= originalLimit) {
+ break;
+ }
+ }
+ return matchedKeys;
+ }
+
+ paramInfo.setStartPrefix(startPrefixObjectPath);
+ // Iterate over for bucket and volume level search
+ matchedKeys.putAll(
+ retrieveKeysFromTable(fileTable, paramInfo));
+ return matchedKeys;
+ }
+
+ /**
+ * Finds all subdirectories under a parent directory in an FSO bucket. It
builds
+ * a list of paths for these subdirectories. These sub-directories are then
used
+ * to search for files in the fileTable.
+ * <p>
+ * How it works:
+ * - Starts from a parent directory identified by parentId.
+ * - Looks through all child directories of this parent.
+ * - For each child, it creates a path that starts with
volumeID/bucketID/parentId,
+ * following our fileTable format
+ * - Adds these paths to a list and explores each child further for more
subdirectories.
+ *
+ * @param parentId The ID of the directory we start exploring from.
+ * @param subPaths A list where we collect paths to all subdirectories.
+ * @param volumeID
+ * @param bucketID
+ * @throws IOException If there are problems accessing directory information.
+ */
+ private void gatherSubPaths(long parentId, List<String> subPaths,
+ long volumeID, long bucketID) throws IOException
{
+ // Fetch the NSSummary object for parentId
+ NSSummary parentSummary =
+ reconNamespaceSummaryManager.getNSSummary(parentId);
+ if (parentSummary == null) {
+ return;
+ }
+
+ Set<Long> childDirIds = parentSummary.getChildDir();
+ for (Long childId : childDirIds) {
+ // Fetch the NSSummary for each child directory
+ NSSummary childSummary =
+ reconNamespaceSummaryManager.getNSSummary(childId);
+ if (childSummary != null) {
+ String subPath =
+ constructObjectPathWithPrefix(volumeID, bucketID, childId);
+ // Add to subPaths
+ subPaths.add(subPath);
+ // Recurse into this child directory
+ gatherSubPaths(childId, subPaths, volumeID, bucketID);
+ }
+ }
+ }
+
+
+ /**
+ * Converts a startPrefix path into an objectId path for FSO buckets, using
IDs.
+ * <p>
+ * This method transforms a user-provided path (e.g., "volume/bucket/dir1")
into
+ * a database-friendly format ("/volumeID/bucketID/ParentId/") by replacing
names
+ * with their corresponding IDs. It simplifies database queries for FSO
bucket operations.
+ *
+ * @param startPrefixPath The path to be converted.
+ * @return The objectId path as "/volumeID/bucketID/ParentId/".
+ * @throws IOException If database access fails.
+ */
+ public String convertStartPrefixPathToObjectIdPath(String startPrefixPath)
+ throws IOException, IllegalArgumentException {
+
+ String[] names = parseRequestPath(
+ normalizePath(startPrefixPath, BucketLayout.FILE_SYSTEM_OPTIMIZED));
+
+ // Root-Level :- Return the original path
+ if (names.length == 0) {
+ return startPrefixPath;
+ }
+
+ // Volume-Level :- Fetch the volumeID
+ String volumeName = names[0];
+ ReconUtils.validateNames(volumeName);
+ String volumeKey = omMetadataManager.getVolumeKey(volumeName);
+ long volumeId = omMetadataManager.getVolumeTable().getSkipCache(volumeKey)
+ .getObjectID();
+ if (names.length == 1) {
+ return constructObjectPathWithPrefix(volumeId);
+ }
+
+ // Bucket-Level :- Fetch the bucketID
+ String bucketName = names[1];
+ ReconUtils.validateNames(bucketName);
+ String bucketKey = omMetadataManager.getBucketKey(volumeName, bucketName);
+ OmBucketInfo bucketInfo =
+ omMetadataManager.getBucketTable().getSkipCache(bucketKey);
+ long bucketId = bucketInfo.getObjectID();
+ if (names.length == 2) {
+ return constructObjectPathWithPrefix(volumeId, bucketId);
+ }
+
+ // Fetch the immediate parentID which could be a directory or the bucket
itself
+ BucketHandler handler =
+ getBucketHandler(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM, bucketInfo);
+ long dirObjectId = handler.getDirInfo(names).getObjectID();
+ return constructObjectPathWithPrefix(volumeId, bucketId, dirObjectId);
+ }
+
+ /**
+ * Common method to retrieve keys from a table based on a search prefix and
a limit.
+ *
+ * @param table The table to retrieve keys from.
+ * @param paramInfo The stats object holds total count, count and limit.
+ * @return A map of keys and their corresponding OmKeyInfo objects.
+ * @throws IOException If there are problems accessing the table.
+ */
+ private Map<String, OmKeyInfo> retrieveKeysFromTable(
+ Table<String, OmKeyInfo> table, ParamInfo paramInfo)
+ throws IOException {
+ boolean skipPrevKey = false;
+ String seekKey = paramInfo.getPrevKey();
+ Map<String, OmKeyInfo> matchedKeys = new LinkedHashMap<>();
+ try (
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
keyIter = table.iterator()) {
+
+ if (!paramInfo.isSkipPrevKeyDone() && StringUtils.isNotBlank(seekKey)) {
+ skipPrevKey = true;
+ Table.KeyValue<String, OmKeyInfo> seekKeyValue =
+ keyIter.seek(seekKey);
+
+ // check if RocksDB was able to seek correctly to the given key prefix
+ // if not, then return empty result
+ // In case of an empty prevKeyPrefix, all the keys are returned
+ if (seekKeyValue == null ||
+ (StringUtils.isNotBlank(paramInfo.getPrevKey()) &&
+ !seekKeyValue.getKey().equals(paramInfo.getPrevKey()))) {
+ return matchedKeys;
+ }
+ } else {
+ keyIter.seek(paramInfo.getStartPrefix());
+ }
+
+ while (keyIter.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> entry = keyIter.next();
+ String dbKey = entry.getKey();
+ paramInfo.setLastKey(dbKey);
+ if (!dbKey.startsWith(paramInfo.getStartPrefix())) {
+ break; // Exit the loop if the key no longer matches the prefix
+ }
+ if (skipPrevKey && dbKey.equals(paramInfo.getPrevKey())) {
+ paramInfo.setSkipPrevKeyDone(true);
+ continue;
+ }
+ if (applyFilters(entry, paramInfo) && matchedKeys.size() <
paramInfo.getLimit()) {
+ matchedKeys.put(dbKey, entry.getValue());
+ }
+ }
+ } catch (IOException exception) {
+ LOG.error("Error retrieving keys from table for path: {}",
paramInfo.getStartPrefix(), exception);
+ throw exception;
+ }
+ return matchedKeys;
+ }
+
+ private boolean applyFilters(Table.KeyValue<String, OmKeyInfo> entry,
ParamInfo paramInfo) throws IOException {
+
+ LOG.debug("Applying filters on : {}", entry.getKey());
+
+ long epochMillis =
+ ReconUtils.convertToEpochMillis(paramInfo.getCreationDate(),
"MM-dd-yyyy HH:mm:ss", TimeZone.getDefault());
+ Predicate<Table.KeyValue<String, OmKeyInfo>> keyAgeFilter = keyData -> {
+ try {
+ return keyData.getValue().getCreationTime() >= epochMillis;
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ };
+ Predicate<Table.KeyValue<String, OmKeyInfo>> keyReplicationFilter =
+ keyData -> {
+ try {
+ return
keyData.getValue().getReplicationConfig().getReplicationType().name()
+ .equals(paramInfo.getReplicationType());
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ };
+ Predicate<Table.KeyValue<String, OmKeyInfo>> keySizeFilter = keyData -> {
+ try {
+ return keyData.getValue().getDataSize() > paramInfo.getKeySize();
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ };
+
+ List<Table.KeyValue<String, OmKeyInfo>> filteredKeyList = Stream.of(entry)
+ .filter(keyData -> !StringUtils.isEmpty(paramInfo.getCreationDate()) ?
keyAgeFilter.test(keyData) : true)
+ .filter(
+ keyData -> !StringUtils.isEmpty(paramInfo.getReplicationType()) ?
keyReplicationFilter.test(keyData) : true)
+ .filter(keySizeFilter)
+ .collect(Collectors.toList());
+
+ LOG.debug("After applying filter on : {}, filtered list size: {}",
entry.getKey(), filteredKeyList.size());
+
+ return (filteredKeyList.size() > 0);
+ }
+
+ /**
+ * Creates a KeyEntityInfo object from an OmKeyInfo object and the
corresponding key.
+ *
+ * @param dbKey The key in the database corresponding to the OmKeyInfo
object.
+ * @param keyInfo The OmKeyInfo object to create the KeyEntityInfo from.
+ * @return The KeyEntityInfo object created from the OmKeyInfo object and
the key.
+ */
+ private KeyEntityInfo createKeyEntityInfoFromOmKeyInfo(String dbKey,
+ OmKeyInfo keyInfo)
throws IOException {
+ KeyEntityInfo keyEntityInfo = new KeyEntityInfo();
+ keyEntityInfo.setKey(dbKey); // Set the DB key
+ keyEntityInfo.setPath(ReconUtils.constructFullPath(keyInfo,
reconNamespaceSummaryManager,
+ omMetadataManager));
+ keyEntityInfo.setSize(keyInfo.getDataSize());
+ keyEntityInfo.setCreationTime(keyInfo.getCreationTime());
+ keyEntityInfo.setModificationTime(keyInfo.getModificationTime());
+ keyEntityInfo.setReplicatedSize(keyInfo.getReplicatedSize());
+ keyEntityInfo.setReplicationConfig(keyInfo.getReplicationConfig());
+ return keyEntityInfo;
+ }
+
+ /**
+ * Constructs an object path with the given IDs.
+ *
+ * @param ids The IDs to construct the object path with.
+ * @return The constructed object path.
+ */
+ private String constructObjectPathWithPrefix(long... ids) {
Review Comment:
I believe this can also be placed inside ReconUtils
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/OMDBInsightEndpoint.java:
##########
@@ -674,6 +685,720 @@ public Response getDeletedDirectorySummary() {
return Response.ok(dirSummary).build();
}
+ /**
+ * This API will list out limited 'count' number of keys after applying
below filters in API parameters:
+ * Default Values of API param filters:
+ * -- replicationType - empty string and filter will not be applied, so
list out all keys irrespective of
+ * replication type.
+ * -- creationTime - empty string and filter will not be applied, so list
out keys irrespective of age.
+ * -- keySize - 0 bytes, which means all keys greater than zero bytes
will be listed, effectively all.
+ * -- startPrefix - /, API assumes that startPrefix path always starts
with /. E.g. /volume/bucket
+ * -- prevKey - ""
+ * -- limit - 1000
+ *
+ * @param replicationType Filter for RATIS or EC replication keys
+ * @param creationDate Filter for keys created after creationDate in
"MM-dd-yyyy HH:mm:ss" string format.
+ * @param keySize Filter for Keys greater than keySize in bytes.
+ * @param startPrefix Filter for startPrefix path.
+ * @param prevKey rocksDB last key of page requested.
+ * @param limit Filter for limited count of keys.
+ *
+ * @return the list of keys in JSON structured format as per respective
bucket layout.
+ *
+ * Now lets consider, we have following OBS, LEGACY and FSO bucket key/files
namespace tree structure
+ *
+ * For OBS Bucket
+ *
+ * /volume1/obs-bucket/key1
+ * /volume1/obs-bucket/key1/key2
+ * /volume1/obs-bucket/key1/key2/key3
+ * /volume1/obs-bucket/key4
+ * /volume1/obs-bucket/key5
+ * /volume1/obs-bucket/key6
+ * For LEGACY Bucket
+ *
+ * /volume1/legacy-bucket/key
+ * /volume1/legacy-bucket/key1/key2
+ * /volume1/legacy-bucket/key1/key2/key3
+ * /volume1/legacy-bucket/key4
+ * /volume1/legacy-bucket/key5
+ * /volume1/legacy-bucket/key6
+ * For FSO Bucket
+ *
+ * /volume1/fso-bucket/dir1/dir2/dir3
+ * /volume1/fso-bucket/dir1/testfile
+ * /volume1/fso-bucket/dir1/file1
+ * /volume1/fso-bucket/dir1/dir2/testfile
+ * /volume1/fso-bucket/dir1/dir2/file1
+ * /volume1/fso-bucket/dir1/dir2/dir3/testfile
+ * /volume1/fso-bucket/dir1/dir2/dir3/file1
+ * Input Request for OBS bucket:
+ *
+ *
`api/v1/keys/listKeys?startPrefix=/volume1/obs-bucket&limit=2&replicationType=RATIS`
+ * Output Response:
+ *
+ * {
+ * "status": "OK",
+ * "path": "/volume1/obs-bucket",
+ * "replicatedDataSize": 62914560,
+ * "unReplicatedDataSize": 62914560,
+ * "lastKey": "/volume1/obs-bucket/key6",
+ * "keys": [
+ * {
+ * "key": "/volume1/obs-bucket/key1",
+ * "path": "volume1/obs-bucket/key1",
+ * "size": 10485760,
+ * "replicatedSize": 10485760,
+ * "replicationInfo": {
Review Comment:
If the structure of the JSON response remains consistent across different
bucket types, there is no need to show multiple example responses for each
bucket type. Instead, we can show one example response and clarify that this
structure applies to all bucket types (OBS, Legacy, and FSO).
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestOmDBInsightEndPoint.java:
##########
@@ -785,10 +1103,284 @@ public void testGetDirectorySizeInfo() throws Exception
{
assertEquals(18L, keyInsightInfoResp.getUnreplicatedDataSize());
}
+ @Test
+ public void testListKeysFSOBucket() {
Review Comment:
The existing test cases cover the following main scenarios which is great!
- Listing keys at different levels (bucket, directory) for FSO and OBS
buckets.
- Verifying the number of keys returned, paths, keys, replication types, and
last keys.
- Testing pagination with different limits and across multiple pages.
- Filtering keys based on replication type, creation date, and key size.
- Handling edge cases like empty results and last page with an empty last
key.
Can we write a few more methods that test out these simple scenarios as well.
Empty Buckets: Verify the behaviour when the bucket is empty.
Non-existent Paths: Test with paths that do not exist to ensure the API
handles such cases gracefully.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/FSOBucketHandler.java:
##########
@@ -300,5 +300,4 @@ public OmDirectoryInfo getDirInfo(String[] names) throws
IOException {
.getDirectoryTable().getSkipCache(dirKey);
return dirInfo;
}
-
Review Comment:
Please Remove this unwanted change, from the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
