[
https://issues.apache.org/jira/browse/HDDS-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumit Agrawal updated HDDS-10634:
---------------------------------
Fix Version/s: 1.5.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Recon - listKeys API for listing of OBS , FSO and Legacy bucket keys with
> filters
> ---------------------------------------------------------------------------------
>
> Key: HDDS-10634
> URL: https://issues.apache.org/jira/browse/HDDS-10634
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Recon
> Reporter: Devesh Kumar Singh
> Assignee: Devesh Kumar Singh
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.5.0
>
>
> New API:
> api/v1/namespace/listKeys?startPrefix=/volume1/obs-bucket/&limit=105
> h5. Default values of API parameters if not provided:
> # replicationType - empty/null, so effectively will list all types of keys.
> # creationTime - empty string and filter will not be applied, so list out
> keys irrespective of age.
> # keySize - 0 bytes, which means all keys greater than zero bytes will be
> listed, effectively all.
> # startPrefix - /
> # prevKey - ""
> # limit - 1000
>
> h5. Behavior of API:
> Now lets consider, we have following OBS, LEGACY and FSO bucket key/files
> namespace tree structure
> {{For OBS Bucket}}
> * /volume1/obs-bucket/key1
> * /volume1/obs-bucket/key1/key2
> * /volume1/obs-bucket/key1/key2/key3
> * /volume1/obs-bucket/key4
> * /volume1/obs-bucket/key5
> * /volume1/obs-bucket/key6
> {{For LEGACY Bucket}}
> * /volume1/legacy-bucket/key
> * /volume1/legacy-bucket/key1/key2
> * /volume1/legacy-bucket/key1/key2/key3
> * /volume1/legacy-bucket/key4
> * /volume1/legacy-bucket/key5
> * /volume1/legacy-bucket/key6
> {{For FSO Bucket}}
> * /volume1/fso-bucket/dir1/dir2/dir3
> * /volume1/fso-bucket/dir1/testfile
> * /volume1/fso-bucket/dir1/file1
> * /volume1/fso-bucket/dir1/dir2/testfile
> * /volume1/fso-bucket/dir1/dir2/file1
> * /volume1/fso-bucket/dir1/dir2/dir3/testfile
> * /volume1/fso-bucket/dir1/dir2/dir3/file1
> *Input Request for OBS bucket:*
> \{{
> `api/v1/keys/listKeys?startPrefix=/volume1/obs-bucket&limit=2&replicationType=RATIS`}}
>
> *Output Response:*
> {{{
> "status": "OK",
> "path": "/volume1/obs-bucket",
> "replicatedDataSize": 20971520,
> "unReplicatedDataSize": 20971520,
> "keyCount": 2,
> "lastKey": "/volume1/obs-bucket/key1/key2",
> "keys": [
> {
> "key": "/volume1/obs-bucket/key1",
> "path": "key1",
> "inStateSince": 1715174266126,
> "size": 10485760,
> "replicatedSize": 10485760,
> "replicationInfo":
> { "replicationFactor": "ONE", "requiredNodes": 1, "replicationType": "RATIS" }
> ,
> "creationTime": 1715174266126,
> "modificationTime": 1715174267480,
> "isKey": true
> },
> {
> "key": "/volume1/obs-bucket/key1/key2",
> "path": "key1/key2",
> "inStateSince": 1715174269510,
> "size": 10485760,
> "replicatedSize": 10485760,
> "replicationInfo":
> { "replicationFactor": "ONE", "requiredNodes": 1, "replicationType": "RATIS" }
> ,
> "creationTime": 1715174269510,
> "modificationTime": 1715174270410,
> "isKey": true
> }
> ]
> }}}
>
> *Input Request for FSO bucket:*
> \{{
> `api/v1/keys/listKeys?startPrefix=/volume1/fso-bucket&limit=2&replicationType=RATIS`}}
>
> *Output Response:*
> {{{
> "status": "OK",
> "path": "/volume1/fso-bucket",
> "replicatedDataSize": 62914560,
> "unReplicatedDataSize": 20971520,
> "keyCount": 2,
> "lastKey":
> "/-9223372036854775552/-9223372036854775040/-9223372036854774525/testfile",
> "keys": [
> {
> "key":
> "/-9223372036854775552/-9223372036854775040/-9223372036854774525/file1",
> "path": "file1",
> "inStateSince": 1715174237440,
> "size": 10485760,
> "replicatedSize": 31457280,
> "replicationInfo":
> { "replicationFactor": "THREE", "requiredNodes": 3, "replicationType":
> "RATIS" }
> ,
> "creationTime": 1715174237440,
> "modificationTime": 1715174238161,
> "isKey": true
> },
> {
> "key":
> "/-9223372036854775552/-9223372036854775040/-9223372036854774525/testfile",
> "path": "testfile",
> "inStateSince": 1715174234840,
> "size": 10485760,
> "replicatedSize": 31457280,
> "replicationInfo":
> { "replicationFactor": "THREE", "requiredNodes": 3, "replicationType":
> "RATIS" }
> ,
> "creationTime": 1715174234840,
> "modificationTime": 1715174235562,
> "isKey": true
> }
> ]
> }}}
>
> *Input Request for Legacy bucket:*
> \{{
> `api/v1/keys/listKeys?startPrefix=/volume1/legacy-bucket&limit=2&replicationType=RATIS`}}
>
> *Output Response:*
> {{{
> "status": "OK",
> "path": "/volume1/legacy-bucket",
> "replicatedDataSize": 52428800,
> "unReplicatedDataSize": 52428800,
> "keyCount": 2,
> "lastKey": "/volume1/legacy-bucket/key1/key2",
> "keys": [
> {
> "key": "/volume1/legacy-bucket/key1",
> "path": "key1",
> "inStateSince": 1715174303702,
> "size": 10485760,
> "replicatedSize": 10485760,
> "replicationInfo":
> { "replicationFactor": "ONE", "requiredNodes": 1, "replicationType": "RATIS" }
> ,
> "creationTime": 1715174303702,
> "modificationTime": 1715174304619,
> "isKey": true
> },
> {
> "key": "/volume1/legacy-bucket/key1/key2",
> "path": "key1/key2",
> "inStateSince": 1715174306641,
> "size": 41943040,
> "replicatedSize": 41943040,
> "replicationInfo":
> { "replicationFactor": "ONE", "requiredNodes": 1, "replicationType": "RATIS" }
> ,
> "creationTime": 1715174306641,
> "modificationTime": 1715174307994,
> "isKey": true
> }
> ]
> }}}
> h5.
> This mean, API will apply filters like *{{replicationType}}* , *{{keySize}}*
> and *{{creationTime}}* on set of records returned after *{{startPrefix}}*
> and number of records returned based *limit* value specified. Pagination
> support will be based on *prevKey* and *limit.*
> *Pagination:*
> Now lets say we have total 6 records for a given set of filters and params
> passed and client wants to fetch in total 3 pages , each page 2 records. If
> each page we will retrieve 2 items, so till we get empty last key, we'll
> continue to fetch and empty last key signifies the last page.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
