[
https://issues.apache.org/jira/browse/HDDS-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devesh Kumar Singh updated HDDS-10634:
--------------------------------------
Description:
New API:
api/v1/namespace/listKeys?startPrefix=/volume1/obs-bucket/&count=105
h5. Default values of API parameters if not provided:
# replicationType - empty/null, so effectively will list all types of keys.
# creationTime - empty string and filter will not be applied, so list out keys
irrespective of age.
# keySize - 0 bytes, which means all keys greater than zero bytes will be
listed, effectively all.
# startPrefix - /
# prevKey - ""
# limit - 1000
h5. Behavior of API:
Now lets consider, we have following OBS, LEGACY and FSO bucket key/files
namespace tree structure
{{For OBS Bucket}}
* /volume1/obs-bucket/key1
* /volume1/obs-bucket/key1/key2
* /volume1/obs-bucket/key1/key2/key3
* /volume1/obs-bucket/key4
* /volume1/obs-bucket/key5
* /volume1/obs-bucket/key6
{{For LEGACY Bucket}}
* /volume1/legacy-bucket/key
* /volume1/legacy-bucket/key1/key2
* /volume1/legacy-bucket/key1/key2/key3
* /volume1/legacy-bucket/key4
* /volume1/legacy-bucket/key5
* /volume1/legacy-bucket/key6
{{For FSO Bucket}}
* /volume1/fso-bucket/dir1/dir2/dir3
* /volume1/fso-bucket/dir1/testfile
* /volume1/fso-bucket/dir1/file1
* /volume1/fso-bucket/dir1/dir2/testfile
* /volume1/fso-bucket/dir1/dir2/file1
* /volume1/fso-bucket/dir1/dir2/dir3/testfile
* /volume1/fso-bucket/dir1/dir2/dir3/file1
*Input Request for OBS bucket:*
{{
`api/v1/keys/listKeys?startPrefix=/volume1/obs-bucket&limit=2&replicationType=RATIS`}}
*Output Response:*
{{{
"status": "OK",
"path": "/volume1/obs-bucket",
"replicatedDataSize": 20971520,
"unReplicatedDataSize": 20971520,
"keyCount": 2,
"lastKey": "/volume1/obs-bucket/key1/key2",
"keys": [
\{
"key": "/volume1/obs-bucket/key1",
"path": "key1",
"inStateSince": 1715174266126,
"size": 10485760,
"replicatedSize": 10485760,
"replicationInfo": {
"replicationFactor": "ONE",
"requiredNodes": 1,
"replicationType": "RATIS"
},
"creationTime": 1715174266126,
"modificationTime": 1715174267480,
"isKey": true
},
\{
"key": "/volume1/obs-bucket/key1/key2",
"path": "key1/key2",
"inStateSince": 1715174269510,
"size": 10485760,
"replicatedSize": 10485760,
"replicationInfo": {
"replicationFactor": "ONE",
"requiredNodes": 1,
"replicationType": "RATIS"
},
"creationTime": 1715174269510,
"modificationTime": 1715174270410,
"isKey": true
}
]
}}}
*Input Request for FSO bucket:*
{{
`api/v1/keys/listKeys?startPrefix=/volume1/fso-bucket&limit=2&replicationType=RATIS`}}
*Output Response:*
{{{
"status": "OK",
"path": "/volume1/fso-bucket",
"replicatedDataSize": 62914560,
"unReplicatedDataSize": 20971520,
"keyCount": 2,
"lastKey":
"/-9223372036854775552/-9223372036854775040/-9223372036854774525/testfile",
"keys": [
\{
"key":
"/-9223372036854775552/-9223372036854775040/-9223372036854774525/file1",
"path": "file1",
"inStateSince": 1715174237440,
"size": 10485760,
"replicatedSize": 31457280,
"replicationInfo": {
"replicationFactor": "THREE",
"requiredNodes": 3,
"replicationType": "RATIS"
},
"creationTime": 1715174237440,
"modificationTime": 1715174238161,
"isKey": true
},
\{
"key":
"/-9223372036854775552/-9223372036854775040/-9223372036854774525/testfile",
"path": "testfile",
"inStateSince": 1715174234840,
"size": 10485760,
"replicatedSize": 31457280,
"replicationInfo": {
"replicationFactor": "THREE",
"requiredNodes": 3,
"replicationType": "RATIS"
},
"creationTime": 1715174234840,
"modificationTime": 1715174235562,
"isKey": true
}
]
}}}
*Input Request for Legacy bucket:*
{{
`api/v1/keys/listKeys?startPrefix=/volume1/legacy-bucket&limit=2&replicationType=RATIS`}}
*Output Response:*
{{{
"status": "OK",
"path": "/volume1/legacy-bucket",
"replicatedDataSize": 52428800,
"unReplicatedDataSize": 52428800,
"keyCount": 2,
"lastKey": "/volume1/legacy-bucket/key1/key2",
"keys": [
\{
"key": "/volume1/legacy-bucket/key1",
"path": "key1",
"inStateSince": 1715174303702,
"size": 10485760,
"replicatedSize": 10485760,
"replicationInfo": {
"replicationFactor": "ONE",
"requiredNodes": 1,
"replicationType": "RATIS"
},
"creationTime": 1715174303702,
"modificationTime": 1715174304619,
"isKey": true
},
\{
"key": "/volume1/legacy-bucket/key1/key2",
"path": "key1/key2",
"inStateSince": 1715174306641,
"size": 41943040,
"replicatedSize": 41943040,
"replicationInfo": {
"replicationFactor": "ONE",
"requiredNodes": 1,
"replicationType": "RATIS"
},
"creationTime": 1715174306641,
"modificationTime": 1715174307994,
"isKey": true
}
]
}}}
h5.
This mean, API will apply filters like *{{replicationType}}* , *{{keySize}}*
and *{{creationTime}}* on set of records returned after *{{startPrefix}}* and
number of records returned based *limit* value specified. Pagination support
will be based on *prevKey* and *limit.*
*Pagination:*
Now lets say we have total 6 records for a given set of filters and params
passed and client wants to fetch in total 3 pages , each page 2 records. If
each page we will retrieve 2 items, so till we get empty last key, we'll
continue to fetch and empty last key signifies the last page.
was:
New API:
api/v1/namespace/listKeys?startPrefix=/volume1/obs-bucket/&count=105
h5. Default values of API parameters if not provided:
# replicationType - empty/null, so effectively will list all types of keys.
# creationTime - empty string and filter will not be applied, so list out keys
irrespective of age.
# keySize - 0 bytes, which means all keys greater than zero bytes will be
listed, effectively all.
# startPrefix - /
# count - 1000
# offset - 0 -> which means records starting with 1st record.
h5. Behavior of API:
h5. For OBS bucket - list out *_count_* number of keys on the provided
path.
This API will implement pagination support using *count params.*
h3. Get List of All Keys:
GET /api/v1/namespace/listKeys
API params:
# replicationType
# creationTime in "MM-dd-yyyy HH:mm:ss" string format.
# startPrefix
# count
# keySize
# offset
*Input Request:*
api/v1/namespace/listKeys?startPrefix=/volume1/obs-bucket/&count=105
*Output Response:*
** {
"status": "OK",
"path": "/volume1/obs-bucket/",
"size": 62914560,
"sizeWithReplica": -1,
"subPathCount": 6,
"subPaths": [
{ "key": true, "path": "key1", "size":
10485760, "sizeWithReplica": -1, "isKey": true,
"replicationType": "RATIS", "creationTime": "1234567",
"modificationTime": "4567259" }
,
{ "key": true, "path": "key1/key2", "size":
10485760, "sizeWithReplica": -1, "isKey": true }
,
{ "key": true, "path": "key1/key2/key3",
"size": 10485760, "sizeWithReplica": -1, "isKey": true,
"replicationType": "RATIS", "creationTime": "1234567",
"modificationTime": "4567259" }
,
{ "key": true, "path": "key4", "size":
10485760, "sizeWithReplica": -1, "isKey": true,
"replicationType": "RATIS", "creationTime": "1234567",
"modificationTime": "4567259" }
,
{ "key": true, "path": "key5", "size":
10485760, "sizeWithReplica": -1, "isKey": true,
"replicationType": "RATIS", "creationTime": "1234567",
"modificationTime": "4567259" }
,
{ "key": true, "path": "key6", "size":
10485760, "sizeWithReplica": -1, "isKey": true,
"replicationType": "RATIS", "creationTime": "1234567",
"modificationTime": "4567259" }
],
"sizeDirectKey": 62914560
}
# Output response will have total count based on filters provided in request,
which tells how many total keys in the start prefix path. This will help client
to continue request for next page.
# Will provide one more param -> {{{}offset{}}}. So e.g. if client provides
{{offset}} as {{0}} and count as {{{}10{}}}, then server will give first {{10}}
records. And if client provides {{offset}} as {{100}} and count as {{{}50{}}},
then server will skip first {{100}} records and provides next {{50}} records.
This mean, API will apply filters like *{{replicationType}}* , *{{keySize}}*
and *{{creationTime}}* on set of records returned after *{{startPrefix}}* and
number of records returned based on *offset* and *count* value specified.
Similar behavior will be for *LEGACY* buckets
Similar behavior will be for *FSO* buckets except the difference of additional
recursive flag in request to list all keys recursively.
> Recon - listKeys API for listing of OBS , FSO and Legacy bucket keys with
> filters
> ---------------------------------------------------------------------------------
>
> Key: HDDS-10634
> URL: https://issues.apache.org/jira/browse/HDDS-10634
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Recon
> Reporter: Devesh Kumar Singh
> Assignee: Devesh Kumar Singh
> Priority: Major
> Labels: pull-request-available
>
> New API:
> api/v1/namespace/listKeys?startPrefix=/volume1/obs-bucket/&count=105
> h5. Default values of API parameters if not provided:
> # replicationType - empty/null, so effectively will list all types of keys.
> # creationTime - empty string and filter will not be applied, so list out
> keys irrespective of age.
> # keySize - 0 bytes, which means all keys greater than zero bytes will be
> listed, effectively all.
> # startPrefix - /
> # prevKey - ""
> # limit - 1000
>
> h5. Behavior of API:
> Now lets consider, we have following OBS, LEGACY and FSO bucket key/files
> namespace tree structure
> {{For OBS Bucket}}
> * /volume1/obs-bucket/key1
> * /volume1/obs-bucket/key1/key2
> * /volume1/obs-bucket/key1/key2/key3
> * /volume1/obs-bucket/key4
> * /volume1/obs-bucket/key5
> * /volume1/obs-bucket/key6
> {{For LEGACY Bucket}}
> * /volume1/legacy-bucket/key
> * /volume1/legacy-bucket/key1/key2
> * /volume1/legacy-bucket/key1/key2/key3
> * /volume1/legacy-bucket/key4
> * /volume1/legacy-bucket/key5
> * /volume1/legacy-bucket/key6
> {{For FSO Bucket}}
> * /volume1/fso-bucket/dir1/dir2/dir3
> * /volume1/fso-bucket/dir1/testfile
> * /volume1/fso-bucket/dir1/file1
> * /volume1/fso-bucket/dir1/dir2/testfile
> * /volume1/fso-bucket/dir1/dir2/file1
> * /volume1/fso-bucket/dir1/dir2/dir3/testfile
> * /volume1/fso-bucket/dir1/dir2/dir3/file1
> *Input Request for OBS bucket:*
> {{
> `api/v1/keys/listKeys?startPrefix=/volume1/obs-bucket&limit=2&replicationType=RATIS`}}
>
> *Output Response:*
> {{{
> "status": "OK",
> "path": "/volume1/obs-bucket",
> "replicatedDataSize": 20971520,
> "unReplicatedDataSize": 20971520,
> "keyCount": 2,
> "lastKey": "/volume1/obs-bucket/key1/key2",
> "keys": [
> \{
> "key": "/volume1/obs-bucket/key1",
> "path": "key1",
> "inStateSince": 1715174266126,
> "size": 10485760,
> "replicatedSize": 10485760,
> "replicationInfo": {
> "replicationFactor": "ONE",
> "requiredNodes": 1,
> "replicationType": "RATIS"
> },
> "creationTime": 1715174266126,
> "modificationTime": 1715174267480,
> "isKey": true
> },
> \{
> "key": "/volume1/obs-bucket/key1/key2",
> "path": "key1/key2",
> "inStateSince": 1715174269510,
> "size": 10485760,
> "replicatedSize": 10485760,
> "replicationInfo": {
> "replicationFactor": "ONE",
> "requiredNodes": 1,
> "replicationType": "RATIS"
> },
> "creationTime": 1715174269510,
> "modificationTime": 1715174270410,
> "isKey": true
> }
> ]
> }}}
>
> *Input Request for FSO bucket:*
> {{
> `api/v1/keys/listKeys?startPrefix=/volume1/fso-bucket&limit=2&replicationType=RATIS`}}
>
> *Output Response:*
> {{{
> "status": "OK",
> "path": "/volume1/fso-bucket",
> "replicatedDataSize": 62914560,
> "unReplicatedDataSize": 20971520,
> "keyCount": 2,
> "lastKey":
> "/-9223372036854775552/-9223372036854775040/-9223372036854774525/testfile",
> "keys": [
> \{
> "key":
> "/-9223372036854775552/-9223372036854775040/-9223372036854774525/file1",
> "path": "file1",
> "inStateSince": 1715174237440,
> "size": 10485760,
> "replicatedSize": 31457280,
> "replicationInfo": {
> "replicationFactor": "THREE",
> "requiredNodes": 3,
> "replicationType": "RATIS"
> },
> "creationTime": 1715174237440,
> "modificationTime": 1715174238161,
> "isKey": true
> },
> \{
> "key":
> "/-9223372036854775552/-9223372036854775040/-9223372036854774525/testfile",
> "path": "testfile",
> "inStateSince": 1715174234840,
> "size": 10485760,
> "replicatedSize": 31457280,
> "replicationInfo": {
> "replicationFactor": "THREE",
> "requiredNodes": 3,
> "replicationType": "RATIS"
> },
> "creationTime": 1715174234840,
> "modificationTime": 1715174235562,
> "isKey": true
> }
> ]
> }}}
>
> *Input Request for Legacy bucket:*
> {{
> `api/v1/keys/listKeys?startPrefix=/volume1/legacy-bucket&limit=2&replicationType=RATIS`}}
>
> *Output Response:*
> {{{
> "status": "OK",
> "path": "/volume1/legacy-bucket",
> "replicatedDataSize": 52428800,
> "unReplicatedDataSize": 52428800,
> "keyCount": 2,
> "lastKey": "/volume1/legacy-bucket/key1/key2",
> "keys": [
> \{
> "key": "/volume1/legacy-bucket/key1",
> "path": "key1",
> "inStateSince": 1715174303702,
> "size": 10485760,
> "replicatedSize": 10485760,
> "replicationInfo": {
> "replicationFactor": "ONE",
> "requiredNodes": 1,
> "replicationType": "RATIS"
> },
> "creationTime": 1715174303702,
> "modificationTime": 1715174304619,
> "isKey": true
> },
> \{
> "key": "/volume1/legacy-bucket/key1/key2",
> "path": "key1/key2",
> "inStateSince": 1715174306641,
> "size": 41943040,
> "replicatedSize": 41943040,
> "replicationInfo": {
> "replicationFactor": "ONE",
> "requiredNodes": 1,
> "replicationType": "RATIS"
> },
> "creationTime": 1715174306641,
> "modificationTime": 1715174307994,
> "isKey": true
> }
> ]
> }}}
> h5.
> This mean, API will apply filters like *{{replicationType}}* , *{{keySize}}*
> and *{{creationTime}}* on set of records returned after *{{startPrefix}}*
> and number of records returned based *limit* value specified. Pagination
> support will be based on *prevKey* and *limit.*
> *Pagination:*
> Now lets say we have total 6 records for a given set of filters and params
> passed and client wants to fetch in total 3 pages , each page 2 records. If
> each page we will retrieve 2 items, so till we get empty last key, we'll
> continue to fetch and empty last key signifies the last page.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
