[
https://issues.apache.org/jira/browse/HDDS-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devesh Kumar Singh updated HDDS-10634:
--------------------------------------
Description:
New API:
api/v1/namespace/listKeys?startPrefix=/volume1/obs-bucket/&limit=105
h5. Default values of API parameters if not provided:
# replicationType - empty/null, so effectively will list all types of keys.
# creationTime - empty string and filter will not be applied, so list out keys
irrespective of age.
# keySize - 0 bytes, which means all keys greater than zero bytes will be
listed, effectively all.
# startPrefix - /
# prevKey - ""
# limit - 1000
h5. Behavior of API:
Now lets consider, we have following OBS, LEGACY and FSO bucket key/files
namespace tree structure
{{For OBS Bucket}}
* /volume1/obs-bucket/key1
* /volume1/obs-bucket/key1/key2
* /volume1/obs-bucket/key1/key2/key3
* /volume1/obs-bucket/key4
* /volume1/obs-bucket/key5
* /volume1/obs-bucket/key6
{{For LEGACY Bucket}}
* /volume1/legacy-bucket/key
* /volume1/legacy-bucket/key1/key2
* /volume1/legacy-bucket/key1/key2/key3
* /volume1/legacy-bucket/key4
* /volume1/legacy-bucket/key5
* /volume1/legacy-bucket/key6
{{For FSO Bucket}}
* /volume1/fso-bucket/dir1/dir2/dir3
* /volume1/fso-bucket/dir1/testfile
* /volume1/fso-bucket/dir1/file1
* /volume1/fso-bucket/dir1/dir2/testfile
* /volume1/fso-bucket/dir1/dir2/file1
* /volume1/fso-bucket/dir1/dir2/dir3/testfile
* /volume1/fso-bucket/dir1/dir2/dir3/file1
*Input Request for OBS bucket:*
\{{
`api/v1/keys/listKeys?startPrefix=/volume1/obs-bucket&limit=2&replicationType=RATIS`}}
*Output Response:*
{{{
"status": "OK",
"path": "/volume1/obs-bucket",
"replicatedDataSize": 20971520,
"unReplicatedDataSize": 20971520,
"keyCount": 2,
"lastKey": "/volume1/obs-bucket/key1/key2",
"keys": [
{
"key": "/volume1/obs-bucket/key1",
"path": "key1",
"inStateSince": 1715174266126,
"size": 10485760,
"replicatedSize": 10485760,
"replicationInfo":
{ "replicationFactor": "ONE", "requiredNodes": 1, "replicationType": "RATIS" }
,
"creationTime": 1715174266126,
"modificationTime": 1715174267480,
"isKey": true
},
{
"key": "/volume1/obs-bucket/key1/key2",
"path": "key1/key2",
"inStateSince": 1715174269510,
"size": 10485760,
"replicatedSize": 10485760,
"replicationInfo":
{ "replicationFactor": "ONE", "requiredNodes": 1, "replicationType": "RATIS" }
,
"creationTime": 1715174269510,
"modificationTime": 1715174270410,
"isKey": true
}
]
}}}
*Input Request for FSO bucket:*
\{{
`api/v1/keys/listKeys?startPrefix=/volume1/fso-bucket&limit=2&replicationType=RATIS`}}
*Output Response:*
{{{
"status": "OK",
"path": "/volume1/fso-bucket",
"replicatedDataSize": 62914560,
"unReplicatedDataSize": 20971520,
"keyCount": 2,
"lastKey":
"/-9223372036854775552/-9223372036854775040/-9223372036854774525/testfile",
"keys": [
{
"key": "/-9223372036854775552/-9223372036854775040/-9223372036854774525/file1",
"path": "file1",
"inStateSince": 1715174237440,
"size": 10485760,
"replicatedSize": 31457280,
"replicationInfo":
{ "replicationFactor": "THREE", "requiredNodes": 3, "replicationType": "RATIS" }
,
"creationTime": 1715174237440,
"modificationTime": 1715174238161,
"isKey": true
},
{
"key":
"/-9223372036854775552/-9223372036854775040/-9223372036854774525/testfile",
"path": "testfile",
"inStateSince": 1715174234840,
"size": 10485760,
"replicatedSize": 31457280,
"replicationInfo":
{ "replicationFactor": "THREE", "requiredNodes": 3, "replicationType": "RATIS" }
,
"creationTime": 1715174234840,
"modificationTime": 1715174235562,
"isKey": true
}
]
}}}
*Input Request for Legacy bucket:*
\{{
`api/v1/keys/listKeys?startPrefix=/volume1/legacy-bucket&limit=2&replicationType=RATIS`}}
*Output Response:*
{{{
"status": "OK",
"path": "/volume1/legacy-bucket",
"replicatedDataSize": 52428800,
"unReplicatedDataSize": 52428800,
"keyCount": 2,
"lastKey": "/volume1/legacy-bucket/key1/key2",
"keys": [
{
"key": "/volume1/legacy-bucket/key1",
"path": "key1",
"inStateSince": 1715174303702,
"size": 10485760,
"replicatedSize": 10485760,
"replicationInfo":
{ "replicationFactor": "ONE", "requiredNodes": 1, "replicationType": "RATIS" }
,
"creationTime": 1715174303702,
"modificationTime": 1715174304619,
"isKey": true
},
{
"key": "/volume1/legacy-bucket/key1/key2",
"path": "key1/key2",
"inStateSince": 1715174306641,
"size": 41943040,
"replicatedSize": 41943040,
"replicationInfo":
{ "replicationFactor": "ONE", "requiredNodes": 1, "replicationType": "RATIS" }
,
"creationTime": 1715174306641,
"modificationTime": 1715174307994,
"isKey": true
}
]
}}}
h5.
This mean, API will apply filters like *{{replicationType}}* , *{{keySize}}*
and *{{creationTime}}* on set of records returned after *{{startPrefix}}* and
number of records returned based *limit* value specified. Pagination support
will be based on *prevKey* and *limit.*
*Pagination:*
Now lets say we have total 6 records for a given set of filters and params
passed and client wants to fetch in total 3 pages , each page 2 records. If
each page we will retrieve 2 items, so till we get empty last key, we'll
continue to fetch and empty last key signifies the last page.
was:
New API:
api/v1/namespace/listKeys?startPrefix=/volume1/obs-bucket/&count=105
h5. Default values of API parameters if not provided:
# replicationType - empty/null, so effectively will list all types of keys.
# creationTime - empty string and filter will not be applied, so list out keys
irrespective of age.
# keySize - 0 bytes, which means all keys greater than zero bytes will be
listed, effectively all.
# startPrefix - /
# prevKey - ""
# limit - 1000
h5. Behavior of API:
Now lets consider, we have following OBS, LEGACY and FSO bucket key/files
namespace tree structure
{{For OBS Bucket}}
* /volume1/obs-bucket/key1
* /volume1/obs-bucket/key1/key2
* /volume1/obs-bucket/key1/key2/key3
* /volume1/obs-bucket/key4
* /volume1/obs-bucket/key5
* /volume1/obs-bucket/key6
{{For LEGACY Bucket}}
* /volume1/legacy-bucket/key
* /volume1/legacy-bucket/key1/key2
* /volume1/legacy-bucket/key1/key2/key3
* /volume1/legacy-bucket/key4
* /volume1/legacy-bucket/key5
* /volume1/legacy-bucket/key6
{{For FSO Bucket}}
* /volume1/fso-bucket/dir1/dir2/dir3
* /volume1/fso-bucket/dir1/testfile
* /volume1/fso-bucket/dir1/file1
* /volume1/fso-bucket/dir1/dir2/testfile
* /volume1/fso-bucket/dir1/dir2/file1
* /volume1/fso-bucket/dir1/dir2/dir3/testfile
* /volume1/fso-bucket/dir1/dir2/dir3/file1
*Input Request for OBS bucket:*
{{
`api/v1/keys/listKeys?startPrefix=/volume1/obs-bucket&limit=2&replicationType=RATIS`}}
*Output Response:*
{{{
"status": "OK",
"path": "/volume1/obs-bucket",
"replicatedDataSize": 20971520,
"unReplicatedDataSize": 20971520,
"keyCount": 2,
"lastKey": "/volume1/obs-bucket/key1/key2",
"keys": [
\{
"key": "/volume1/obs-bucket/key1",
"path": "key1",
"inStateSince": 1715174266126,
"size": 10485760,
"replicatedSize": 10485760,
"replicationInfo": {
"replicationFactor": "ONE",
"requiredNodes": 1,
"replicationType": "RATIS"
},
"creationTime": 1715174266126,
"modificationTime": 1715174267480,
"isKey": true
},
\{
"key": "/volume1/obs-bucket/key1/key2",
"path": "key1/key2",
"inStateSince": 1715174269510,
"size": 10485760,
"replicatedSize": 10485760,
"replicationInfo": {
"replicationFactor": "ONE",
"requiredNodes": 1,
"replicationType": "RATIS"
},
"creationTime": 1715174269510,
"modificationTime": 1715174270410,
"isKey": true
}
]
}}}
*Input Request for FSO bucket:*
{{
`api/v1/keys/listKeys?startPrefix=/volume1/fso-bucket&limit=2&replicationType=RATIS`}}
*Output Response:*
{{{
"status": "OK",
"path": "/volume1/fso-bucket",
"replicatedDataSize": 62914560,
"unReplicatedDataSize": 20971520,
"keyCount": 2,
"lastKey":
"/-9223372036854775552/-9223372036854775040/-9223372036854774525/testfile",
"keys": [
\{
"key":
"/-9223372036854775552/-9223372036854775040/-9223372036854774525/file1",
"path": "file1",
"inStateSince": 1715174237440,
"size": 10485760,
"replicatedSize": 31457280,
"replicationInfo": {
"replicationFactor": "THREE",
"requiredNodes": 3,
"replicationType": "RATIS"
},
"creationTime": 1715174237440,
"modificationTime": 1715174238161,
"isKey": true
},
\{
"key":
"/-9223372036854775552/-9223372036854775040/-9223372036854774525/testfile",
"path": "testfile",
"inStateSince": 1715174234840,
"size": 10485760,
"replicatedSize": 31457280,
"replicationInfo": {
"replicationFactor": "THREE",
"requiredNodes": 3,
"replicationType": "RATIS"
},
"creationTime": 1715174234840,
"modificationTime": 1715174235562,
"isKey": true
}
]
}}}
*Input Request for Legacy bucket:*
{{
`api/v1/keys/listKeys?startPrefix=/volume1/legacy-bucket&limit=2&replicationType=RATIS`}}
*Output Response:*
{{{
"status": "OK",
"path": "/volume1/legacy-bucket",
"replicatedDataSize": 52428800,
"unReplicatedDataSize": 52428800,
"keyCount": 2,
"lastKey": "/volume1/legacy-bucket/key1/key2",
"keys": [
\{
"key": "/volume1/legacy-bucket/key1",
"path": "key1",
"inStateSince": 1715174303702,
"size": 10485760,
"replicatedSize": 10485760,
"replicationInfo": {
"replicationFactor": "ONE",
"requiredNodes": 1,
"replicationType": "RATIS"
},
"creationTime": 1715174303702,
"modificationTime": 1715174304619,
"isKey": true
},
\{
"key": "/volume1/legacy-bucket/key1/key2",
"path": "key1/key2",
"inStateSince": 1715174306641,
"size": 41943040,
"replicatedSize": 41943040,
"replicationInfo": {
"replicationFactor": "ONE",
"requiredNodes": 1,
"replicationType": "RATIS"
},
"creationTime": 1715174306641,
"modificationTime": 1715174307994,
"isKey": true
}
]
}}}
h5.
This mean, API will apply filters like *{{replicationType}}* , *{{keySize}}*
and *{{creationTime}}* on set of records returned after *{{startPrefix}}* and
number of records returned based *limit* value specified. Pagination support
will be based on *prevKey* and *limit.*
*Pagination:*
Now lets say we have total 6 records for a given set of filters and params
passed and client wants to fetch in total 3 pages , each page 2 records. If
each page we will retrieve 2 items, so till we get empty last key, we'll
continue to fetch and empty last key signifies the last page.
> Recon - listKeys API for listing of OBS , FSO and Legacy bucket keys with
> filters
> ---------------------------------------------------------------------------------
>
> Key: HDDS-10634
> URL: https://issues.apache.org/jira/browse/HDDS-10634
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Recon
> Reporter: Devesh Kumar Singh
> Assignee: Devesh Kumar Singh
> Priority: Major
> Labels: pull-request-available
>
> New API:
> api/v1/namespace/listKeys?startPrefix=/volume1/obs-bucket/&limit=105
> h5. Default values of API parameters if not provided:
> # replicationType - empty/null, so effectively will list all types of keys.
> # creationTime - empty string and filter will not be applied, so list out
> keys irrespective of age.
> # keySize - 0 bytes, which means all keys greater than zero bytes will be
> listed, effectively all.
> # startPrefix - /
> # prevKey - ""
> # limit - 1000
>
> h5. Behavior of API:
> Now lets consider, we have following OBS, LEGACY and FSO bucket key/files
> namespace tree structure
> {{For OBS Bucket}}
> * /volume1/obs-bucket/key1
> * /volume1/obs-bucket/key1/key2
> * /volume1/obs-bucket/key1/key2/key3
> * /volume1/obs-bucket/key4
> * /volume1/obs-bucket/key5
> * /volume1/obs-bucket/key6
> {{For LEGACY Bucket}}
> * /volume1/legacy-bucket/key
> * /volume1/legacy-bucket/key1/key2
> * /volume1/legacy-bucket/key1/key2/key3
> * /volume1/legacy-bucket/key4
> * /volume1/legacy-bucket/key5
> * /volume1/legacy-bucket/key6
> {{For FSO Bucket}}
> * /volume1/fso-bucket/dir1/dir2/dir3
> * /volume1/fso-bucket/dir1/testfile
> * /volume1/fso-bucket/dir1/file1
> * /volume1/fso-bucket/dir1/dir2/testfile
> * /volume1/fso-bucket/dir1/dir2/file1
> * /volume1/fso-bucket/dir1/dir2/dir3/testfile
> * /volume1/fso-bucket/dir1/dir2/dir3/file1
> *Input Request for OBS bucket:*
> \{{
> `api/v1/keys/listKeys?startPrefix=/volume1/obs-bucket&limit=2&replicationType=RATIS`}}
>
> *Output Response:*
> {{{
> "status": "OK",
> "path": "/volume1/obs-bucket",
> "replicatedDataSize": 20971520,
> "unReplicatedDataSize": 20971520,
> "keyCount": 2,
> "lastKey": "/volume1/obs-bucket/key1/key2",
> "keys": [
> {
> "key": "/volume1/obs-bucket/key1",
> "path": "key1",
> "inStateSince": 1715174266126,
> "size": 10485760,
> "replicatedSize": 10485760,
> "replicationInfo":
> { "replicationFactor": "ONE", "requiredNodes": 1, "replicationType": "RATIS" }
> ,
> "creationTime": 1715174266126,
> "modificationTime": 1715174267480,
> "isKey": true
> },
> {
> "key": "/volume1/obs-bucket/key1/key2",
> "path": "key1/key2",
> "inStateSince": 1715174269510,
> "size": 10485760,
> "replicatedSize": 10485760,
> "replicationInfo":
> { "replicationFactor": "ONE", "requiredNodes": 1, "replicationType": "RATIS" }
> ,
> "creationTime": 1715174269510,
> "modificationTime": 1715174270410,
> "isKey": true
> }
> ]
> }}}
>
> *Input Request for FSO bucket:*
> \{{
> `api/v1/keys/listKeys?startPrefix=/volume1/fso-bucket&limit=2&replicationType=RATIS`}}
>
> *Output Response:*
> {{{
> "status": "OK",
> "path": "/volume1/fso-bucket",
> "replicatedDataSize": 62914560,
> "unReplicatedDataSize": 20971520,
> "keyCount": 2,
> "lastKey":
> "/-9223372036854775552/-9223372036854775040/-9223372036854774525/testfile",
> "keys": [
> {
> "key":
> "/-9223372036854775552/-9223372036854775040/-9223372036854774525/file1",
> "path": "file1",
> "inStateSince": 1715174237440,
> "size": 10485760,
> "replicatedSize": 31457280,
> "replicationInfo":
> { "replicationFactor": "THREE", "requiredNodes": 3, "replicationType":
> "RATIS" }
> ,
> "creationTime": 1715174237440,
> "modificationTime": 1715174238161,
> "isKey": true
> },
> {
> "key":
> "/-9223372036854775552/-9223372036854775040/-9223372036854774525/testfile",
> "path": "testfile",
> "inStateSince": 1715174234840,
> "size": 10485760,
> "replicatedSize": 31457280,
> "replicationInfo":
> { "replicationFactor": "THREE", "requiredNodes": 3, "replicationType":
> "RATIS" }
> ,
> "creationTime": 1715174234840,
> "modificationTime": 1715174235562,
> "isKey": true
> }
> ]
> }}}
>
> *Input Request for Legacy bucket:*
> \{{
> `api/v1/keys/listKeys?startPrefix=/volume1/legacy-bucket&limit=2&replicationType=RATIS`}}
>
> *Output Response:*
> {{{
> "status": "OK",
> "path": "/volume1/legacy-bucket",
> "replicatedDataSize": 52428800,
> "unReplicatedDataSize": 52428800,
> "keyCount": 2,
> "lastKey": "/volume1/legacy-bucket/key1/key2",
> "keys": [
> {
> "key": "/volume1/legacy-bucket/key1",
> "path": "key1",
> "inStateSince": 1715174303702,
> "size": 10485760,
> "replicatedSize": 10485760,
> "replicationInfo":
> { "replicationFactor": "ONE", "requiredNodes": 1, "replicationType": "RATIS" }
> ,
> "creationTime": 1715174303702,
> "modificationTime": 1715174304619,
> "isKey": true
> },
> {
> "key": "/volume1/legacy-bucket/key1/key2",
> "path": "key1/key2",
> "inStateSince": 1715174306641,
> "size": 41943040,
> "replicatedSize": 41943040,
> "replicationInfo":
> { "replicationFactor": "ONE", "requiredNodes": 1, "replicationType": "RATIS" }
> ,
> "creationTime": 1715174306641,
> "modificationTime": 1715174307994,
> "isKey": true
> }
> ]
> }}}
> h5.
> This mean, API will apply filters like *{{replicationType}}* , *{{keySize}}*
> and *{{creationTime}}* on set of records returned after *{{startPrefix}}*
> and number of records returned based *limit* value specified. Pagination
> support will be based on *prevKey* and *limit.*
> *Pagination:*
> Now lets say we have total 6 records for a given set of filters and params
> passed and client wants to fetch in total 3 pages , each page 2 records. If
> each page we will retrieve 2 items, so till we get empty last key, we'll
> continue to fetch and empty last key signifies the last page.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
