devmadhuu opened a new pull request, #6658:
URL: https://github.com/apache/ozone/pull/6658
This PR adds a new API in Recon for listing keys for OBS buckets, Legacy
buckets with filters and recursively in a flat structure for FSO buckets.
New API:
api/v1/namespace/listKeys?startPrefix=/volume1/obs-bucket/&count=105
Default values of API parameters if not provided:
1. `replicationType` - empty string and filter will not be applied, so list
out all keys irrespective of replication type.
2. `creationTime` - empty string and filter will not be applied, so list out
keys irrespective of age, else list out keys which got created on or after
provided creationTime
3. `keySize` - 0 bytes, which means all keys greater than zero bytes will be
listed, effectively all.
4. `startPrefix` - /
5. `prevKey` - ""
6. `limit` - 1000
Behavior of API:
For OBS bucket - list out `limit` number of keys on the provided path.
This API will implement pagination support using `prevKey` and `limit`
params.
Get List of All Keys:
GET /api/v1/keys/listKeys
API params:
1. `replicationType` - Filter for RATIS or EC replication keys
2. `creationDate` in "MM-dd-yyyy HH:mm:ss" string format.
3. `startPrefix`
4. `prevKey`
5. `limit`
6. `keySize`
Now lets consider, we have following OBS, LEGACY and FSO bucket key/files
namespace tree structure
`For OBS Bucket`
- /volume1/obs-bucket/key1
- /volume1/obs-bucket/key1/key2
- /volume1/obs-bucket/key1/key2/key3
- /volume1/obs-bucket/key4
- /volume1/obs-bucket/key5
- /volume1/obs-bucket/key6
`For LEGACY Bucket`
- /volume1/legacy-bucket/key
- /volume1/legacy-bucket/key1/key2
- /volume1/legacy-bucket/key1/key2/key3
- /volume1/legacy-bucket/key4
- /volume1/legacy-bucket/key5
- /volume1/legacy-bucket/key6
`For FSO Bucket`
- /volume1/fso-bucket/dir1/dir2/dir3
- /volume1/fso-bucket/dir1/testfile
- /volume1/fso-bucket/dir1/file1
- /volume1/fso-bucket/dir1/dir2/testfile
- /volume1/fso-bucket/dir1/dir2/file1
- /volume1/fso-bucket/dir1/dir2/dir3/testfile
- /volume1/fso-bucket/dir1/dir2/dir3/file1
**Input Request for OBS bucket:**
`api/v1/keys/listKeys?startPrefix=/volume1/obs-bucket&limit=2&replicationType=RATIS`
**Output Response:**
```
{
"status": "OK",
"path": "/volume1/obs-bucket",
"replicatedDataSize": 20971520,
"unReplicatedDataSize": 20971520,
"keyCount": 2,
"lastKey": "/volume1/obs-bucket/key1/key2",
"keys": [
{
"key": "/volume1/obs-bucket/key1",
"path": "key1",
"inStateSince": 1715174266126,
"size": 10485760,
"replicatedSize": 10485760,
"replicationInfo": {
"replicationFactor": "ONE",
"requiredNodes": 1,
"replicationType": "RATIS"
},
"creationTime": 1715174266126,
"modificationTime": 1715174267480,
"isKey": true
},
{
"key": "/volume1/obs-bucket/key1/key2",
"path": "key1/key2",
"inStateSince": 1715174269510,
"size": 10485760,
"replicatedSize": 10485760,
"replicationInfo": {
"replicationFactor": "ONE",
"requiredNodes": 1,
"replicationType": "RATIS"
},
"creationTime": 1715174269510,
"modificationTime": 1715174270410,
"isKey": true
}
]
}
```
**Input Request for FSO bucket:**
`api/v1/keys/listKeys?startPrefix=/volume1/fso-bucket&limit=2&replicationType=RATIS`
**Output Response:**
```
{
"status": "OK",
"path": "/volume1/fso-bucket",
"replicatedDataSize": 62914560,
"unReplicatedDataSize": 20971520,
"keyCount": 2,
"lastKey":
"/-9223372036854775552/-9223372036854775040/-9223372036854774525/testfile",
"keys": [
{
"key":
"/-9223372036854775552/-9223372036854775040/-9223372036854774525/file1",
"path": "file1",
"inStateSince": 1715174237440,
"size": 10485760,
"replicatedSize": 31457280,
"replicationInfo": {
"replicationFactor": "THREE",
"requiredNodes": 3,
"replicationType": "RATIS"
},
"creationTime": 1715174237440,
"modificationTime": 1715174238161,
"isKey": true
},
{
"key":
"/-9223372036854775552/-9223372036854775040/-9223372036854774525/testfile",
"path": "testfile",
"inStateSince": 1715174234840,
"size": 10485760,
"replicatedSize": 31457280,
"replicationInfo": {
"replicationFactor": "THREE",
"requiredNodes": 3,
"replicationType": "RATIS"
},
"creationTime": 1715174234840,
"modificationTime": 1715174235562,
"isKey": true
}
]
}
```
**Input Request for Legacy bucket:**
`api/v1/keys/listKeys?startPrefix=/volume1/legacy-bucket&limit=2&replicationType=RATIS`
**Output Response:**
```
{
"status": "OK",
"path": "/volume1/legacy-bucket",
"replicatedDataSize": 52428800,
"unReplicatedDataSize": 52428800,
"keyCount": 2,
"lastKey": "/volume1/legacy-bucket/key1/key2",
"keys": [
{
"key": "/volume1/legacy-bucket/key1",
"path": "key1",
"inStateSince": 1715174303702,
"size": 10485760,
"replicatedSize": 10485760,
"replicationInfo": {
"replicationFactor": "ONE",
"requiredNodes": 1,
"replicationType": "RATIS"
},
"creationTime": 1715174303702,
"modificationTime": 1715174304619,
"isKey": true
},
{
"key": "/volume1/legacy-bucket/key1/key2",
"path": "key1/key2",
"inStateSince": 1715174306641,
"size": 41943040,
"replicatedSize": 41943040,
"replicationInfo": {
"replicationFactor": "ONE",
"requiredNodes": 1,
"replicationType": "RATIS"
},
"creationTime": 1715174306641,
"modificationTime": 1715174307994,
"isKey": true
}
]
}
```
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10634
## How was this patch tested?
Added Junit test cases and tested various assertions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
