devmadhuu opened a new pull request, #6503:
URL: https://github.com/apache/ozone/pull/6503
## What changes were proposed in this pull request?
This PR adds a new API in Recon for listing keys for OBS buckets, Legacy
buckets with filters and recursively in a flat structure for FSO buckets.
New API:
api/v1/namespace/listKeys?startPrefix=/volume1/obs-bucket/&count=105
Default values of API parameters if not provided:
1. replicationType - RATIS
2. creationTime - empty string and filter will not be applied, so list
out keys irrespective of age, else list out keys which got created on or after
provided creationTime
3. keySize - 0 bytes, which means all keys greater than zero bytes will
be listed, effectively all.
4. startPrefix - /
5. count - 1000
Behavior of API:
For OBS bucket - list out count number of keys on the provided path.
This API will implement pagination support using count params.
Get List of All Keys:
GET /api/v1/namespace/listKeys
API params:
1. replicationType
2. creationTime in "MM-dd-yyyy HH:mm:ss" string format.
3. startPrefix
4. count
5. keySize
**Input Request for OBS bucket:**
`api/v1/namespace/listKeys?startPrefix=/volume1/obs-bucket/&count=105`
**Output Response:**
```
{
"status": "OK",
"path": "/volume1/obs-bucket",
"size": 73400320,
"sizeWithReplica": 81788928,
"subPathCount": 6,
"subPaths": [
{
"key": true,
"path": "key1",
"size": 10485760,
"sizeWithReplica": 10485760,
"isKey": true,
"replicationType": "RATIS",
"creationTime": 1712680854675,
"modificationTime": 1712680855695
},
{
"key": true,
"path": "key1/key2",
"size": 10485760,
"sizeWithReplica": 10485760,
"isKey": true,
"replicationType": "RATIS",
"creationTime": 1712680857753,
"modificationTime": 1712680858666
},
{
"key": true,
"path": "key1/key2/key3",
"size": 10485760,
"sizeWithReplica": 10485760,
"isKey": true,
"replicationType": "RATIS",
"creationTime": 1712680860801,
"modificationTime": 1712680861870
},
{
"key": true,
"path": "key4",
"size": 10485760,
"sizeWithReplica": 10485760,
"isKey": true,
"replicationType": "RATIS",
"creationTime": 1712680863937,
"modificationTime": 1712680864899
},
{
"key": true,
"path": "key5",
"size": 10485760,
"sizeWithReplica": 10485760,
"isKey": true,
"replicationType": "RATIS",
"creationTime": 1712680866996,
"modificationTime": 1712680868187
},
{
"key": true,
"path": "key6",
"size": 10485760,
"sizeWithReplica": 10485760,
"isKey": true,
"replicationType": "RATIS",
"creationTime": 1712680870182,
"modificationTime": 1712680871044
}
],
"sizeDirectKey": 73400320
}
```
**Input Request for FSO bucket:**
`api/v1/namespace/listKeys?startPrefix=/volume1/fso-bucket&recursive=true`
**Output Response:**
```
{
"status": "OK",
"path": "/volume1/fso-bucket",
"size": 62914560,
"sizeWithReplica": 188743680,
"subPathCount": 6,
"subPaths": [
{
"key": true,
"path": "volume1/fso-bucket/dir1/file1",
"size": 10485760,
"sizeWithReplica": 31457280,
"isKey": true,
"replicationType": "RATIS",
"creationTime": 1712680835581,
"modificationTime": 1712680836508
},
{
"key": true,
"path": "volume1/fso-bucket/dir1/testfile",
"size": 10485760,
"sizeWithReplica": 31457280,
"isKey": true,
"replicationType": "RATIS",
"creationTime": 1712680832118,
"modificationTime": 1712680833528
},
{
"key": true,
"path": "volume1/fso-bucket/dir1/dir2/file1",
"size": 10485760,
"sizeWithReplica": 31457280,
"isKey": true,
"replicationType": "RATIS",
"creationTime": 1712680841158,
"modificationTime": 1712680842040
},
{
"key": true,
"path": "volume1/fso-bucket/dir1/dir2/testfile",
"size": 10485760,
"sizeWithReplica": 31457280,
"isKey": true,
"replicationType": "RATIS",
"creationTime": 1712680838434,
"modificationTime": 1712680839254
},
{
"key": true,
"path": "volume1/fso-bucket/dir1/dir2/dir3/file1",
"size": 10485760,
"sizeWithReplica": 31457280,
"isKey": true,
"replicationType": "RATIS",
"creationTime": 1712680847287,
"modificationTime": 1712680850660
},
{
"key": true,
"path": "volume1/fso-bucket/dir1/dir2/dir3/testfile",
"size": 10485760,
"sizeWithReplica": 31457280,
"isKey": true,
"replicationType": "RATIS",
"creationTime": 1712680843959,
"modificationTime": 1712680844890
}
],
"sizeDirectKey": 0
}
```
**Input Request for LEGACY bucket:**
`api/v1/namespace/listKeys?startPrefix=/volume1/legacy-bucket
**Output Response:**
```
{
"status": "OK",
"path": "/volume1/legacy-bucket",
"size": 157286400,
"sizeWithReplica": 157286400,
"subPathCount": 6,
"subPaths": [
{
"key": true,
"path": "key1",
"size": 10485760,
"sizeWithReplica": 10485760,
"isKey": true,
"replicationType": "RATIS",
"creationTime": 1712680878239,
"modificationTime": 1712680879179
},
{
"key": true,
"path": "key1/key2",
"size": 41943040,
"sizeWithReplica": 41943040,
"isKey": true,
"replicationType": "RATIS",
"creationTime": 1712680881331,
"modificationTime": 1712680882611
},
{
"key": true,
"path": "key1/key2/key3",
"size": 10485760,
"sizeWithReplica": 10485760,
"isKey": true,
"replicationType": "RATIS",
"creationTime": 1712680884664,
"modificationTime": 1712680885522
},
{
"key": true,
"path": "key4",
"size": 41943040,
"sizeWithReplica": 41943040,
"isKey": true,
"replicationType": "RATIS",
"creationTime": 1712680887558,
"modificationTime": 1712680888590
},
{
"key": true,
"path": "key5",
"size": 10485760,
"sizeWithReplica": 10485760,
"isKey": true,
"replicationType": "RATIS",
"creationTime": 1712680890644,
"modificationTime": 1712680891447
},
{
"key": true,
"path": "key6",
"size": 41943040,
"sizeWithReplica": 41943040,
"isKey": true,
"replicationType": "RATIS",
"creationTime": 1712680907002,
"modificationTime": 1712680908210
}
],
"sizeDirectKey": 157286400
}
```
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10634
## How was this patch tested?
Added Junit test cases and tested various assertions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
