Hi Jayan,
This looks great and the APIs are clear enough to understand. Specifically,
automatic documentation generation is a very nice feature. I would like to
suggest a few modifications to the replica data models though.
I understand that you got an idea about the replica data models through
Airavata but there are a few modifications/generalizations that we need to
do here. For example, the Replica location category [6] might not make
sense here as you already specify the storage type. UserInfo, GroupInfo,
and Permissions in the same proto file do not apply to this design as those
are handled at the Data Catalog level. What is your expectation in
ReplicaListEntry below?
message ReplicaListEntry {
string data_replica_id = 1;
string replica_name = 2;
StorageType storage_type = 3;
}
If you are planning to provide the grouping of Replica Items through that,
I suggest updating it in the following way
message ReplicaGroupEntry {
string replica_group_id = 1
repeated ReplicaGroupEntry directories = 2;
repeated DataReplicaLocation files = 3;
}
This will provide both grouping and hierarchical replica registration and
you can emulate it as a virtual file hierarchy.
[6]
https://github.com/Jayancv/airavata-replica-catalog/blob/master/replica-catalog-api/stubs/src/main/proto/catalogapi/ReplicaCatalogAPI.proto#L60
Thanks
Dimuthu
On Sat, Feb 18, 2023 at 3:10 PM Jayan Vidanapathirana <
jcvidanapathir...@gmail.com> wrote:
> Hi All,
>
> As a new contributor to the Cybershuttle project, I have been actively
> involved in implementing the Data Replica Catalog. This new catalog is
> designed to interface with both the Apache Airavata Data Catalog [1] and
> Airavata MFT[2]. This replica catalog should be able to store each replica
> resource storage details and secret/credential details specific to the
> storage type. The proposed high-level architecture will be as follows:
>
>
> I will mainly work on the highlighted area (red color box) and as an
> initial step started defining APIs which communicate with Replica Catalog.
> This API calls will be gRPC APIs and following methods will be implement,
>
> Replica Registration
>
>
> 1.
>
> registerReplicaLocation(DataReplicaCreateRequest createRequest)
> 2.
>
> updateReplicaLocation(DataReplicaCreateRequest updateRequest)
> 3.
>
> DataReplicaLocationModel getReplicaLocation(DataReplicaGetRequest
> getReplicaRequest)
> 4.
>
> removeReplicaLocation(DataReplicaDeleteRequest deleteReplicaRequest)
> 5.
>
> getAllReplicaLocations(AllDataReplicaGetRequest allDataGetRequest)
> 6.
>
> removeAllReplicaLocations(AllDataReplicaDeleteRequest
> allDataDeleteRequest)
>
>
> Storage Registration
>
> registerSecretForStorage(SecretForStorage request)
>
> deleteSecretsForStorage(SecretForStorageDeleteRequest request)
>
> getSecretForStorage(SecretForStorageGetRequest request)
>
> searchStorages(StorageSearchRequest request)
>
> listStorages(StorageListRequest request)
>
> resolveStorageType (StorageTypeResolveRequest request)
>
> Storage - Internal APIs
>
> S3StorageListResponse listS3Storage(S3StorageListRequest request)
>
> Optional<S3Storage> getS3Storage(S3StorageGetRequest request)
>
> S3Storage createS3Storage(S3StorageCreateRequest request)
>
> boolean updateS3Storage(S3StorageUpdateRequest request)
>
> boolean deleteS3Storage(S3StorageDeleteRequest request)
>
> AzureStorageListResponse listAzureStorage(AzureStorageListRequest request)
>
> Optional<AzureStorage> getAzureStorage(AzureStorageGetRequest request)
>
> AzureStorage createAzureStorage(AzureStorageCreateRequest request)
>
> boolean updateAzureStorage(AzureStorageUpdateRequest request)
>
> boolean deleteAzureStorage(AzureStorageDeleteRequest request)
>
> GCSStorageListResponse listGCSStorage(GCSStorageListRequest request)
>
> Optional<GCSStorage> getGCSStorage(GCSStorageGetRequest request)
>
> GCSStorage createGCSStorage(GCSStorageCreateRequest request)
>
> boolean updateGCSStorage(GCSStorageUpdateRequest request)
>
> boolean deleteGCSStorage(GCSStorageDeleteRequest request)
>
> Secret Registration
>
> registerSecret(SecretRegistrationRequest request)
>
> deleteSecret(SecretDeleteRequest request)
>
> resolveStorageType (StorageTypeResolveRequest request)
>
> Secret - Internal APIs
>
>
> Optional<S3Secret> getS3Secret(S3SecretGetRequest request)
>
> S3Secret createS3Secret(S3SecretCreateRequest request)
>
> boolean updateS3Secret(S3SecretUpdateRequest request)
>
> boolean deleteS3Secret(S3SecretDeleteRequest request)
>
> Optional<AzureSecret> getAzureSecret(AzureSecretGetRequest request)
>
> AzureSecret createAzureSecret(AzureSecretCreateRequest request)
>
> boolean updateAzureSecret(AzureSecretUpdateRequest request)
>
> boolean deleteAzureSecret(AzureSecretDeleteRequest request)
>
> Optional<GCSSecret> getGCSSecret(GCSSecretGetRequest request)
>
> GCSSecret createGCSSecret(GCSSecretCreateRequest request)
>
> boolean updateGCSSecret(GCSSecretUpdateRequest request)
>
> boolean deleteGCSSecret(GCSSecretDeleteRequest request)
>
>
> Poc[3] : https://github.com/Jayancv/airavata-replica-catalog (Defining
> API calls)
>
> Draft APIs : refer the attachment replicaCatalogAPIsDocumentation.html[4]
> which generated using the Poc [3]
>
> I greatly appreciate your thoughts and feedback
> <https://docs.google.com/document/d/1dQUpHVkccx-O9mbYuAo-wtcLQWJ1LaKUzBpaBMCgSac/edit?usp=sharing>
> on the designs[5], as they can help us improve and adopt a more generalized
> approach. Additionally, I would like to identify any other factors that we
> should take into account to minimize potential issues in the future. Are
> there any other considerations that we should keep in mind?
>
>
> [1] - https://github.com/apache/airavata-data-catalog
>
> [2] - https://github.com/apache/airavata-mft
>
> [3] - https://github.com/Jayancv/airavata-replica-catalog
>
> [4] -
> https://drive.google.com/file/d/1C4_H_Y5fZ4-5fmIHBNZyh3lXbV7vL5Ah/view?usp=sharing
>
> [5] -
> https://docs.google.com/document/d/1dQUpHVkccx-O9mbYuAo-wtcLQWJ1LaKUzBpaBMCgSac/edit?usp=sharing
>
> Thanks.
> --
> Best Regards
>
> Jayan Vidanapathirana
> [image: https://lk.linkedin.com/in/jayancv]
> <https://lk.linkedin.com/in/jayancv>
>
