Hi Isuru, I was testing some features with the client sdk and with the recent changes in the develop branch I'm getting a 403 when adding kv credentials. Any idea why this is happening?
Regards On Wed, May 19, 2021 at 7:33 AM Hasitha Jayasundara < [email protected]> wrote: > Hi Isuru, > > Thanks for the reply. Based on your comment I have gone through the > java-sdk for for resource-secret-management-client first as this will be > used in the cert-renewal feature. Several implementations are missing > there, including get/create of certificate credentials. I created an issue > [1] to track the progress of adding missing implementations. WIll send a PR > as soon as possible. > > Regards > > [1]. https://github.com/apache/airavata-custos/issues/164 > <https://github.com/apache/airavata-custos/issues/164> > > On Tue, May 18, 2021 at 9:21 PM Isuru Ranawaka <[email protected]> wrote: > >> Hi Hasitha, >> >> Great progress. You can start Custos integration using >> custos-java-sdk[1]. >> There could be some unimplemented methods and it would be great if you can >> implement and submit PRs. >> >> thanks >> Isuru >> >> >> [1] >> >> https://github.com/apache/airavata-custos/tree/develop/custos-client-sdks/custos-java-sdk >> >> >> On Mon, May 17, 2021 at 11:59 PM Hasitha Jayasundara < >> [email protected]> wrote: >> >> > Hi Isuru, >> > >> > I completed the cert renewal part of the feature and the implementation >> can >> > be found at [1]. >> > >> > At the moment I'm doing some refactoring according to acme protocol best >> > practices. After that I'm hoping to start working on custos integration >> > part of the feature using Java SDK. Basically the idea is to store the >> > updated certificates in Custos. >> > >> > I am using pebble [2] (a lightweight version of letsencrypt boulder CA >> > server) as the CA server for testing purposes. The main reason for using >> > Pebble is that the Lets Encrypt CA server endpoint (staging) is not >> > working as expected in the dev environment for a fake domain. >> > >> > As for the current implementation I'm using a nginx server with Lua >> module >> > as acme challenge server to store files required for completing the >> > challenge. Lua module is used to embed Lua programming language into >> nginx >> > server and create files required for acme challenge. (CGI with Nginx and >> > SpringBoot are two other options for this and I'm thinking of do a >> proper >> > poc/rfc after initial feature implementation is completed) >> > >> > Regards >> > >> > [1]. https://github.com/hasithajayasundara/cert-updater >> > [2]. https://github.com/letsencrypt/pebble >> > >> > On Fri, May 14, 2021 at 2:41 PM Hasitha Jayasundara < >> > [email protected]> wrote: >> > >> > > Hi Isuru, >> > > >> > > Thanks for the explanation. I think I have a clear idea on how to >> proceed >> > > with the Let's Encrypt part of the implementation now. >> > > >> > > I already started the implementation and am currently designing a >> > solution >> > > for handling the Let's Encrypt challenge. Once that is done I am >> thinking >> > > of starting the certificates store process in Custos. I think I'll >> > probably >> > > need some setup completed first for making authenticated requests to >> > Custos. >> > > >> > > Found an issue with the certificate store path in the current >> > > implementation and I've sent a fix for that. [1] >> > > >> > > Regards >> > > >> > > [1]. https://github.com/apache/airavata-custos/pull/160 >> > > >> > > On Thu, May 13, 2021 at 9:31 AM Isuru Ranawaka <[email protected]> >> > wrote: >> > > >> > >> Hi Hasitha, >> > >> >> > >> Glad to hear that you have gone through the code. Please feel free to >> > >> submit bug fixes as you see. >> > >> >> > >> We use HarshiCorp vault to store secrets. The paths you mentioned are >> > >> actually created in the vault. Precisely, owner_id is the id of the >> user >> > >> or >> > >> the service account that owns the certificate. Token is an identifier >> > that >> > >> can be used to retrieve the certificate later. >> > >> >> > >> thanks >> > >> Isuru >> > >> >> > >> On Wed, May 12, 2021 at 11:08 PM Hasitha Jayasundara < >> > >> [email protected]> wrote: >> > >> >> > >> > Hi Isuru, >> > >> > >> > >> > Thanks for the reply. Based on your comment that custos already >> has an >> > >> > endpoint to store certificates I took some time to go through the >> > Secret >> > >> > Management Service and the proto for Resource Secret Service. >> > >> > >> > >> > According to my observation, I think I can use *POST - >> > >> > /resource-secret-management/v1.0.0/secret/certificate* endpoint to >> do >> > >> > certificate uploading. (Correct me if I'm wrong) >> > >> > >> > >> > Going through the code I found that certificate is stored in >> > >> > */resourcesecret/<tenant_id>/<owner_id>/ssh/<token>*. The >> <tenant_id>, >> > >> > <owner_id> and <token> are received through the request body. I am >> not >> > >> 100% >> > >> > clear about <owner_id> and <token> fields in the context of custos >> > >> secret >> > >> > manager service. Could you please explain a bit about <owner_id> >> and >> > the >> > >> > <token>? >> > >> > >> > >> > Regards >> > >> > >> > >> > On Wed, May 12, 2021 at 7:30 AM Isuru Ranawaka <[email protected] >> > >> > >> wrote: >> > >> > >> > >> > > Hi Hasitha, >> > >> > > >> > >> > > We really appreciate your interest in Custos project. Please find >> > the >> > >> > > inline comments. >> > >> > > >> > >> > > On Tue, May 11, 2021 at 9:33 AM Hasitha Jayasundara < >> > >> > > [email protected]> wrote: >> > >> > > >> > >> > > > Hi all, >> > >> > > > >> > >> > > > I'm Hasitha and I'm currently working as a Full Stack Engineer >> for >> > >> a UK >> > >> > > > based tech company. I graduated from University Of Moratuwa >> > >> Department >> > >> > of >> > >> > > > Electronic and Telecommunication Engineering Sri Lanka and >> > >> completed a >> > >> > > > software engineering internship at a company called WSO2 which >> is >> > an >> > >> > open >> > >> > > > source technology provider based in Sri Lanka. I also >> participated >> > >> in >> > >> > > > Google Summer of Code 2017 and successfully completed a project >> > for >> > >> > > > implementing an inbound endpoint and a connector for IBM-MQ. >> > >> > > > >> > >> > > > I'm enthusiastic about the airavata-custos project and would be >> > >> > extremely >> > >> > > > interested in contributing to the project. I've already >> followed >> > the >> > >> > wiki >> > >> > > > page[1] and the tutorial presented at the Gateways 2020 >> Conference >> > >> [2] >> > >> > > and >> > >> > > > I think I have a clear understanding about how custos operates >> in >> > >> the >> > >> > > real >> > >> > > > world. >> > >> > > > >> > >> > > > As a starting point I selected the issue #152 [3] which is to >> > >> > implement a >> > >> > > > cert manager task using Custos. The idea is to automate the >> cert >> > >> > renewal >> > >> > > > process so that the client's won't need to manually do it by >> going >> > >> > > through >> > >> > > > Let's Encrypt cert renewal process. The task (cron job) will be >> > >> > > responsible >> > >> > > > for, >> > >> > > > >> > >> > > > - Connecting to Let’s Encrypt APIs and create certificates, >> > >> update >> > >> > > > certificates >> > >> > > > - Saving updated certificates into a given location and >> where >> > the >> > >> > > server >> > >> > > > application can use it >> > >> > > > >> > >> > > > The overall process can be illustrated as follows. >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > > - Custos Cert Renewer task (Cron Job) will be executed on >> > >> periodic >> > >> > > > configurable time period >> > >> > > > - Task will communicate with Let's Encrypt using >> ACME(Automated >> > >> > > > Certificate Management Environments) protocol. (I've already >> > >> tested >> > >> > > this >> > >> > > > with acme4j [4] which is a java client for ACME protocol and >> > it's >> > >> > > > working >> > >> > > > fine without any issues) >> > >> > > > - Task will then obtain the updated certificate and store it >> > in a >> > >> > > given >> > >> > > > path in Custos (probably in a persistent volume) >> > >> > > > - Client applications can query certificates from Custos and >> > add >> > >> > them >> > >> > > to >> > >> > > > client trustore on demand to call a server application >> > >> > > > >> > >> > > > Following I listed out some of the concerns I'm having at the >> > >> moment. >> > >> > > > >> > >> > > > - In order to obtain an updated certificates, >> > >> > > > >> > >> > > > >> > >> > > > 1. we need to have an account in CA with a public and a >> private >> > >> key >> > >> > > > pair. Do we already have an account in Let's Encrypt for >> this >> > >> type >> > >> > of >> > >> > > > work >> > >> > > > or Is it possible to create a dedicated account for the cert >> > >> renewer >> > >> > > > task? >> > >> > > > >> > >> > > >> > >> > > yes, we can create a test account in Let's Encrypt. >> > >> > > >> > >> > > 2. we need to provide CNs (domain names) for Let's Encrypt. >> Where >> > >> do >> > >> > we >> > >> > > > store the domain names of the clients? >> > >> > > > >> > >> > > >> > >> > > I guess you are referring to the server application domains. >> those >> > >> should >> > >> > > be configurable and can be stored in a configuration file. >> > >> > > >> > >> > > >> > >> > > > - After obtaining the certificate, it's required to save it >> in >> > >> > Custos >> > >> > > in >> > >> > > > a given path. Do we already have an endpoint to achieve >> this? >> > >> > > > >> > >> > > >> > >> > > Yes, Custos has a secret management service to save certificates. >> > >> > > >> > >> > > >> > >> > > > - Client applications can query the certificates in Custos. >> How >> > >> can >> > >> > we >> > >> > > > provide the clients the information about the path to query? >> > >> > > > >> > >> > > >> > >> > > >> > >> > > Client applications can use Custos service accounts to access >> Custos >> > >> > APIs, >> > >> > > >> > >> > > >> > >> > > > >> > >> > > > Please have a look at the above and let me know your opinion. >> > >> > > > >> > >> > > > Regards >> > >> > > > >> > >> > > > [1]. >> > >> > > > >> > >> > > > >> > >> > > >> > >> > >> > >> >> > >> https://cwiki.apache.org/confluence/display/CUSTOS/Custos+Architecture+and+Internals >> > >> > > > [2]. https://youtu.be/CuBvFj194Kg >> > >> > > > [3]. https://github.com/apache/airavata-custos/issues/152 >> > >> > > > [4]. https://github.com/shred/acme4j >> > >> > > > >> > >> > > >> > >> > > >> > >> > > -- >> > >> > > Research Software Engineer >> > >> > > Indiana University, IN >> > >> > > >> > >> > >> > >> >> > >> >> > >> -- >> > >> Research Software Engineer >> > >> Indiana University, IN >> > >> >> > > >> > >> >> >> -- >> Research Software Engineer >> Indiana University, IN >> >
