Hi hasitha, Are you still contributing to this feature? Do you have any update?
On Mon, Jun 21, 2021 at 9:15 AM Hasitha Jayasundara < [email protected]> wrote: > Hi Isuru, > > This is working fine now. I'm not getting any 403 errors. > > Regards > > On Mon, Jun 21, 2021 at 1:21 AM Isuru Ranawaka <[email protected]> wrote: > > > Hi Hasitha, > > > > Is this issue still exist? > > > > On Sat, Jun 19, 2021 at 11:14 PM Hasitha Jayasundara < > > [email protected]> wrote: > > > > > Hi Isuru, > > > > > > I was testing some features with the client sdk and with the recent > > changes > > > in the develop branch I'm getting a 403 when adding kv credentials. Any > > > idea why this is happening? > > > > > > Regards > > > > > > On Wed, May 19, 2021 at 7:33 AM Hasitha Jayasundara < > > > [email protected]> wrote: > > > > > > > Hi Isuru, > > > > > > > > Thanks for the reply. Based on your comment I have gone through the > > > > java-sdk for for resource-secret-management-client first as this will > > be > > > > used in the cert-renewal feature. Several implementations are missing > > > > there, including get/create of certificate credentials. I created an > > > issue > > > > [1] to track the progress of adding missing implementations. WIll > send > > a > > > PR > > > > as soon as possible. > > > > > > > > Regards > > > > > > > > [1]. https://github.com/apache/airavata-custos/issues/164 > > > > <https://github.com/apache/airavata-custos/issues/164> > > > > > > > > On Tue, May 18, 2021 at 9:21 PM Isuru Ranawaka <[email protected]> > > > wrote: > > > > > > > >> Hi Hasitha, > > > >> > > > >> Great progress. You can start Custos integration using > > > >> custos-java-sdk[1]. > > > >> There could be some unimplemented methods and it would be great if > you > > > can > > > >> implement and submit PRs. > > > >> > > > >> thanks > > > >> Isuru > > > >> > > > >> > > > >> [1] > > > >> > > > >> > > > > > > https://github.com/apache/airavata-custos/tree/develop/custos-client-sdks/custos-java-sdk > > > >> > > > >> > > > >> On Mon, May 17, 2021 at 11:59 PM Hasitha Jayasundara < > > > >> [email protected]> wrote: > > > >> > > > >> > Hi Isuru, > > > >> > > > > >> > I completed the cert renewal part of the feature and the > > > implementation > > > >> can > > > >> > be found at [1]. > > > >> > > > > >> > At the moment I'm doing some refactoring according to acme > protocol > > > best > > > >> > practices. After that I'm hoping to start working on custos > > > integration > > > >> > part of the feature using Java SDK. Basically the idea is to store > > the > > > >> > updated certificates in Custos. > > > >> > > > > >> > I am using pebble [2] (a lightweight version of letsencrypt > boulder > > CA > > > >> > server) as the CA server for testing purposes. The main reason for > > > using > > > >> > Pebble is that the Lets Encrypt CA server endpoint (staging) is > not > > > >> > working as expected in the dev environment for a fake domain. > > > >> > > > > >> > As for the current implementation I'm using a nginx server with > Lua > > > >> module > > > >> > as acme challenge server to store files required for completing > the > > > >> > challenge. Lua module is used to embed Lua programming language > into > > > >> nginx > > > >> > server and create files required for acme challenge. (CGI with > Nginx > > > and > > > >> > SpringBoot are two other options for this and I'm thinking of do a > > > >> proper > > > >> > poc/rfc after initial feature implementation is completed) > > > >> > > > > >> > Regards > > > >> > > > > >> > [1]. https://github.com/hasithajayasundara/cert-updater > > > >> > [2]. https://github.com/letsencrypt/pebble > > > >> > > > > >> > On Fri, May 14, 2021 at 2:41 PM Hasitha Jayasundara < > > > >> > [email protected]> wrote: > > > >> > > > > >> > > Hi Isuru, > > > >> > > > > > >> > > Thanks for the explanation. I think I have a clear idea on how > to > > > >> proceed > > > >> > > with the Let's Encrypt part of the implementation now. > > > >> > > > > > >> > > I already started the implementation and am currently designing > a > > > >> > solution > > > >> > > for handling the Let's Encrypt challenge. Once that is done I am > > > >> thinking > > > >> > > of starting the certificates store process in Custos. I think > I'll > > > >> > probably > > > >> > > need some setup completed first for making authenticated > requests > > to > > > >> > Custos. > > > >> > > > > > >> > > Found an issue with the certificate store path in the current > > > >> > > implementation and I've sent a fix for that. [1] > > > >> > > > > > >> > > Regards > > > >> > > > > > >> > > [1]. https://github.com/apache/airavata-custos/pull/160 > > > >> > > > > > >> > > On Thu, May 13, 2021 at 9:31 AM Isuru Ranawaka < > > [email protected]> > > > >> > wrote: > > > >> > > > > > >> > >> Hi Hasitha, > > > >> > >> > > > >> > >> Glad to hear that you have gone through the code. Please feel > > free > > > to > > > >> > >> submit bug fixes as you see. > > > >> > >> > > > >> > >> We use HarshiCorp vault to store secrets. The paths you > mentioned > > > are > > > >> > >> actually created in the vault. Precisely, owner_id is the id of > > the > > > >> user > > > >> > >> or > > > >> > >> the service account that owns the certificate. Token is an > > > identifier > > > >> > that > > > >> > >> can be used to retrieve the certificate later. > > > >> > >> > > > >> > >> thanks > > > >> > >> Isuru > > > >> > >> > > > >> > >> On Wed, May 12, 2021 at 11:08 PM Hasitha Jayasundara < > > > >> > >> [email protected]> wrote: > > > >> > >> > > > >> > >> > Hi Isuru, > > > >> > >> > > > > >> > >> > Thanks for the reply. Based on your comment that custos > already > > > >> has an > > > >> > >> > endpoint to store certificates I took some time to go through > > the > > > >> > Secret > > > >> > >> > Management Service and the proto for Resource Secret Service. > > > >> > >> > > > > >> > >> > According to my observation, I think I can use *POST - > > > >> > >> > /resource-secret-management/v1.0.0/secret/certificate* > endpoint > > > to > > > >> do > > > >> > >> > certificate uploading. (Correct me if I'm wrong) > > > >> > >> > > > > >> > >> > Going through the code I found that certificate is stored in > > > >> > >> > */resourcesecret/<tenant_id>/<owner_id>/ssh/<token>*. The > > > >> <tenant_id>, > > > >> > >> > <owner_id> and <token> are received through the request > body. I > > > am > > > >> not > > > >> > >> 100% > > > >> > >> > clear about <owner_id> and <token> fields in the context of > > > custos > > > >> > >> secret > > > >> > >> > manager service. Could you please explain a bit about > > <owner_id> > > > >> and > > > >> > the > > > >> > >> > <token>? > > > >> > >> > > > > >> > >> > Regards > > > >> > >> > > > > >> > >> > On Wed, May 12, 2021 at 7:30 AM Isuru Ranawaka < > > > [email protected] > > > >> > > > > >> > >> wrote: > > > >> > >> > > > > >> > >> > > Hi Hasitha, > > > >> > >> > > > > > >> > >> > > We really appreciate your interest in Custos project. > Please > > > find > > > >> > the > > > >> > >> > > inline comments. > > > >> > >> > > > > > >> > >> > > On Tue, May 11, 2021 at 9:33 AM Hasitha Jayasundara < > > > >> > >> > > [email protected]> wrote: > > > >> > >> > > > > > >> > >> > > > Hi all, > > > >> > >> > > > > > > >> > >> > > > I'm Hasitha and I'm currently working as a Full Stack > > > Engineer > > > >> for > > > >> > >> a UK > > > >> > >> > > > based tech company. I graduated from University Of > Moratuwa > > > >> > >> Department > > > >> > >> > of > > > >> > >> > > > Electronic and Telecommunication Engineering Sri Lanka > and > > > >> > >> completed a > > > >> > >> > > > software engineering internship at a company called WSO2 > > > which > > > >> is > > > >> > an > > > >> > >> > open > > > >> > >> > > > source technology provider based in Sri Lanka. I also > > > >> participated > > > >> > >> in > > > >> > >> > > > Google Summer of Code 2017 and successfully completed a > > > project > > > >> > for > > > >> > >> > > > implementing an inbound endpoint and a connector for > > IBM-MQ. > > > >> > >> > > > > > > >> > >> > > > I'm enthusiastic about the airavata-custos project and > > would > > > be > > > >> > >> > extremely > > > >> > >> > > > interested in contributing to the project. I've already > > > >> followed > > > >> > the > > > >> > >> > wiki > > > >> > >> > > > page[1] and the tutorial presented at the Gateways 2020 > > > >> Conference > > > >> > >> [2] > > > >> > >> > > and > > > >> > >> > > > I think I have a clear understanding about how custos > > > operates > > > >> in > > > >> > >> the > > > >> > >> > > real > > > >> > >> > > > world. > > > >> > >> > > > > > > >> > >> > > > As a starting point I selected the issue #152 [3] which > is > > to > > > >> > >> > implement a > > > >> > >> > > > cert manager task using Custos. The idea is to automate > the > > > >> cert > > > >> > >> > renewal > > > >> > >> > > > process so that the client's won't need to manually do it > > by > > > >> going > > > >> > >> > > through > > > >> > >> > > > Let's Encrypt cert renewal process. The task (cron job) > > will > > > be > > > >> > >> > > responsible > > > >> > >> > > > for, > > > >> > >> > > > > > > >> > >> > > > - Connecting to Let’s Encrypt APIs and create > > > certificates, > > > >> > >> update > > > >> > >> > > > certificates > > > >> > >> > > > - Saving updated certificates into a given location > and > > > >> where > > > >> > the > > > >> > >> > > server > > > >> > >> > > > application can use it > > > >> > >> > > > > > > >> > >> > > > The overall process can be illustrated as follows. > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > - Custos Cert Renewer task (Cron Job) will be executed > > on > > > >> > >> periodic > > > >> > >> > > > configurable time period > > > >> > >> > > > - Task will communicate with Let's Encrypt using > > > >> ACME(Automated > > > >> > >> > > > Certificate Management Environments) protocol. (I've > > > already > > > >> > >> tested > > > >> > >> > > this > > > >> > >> > > > with acme4j [4] which is a java client for ACME > protocol > > > and > > > >> > it's > > > >> > >> > > > working > > > >> > >> > > > fine without any issues) > > > >> > >> > > > - Task will then obtain the updated certificate and > > store > > > it > > > >> > in a > > > >> > >> > > given > > > >> > >> > > > path in Custos (probably in a persistent volume) > > > >> > >> > > > - Client applications can query certificates from > Custos > > > and > > > >> > add > > > >> > >> > them > > > >> > >> > > to > > > >> > >> > > > client trustore on demand to call a server application > > > >> > >> > > > > > > >> > >> > > > Following I listed out some of the concerns I'm having at > > the > > > >> > >> moment. > > > >> > >> > > > > > > >> > >> > > > - In order to obtain an updated certificates, > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > 1. we need to have an account in CA with a public and > a > > > >> private > > > >> > >> key > > > >> > >> > > > pair. Do we already have an account in Let's Encrypt > for > > > >> this > > > >> > >> type > > > >> > >> > of > > > >> > >> > > > work > > > >> > >> > > > or Is it possible to create a dedicated account for > the > > > cert > > > >> > >> renewer > > > >> > >> > > > task? > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > yes, we can create a test account in Let's Encrypt. > > > >> > >> > > > > > >> > >> > > 2. we need to provide CNs (domain names) for Let's > > Encrypt. > > > >> Where > > > >> > >> do > > > >> > >> > we > > > >> > >> > > > store the domain names of the clients? > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > I guess you are referring to the server application > domains. > > > >> those > > > >> > >> should > > > >> > >> > > be configurable and can be stored in a configuration file. > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > - After obtaining the certificate, it's required to > save > > > it > > > >> in > > > >> > >> > Custos > > > >> > >> > > in > > > >> > >> > > > a given path. Do we already have an endpoint to > achieve > > > >> this? > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > Yes, Custos has a secret management service to save > > > certificates. > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > - Client applications can query the certificates in > > > Custos. > > > >> How > > > >> > >> can > > > >> > >> > we > > > >> > >> > > > provide the clients the information about the path to > > > query? > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > Client applications can use Custos service accounts to > access > > > >> Custos > > > >> > >> > APIs, > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > > >> > >> > > > Please have a look at the above and let me know your > > opinion. > > > >> > >> > > > > > > >> > >> > > > Regards > > > >> > >> > > > > > > >> > >> > > > [1]. > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/CUSTOS/Custos+Architecture+and+Internals > > > >> > >> > > > [2]. https://youtu.be/CuBvFj194Kg > > > >> > >> > > > [3]. > https://github.com/apache/airavata-custos/issues/152 > > > >> > >> > > > [4]. https://github.com/shred/acme4j > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > -- > > > >> > >> > > Research Software Engineer > > > >> > >> > > Indiana University, IN > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > >> > > > >> > >> -- > > > >> > >> Research Software Engineer > > > >> > >> Indiana University, IN > > > >> > >> > > > >> > > > > > >> > > > > >> > > > >> > > > >> -- > > > >> Research Software Engineer > > > >> Indiana University, IN > > > >> > > > > > > > > > > > > > -- > > Research Software Engineer > > Indiana University, IN > > > -- Research Software Engineer Indiana University, IN
