Hi All, I have submitted a PR [2] with initial framework for credential store and its API. Credential backend it using a Hashicorp Vault [3] store and API is implemented as a REST API. Application is implemented as a Spring Boot application with dependency injection for much better flexibility among components.
[2] https://github.com/apache/airavata-custos/pull/3 [3] https://www.vaultproject.io/ Thanks Dimuthu On Tue, Jul 9, 2019 at 3:37 PM Thejaka Amila J Kanewala < [email protected]> wrote: > I am indifferent about the choice. Honestly, i dont have much > recent experience with either, though i see lot of people use REST. So > chose whichever you are comfortable with. > > Thejaka > > On Mon, Jul 8, 2019, 8:11 PM Suresh Marru <[email protected]> wrote: > > > Hi Dimuthu, > > > > I am + 0 if you choose to go with REST. > > > > For Airavata the data models are complex and gateways use cases were very > > diverse which made thrift a good choice. For Custos in general and the > > credential store API in particular, I agree REST might be a good choice. > > > > Suresh > > > > > On Jul 8, 2019, at 12:52 PM, DImuthu Upeksha < > [email protected]> > > wrote: > > > > > > Hi Amila, Aarushi, All, > > > > > > Thanks for the clarification. I'm working on revamping current > credential > > > store to support multiple credential types with the Vault [1] as the > > > backend. I will start a new thread to discuss more on that with a PR. > > > However do we have an idea on how our external APIs should be? Would > that > > > be Thrift based or do we have any other reasons to look for an > > alternative > > > approach. I personally +0 for Thrift and +1 for a REST based approach > > > because users can easily understand the usage by looking at the > > > documentation. What do you think? > > > > > > [1] https://www.vaultproject.io/ > > > > > > Thanks > > > Dimuthu > > > > > > On Wed, Jun 26, 2019 at 3:27 AM Thejaka Amila J Kanewala < > > > [email protected]> wrote: > > > > > >> Hi Aarushi, > > >> > > >> Well thought questions. > > >> Some answers/comments inline. > > >> > > >> Thanks > > >> -Thejaka > > >> > > >> On Tue, Jun 25, 2019 at 6:38 AM Bisht, Aarushi <[email protected]> wrote: > > >> > > >>> Hi Thejaka, > > >>> > > >>> Thank you for the project structure. Besides the above-mentioned > > modules, > > >>> I have also added > > >>> > > >>> 1. custos-commons - for common utility methods used across custos > > >> services > > >>> 2. custos-profile-service - which has profile service APIs, handlers > > and > > >>> other utility function for user profile service, tenant profile > service > > >> and > > >>> iamadminservice > > >>> 3. thrift-interfaces - thrift files for all models and custos APIs > > >>> > > >>> Please let me know if any reorganization is required for the above > > >> modules. > > >>> > > >> > > >> This is good. We need these. > > >> > > >> > > >>> > > >>> I also have a couple of doubts related to some of the points > mentioned > > in > > >>> the mail. > > >>> > > >>> 1. As you mentioned that Custos will provide multi-tenancy > > >>> functionalities. If I understand correctly, this means multiple > tenants > > >> can > > >>> be initialized in Custos using Tenant Profile Service(which is > > currently > > >>> present in Airavata). Since I am migrating this service to custos, I > > want > > >>> to understand what does it mean for Airavata. Since Airavata will use > > >>> custos profile services, does this mean all tenant operation to > > Airavata > > >>> will be delegated to Custos? > > >>> > > >> > > >> Yes (Assuming "tenant operation" is referring to add/delete/modify > > tenants, > > >> add users to tenants, etc.). Airavata should only work with a > reference > > to > > >> the tenant (e.g., tenant id). However, we need to make sure tenant > > >> isolation is properly implemented in Airavata (i.e., users in tenant1 > > >> should not be able to access any resource belong to a user in tenant2 > > >> (unless if they are shared)). > > >> > > >> > > >>> > > >>> Also, I am a little confused with the terminology. Airavata is a > > gateway, > > >>> what does allowing multi-tenants in airavata mean? When "public > Gateway > > >>> addGateway(AuthzToken authzToken, Gateway gateway)" of > > >> TenantProfileService > > >>> is invoked, does gateway here refers to tenants like PGA, Seagrid? > > >>> > > >> > > >> This is a very good question and I believe Marlon already answered > > this. As > > >> Marlon described, Airavata is a gateway middleware. Scientists can use > > >> Airavata to build gateways. Each tenant is a separate gateway. > > >> > > >> "does gateway here refer to tenants like PGA, Seagrid?" -- correct. > > >> > > >> > > >>> 2. There are modules related to authentication and authorization in > > >>> Custos. I can understand the new authenticate flow but I am not very > > >> clear > > >>> about authorization. When a user invokes an API call, its > > authentication > > >>> will be delegated by Airavata to Custos which will authenticate the > > user > > >>> using the authztoken with Keycloak and return the result to Airavata. > > >>> > > >>> In case of authorization, every Airavata API call is authorized based > > on > > >>> the group user belongs to, which is either AdminGroup, > > >> ReadsOnlyAdminGroup > > >>> or a gateway user. The user group information and what groups are > > >>> authorized for which API call is kept with Airavata. I do not > > understand > > >>> how will authorization in Custos work? > > >>> > > >> > > >> Good question and I am not 100% sure whether I know the answer for > > this. I > > >> have not touched Airavata code base for a while and I don't clearly > > >> understand why we maintain groups within Airavata (why not in Keycloak > > >> groups and roles?). Maybe Suresh/Dimuthu or Marlon can explain why we > > >> maintain groups in Airavata but not in Keycloak. > > >> > > >> The way I envision the authorization functionality should be > > implemented is > > >> as follows: > > >> > > >> The authorization is performed on a particular resource. A resource > can > > be > > >> represented using a unique string (sometimes called "permission > string") > > >> and this string can be associated with an API function. Sometimes > these > > >> resources hierarchical (e.g., a directory structure) in such cases > those > > >> string can also be hierarchical ("api/addUser" etc.). Then, there is > an > > >> action that can be performed on a resource (e.g., "read/write/invoke" > > >> etc.). In an API call, we need to make sure the current user has > > "invoke" > > >> permission on the resource associated with the API. So, in summary, > > Custos > > >> need to provide an API like following for authorization: > > >> > > >> "bool isAuthorize(tenantId, userId, permissionString, action)" > > >> > > >> Airavata should invoke above Custos API upon invoking the Airavata > API. > > The > > >> underlying authorization method is irrelevant to Airavata (i.e., > whether > > >> Custos is using RBAC or fine-grained authorization etc.) and possibly > > above > > >> method should inspect all inheriting permissions through groups, roles > > and > > >> sharing as well. > > >> > > >> > > >>> Also, I am currently following the first approach to integrate > Keycloak > > >>> with Custos ie using keycloak as an external service as Airavata was > > >> using > > >>> this approach. Let me know if this needs to be changed. > > >>> > > >> > > >> +1, let's use this approach. > > >> > > >> > > >>> > > >>> Thanks & Regards, > > >>> Aarushi Bisht > > >>> ________________________________________ > > >>> From: Thejaka Amila J Kanewala <[email protected]> > > >>> Sent: Monday, June 24, 2019 1:07 PM > > >>> To: [email protected] > > >>> Subject: Re: Module structure > > >>> > > >>> Hi Dimuthu, > > >>> > > >>> I am sorry for the late reply. I thought I play around this first and > > >> reply > > >>> to your email but I got busy during the weekend and did not get a > > chance > > >> to > > >>> experiment around this. Therefore, I do not want to hold you on this > > >>> anymore. > > >>> > > >>> Here is what I had in mind: > > >>> > > >>> Custos provides the following functionalities: > > >>> > > >>> 1. Delegate science gateway authentication > > >>> 2. Provide transparent resource authentication to science gateways > > >>> 3. Provide sharing/multi-tenancy functionalities (what is already in > > >>> Airavata) > > >>> > > >>> To provide some of these functionalities we need an identity server > > like > > >>> Keycloak. So there are two ways we can integrate Keycloak with > Custos: > > 1. > > >>> As an external service, 2. As an embedded server running inside > Custos > > >>> server. > > >>> We had different opinions about these two approaches, however, I > > favored > > >>> the second approach as it simplifies the deployment complexity (and I > > >> guess > > >>> Suresh favored the first approach). The issue with the second > approach > > is > > >>> there is no straightforward way to run Keycloak inside Tomcat as an > > >>> embedded server. There are a couple of blog posts saying that it is > > >>> possible however, I did not get enough time to successfully run them > > >>> (theoretically this is possible). > > >>> > > >>> So, back to your question about the structure. Assuming we take the > > first > > >>> approach (deploy Keycloak as an external server), I would imagine the > > >>> directory structure would be something similar to the following: > > >>> > > >>> modules > > >>> - custos-connectors (Connectors to connect to Keycloak or any other > > >>> identity server) > > >>> - custos-authentication (Delegated authentication) > > >>> - custos-authorisation > > >>> - custos-resource-authentication (Resource authentication e.g., > Amazon, > > >>> Google clouds, SSH etc) > > >>> - custos-sharing (Airavata sharing) > > >>> - custos-credential-store > > >>> - custos-dist (Distribution + tomcat server) > > >>> - custos-client (Custos client library which will be used by science > > >>> gateways) > > >>> - custos-test (Integration tests) > > >>> > > >>> If we use the second approach, in addition to above modules, we will > > >> have a > > >>> module called "custos-server" which includes changes to embed > keycloak > > >>> within Tomcat. > > >>> > > >>> In either case, the above-described structure is not static, it's > just > > a > > >>> starting point. We definitely need to re-organize as we develop > > >>> functionality. > > >>> > > >>> Thanks > > >>> Best Regards, > > >>> Thejaka Amila Kanewala, PhD > > >>> https://github.com/thejkane/agm > > >>> > > >>> On Sun, Jun 16, 2019 at 9:17 PM DImuthu Upeksha < > > >>> [email protected]> > > >>> wrote: > > >>> > > >>>> Hi Folks, > > >>>> > > >>>> Is there a decided module structure for the project? If you have any > > >>>> suggestion, please let me know. I'm working on refactoring > credential > > >>> store > > >>>> to support multiple protocols (SSH, S3, Azure, Box and etc) and I > > >> believe > > >>>> that this might me useful for Custos as well. > > >>>> > > >>>> Thanks > > >>>> Dimuthu > > >>>> > > >>> > > >>> > > >>> -- > > >>> > > >> > > >> > > >> -- > > >> Best Regards, > > >> Thejaka Amila Kanewala, PhD > > >> https://github.com/thejkane/agm > > >> > > > > >
