Hello Aarushi, Nice analysis.
To answer your specific question: i.e., whether to make API calls synchronous or asynchronous in future Custos: Custos is going to provide a client API to manage to "resource sharing" (as per our last discussion). Therefore, it is important to know whether a particular operation got successful or not. It seems most of the operations in SharingRegistryService are database operations and my perception is these operations are not going to consume much time. Therefore, I believe there is no harm in making those API operations synchronous (I prefer thrift API calls). Synchronous communication enables to handle errors easily also. In synchronous API invocations, we can just throw an exception. However, in asynchronous communication error handling becomes more manual. It makes sense to make an API call asynchronous if it consumes more time (e.g., works on a bulk data and execute a number of queries, etc.) My guess is we do not have such calls in SharingRegistry service. If there is such call, I like your option 3 (Expose Thrift API call which internally uses RabbitMQ for asynchronous communication), but internal queue does not need to be a RabbitMQ, you can use plain Java concurrent queue. Further, make sure to return some form an identifier, so that caller of the API can query the status of the operation later by invoking another API method. Hope this helps. Thanks -Thejaka On Tue, Jun 11, 2019 at 12:13 PM Suresh Marru <[email protected]> wrote: > Hi Aarushi, > > For keeping the user profiles coherent across Airavata and Custos (and > other gateways in future), you may want to also look at SCIM - > http://www.simplecloud.info/ <http://www.simplecloud.info/> > > Suresh > > > On Jun 11, 2019, at 11:43 AM, Pierce, Marlon <[email protected]> wrote: > > > > To partially answer my own question, some of the thought behind the use > of RabbitMQ is captured by a class project described here: > https://github.com/airavata-courses/spring17-microservice-data-management/wiki. > We can also dig up the Airavata dev list postings by Gourav Shenoy and > Ajinkya Dhamnaskar about the Airavata implementation. > > > > Marlon > > > > > > On 6/10/19, 12:47 PM, "Pierce, Marlon" <[email protected]> wrote: > > > > Hi Aarushi, > > > > Thanks for the nice problem description. It may also be helpful to > understand why the sharing and profile services communicate over RabbitMQ > while the others are direct Thrift calls. > > > > Marlon > > > > > > On 6/10/19, 10:26 AM, "Bisht, Aarushi" <[email protected]> wrote: > > > > This message was sent from a non-IU address. Please exercise > caution when clicking links or opening attachments from external sources. > > ------- > > > > ?Hi All, > > > > > > For those of you who don't know me, I am a GA in IU Bloomington, > working on the Custos project. Since the past week I have been working on > migrating Sharing Service as a standalone Custos service. > > > > > > I have been trying to remove all dependencies of airavata from > sharing registry service. So far I am able to start it as an independent > service but there are few code dependencies on airavata's utility modules. > > > > > > On investigating these dependencies I found out few things and > would like to get advice on some decisions. > > > > > > The sharing registry service is being used by the following > services in airavata: > > > > 1. Group Manager service > > > > 2. Services-security > > > > 3. Airavata-api-Server > > > > 4.Tenant Profile service > > > > 5. User Profile service > > > > 5. Registry server > > > > > > All the above services are communicating with the sharing service > through a thrift client except for user/tenant profile service and registry > service. > > > > > > The communication between profile service and sharing service is > done asynchronously through RabbitMQ . When a new user or tenant is > registered, a message is added to the PROFILE / TENANT queue which is then > consumed by the sharing service and corresponding database entries are done > in sharing database. > > > > > > Similarly registry service consumes the message from PROFILE > queue and sends a message to PROJECT queue to create a default project for > the new user account. This message is consumed by the sharing registry > service. > > > > > > I want to seek advice on how the communication with the > standalone sharing registry service should be handled in future so that it > aligns with custos project goals. > > > > > > There are couple of approaches that I can think of: > > > > 1. Thrift API calls > > > > Any communication with sharing service will be done only through > a thrift client. This will involve making changes to registry and profile > services. This makes the call synchronous, and if required, the consuming > services will need to handle asynchronous calls and error handling. > > > > > > 2. Using RabbitMQ > > > > ?The advantage of using this approach is that communication will > be asynchronous but the services consuming sharing service will have to > manage RabbitMQ connection. > > > > > > 3. Expose Thrift API call which internally use RabbitMQ for > asynchronous communication > > > > This will involve adding API call in sharing service which can be > used by other services to add a message to the sharing service queue. The > advantage of this approach is that other services consuming sharing service > will not have to manage RabbitMQ connection and the communication will > still be asynchronous (other than the call to push request in RabbitMQ) > > > > > > The decision mainly boils down to choosing between synchronous > and asynchronous approach. Asynchronous will mean faster response time for > APIs, with each service responsible for handling and resolving its own > errors. But this also means that there might be few seconds delay in > processing of requests as these are asynchronous. On other hand, > synchronous approach will ensure that all requests are processed > immediately, which increases the API response time but also makes it tricky > to handle failure edge cases as if any request fails, the corresponding DB > entries needs to be rolled back which may include another thrift call (if > entries were made via thrift). If rollback fails due to some reason, there > is no easy way to ensure data consistency, though this is a edge case > scenario. > > > > > > Thanks & Regards, > > > > Aarushi Bisht > > > > > > > > > > -- Best Regards, Thejaka Amila Kanewala, PhD https://github.com/thejkane/agm
