Hi All, Thank you for all the inputs. I agree with Thejaka, most of the operations in SharingRegistryService are database operations and will not take a significant amount of time. But as per my understanding, the scenario becomes a little tricky in case of database updates. In the current architecture, multiple databases of different airavata services need to be updated on user action. In this case, there could be multiple points of failures. For example, when a new user is created, all the user info is stored in user profile database and some metadata of the user is also required to be stored in sharing registry database and registry database. Now if this is done through thrift calls synchronously and let's say update to the user database and registry database succeeds but update to sharing database fails. In this case, 2 rollbacks have to be performed which will again require thrift calls. Again if one of the rollbacks fails it could lead to an inconsistent state.
I still believe that using rabbitMQ internally (in case option 3 in my previous mail is used) could be a better solution instead of using java concurrent queues. The reason being java concurrent queues will be in-memory and restart to the server will result in loss of queued data. Also, rabbitMQ gives easy handling of failures for example requeuing of messages which are not successfully acknowledged by the subscriber. Marlon, thank you for sharing the course project. I looked into the investigation done earlier to decide on the approach to manage data sharing across different services in Airavata. I will briefly summarize the report The report explores 2 possible solutions and their pros and cons. 1. 2-Phase commits 2. Event-driven approach In 2-Phase commits, there is a coordinator that maintains the integrity of the global database, ie the collection of databases participating in the transaction. It ensures that all the commits in different databases are either successful or are rolled back. This maintains consistency over the databases but availability is compromised as the coordinator blocks until all the commits are performed. The disadvantage is that it has a single point of failure, ie if the coordinator is down no commits will be performed. The event-driven approach is the current implemented solution. It ensures eventual consistency. The service will be highly available at the cost of immediate data consistency which will "eventually" be achieved. I understand the motivation of this decision at the time when all these services were within airavata. Now when sharing and profile services will be migrated to Custos and gateways other than Airavata will be interacting with Custos, the decisions can be reconsidered. Reconsidering the 3 options I proposed in my last mail. 1. Thrift API calls: If a gateway like Airavata or Galaxy is using only the standalone Custos Sharing service, the interaction could be through thrift API calls and the gateway will be responsible to handle the failures. 2. Using RabbitMQ: Enforcing a gateway to use a pub/sub model like RabbitMQ and publishing messages to queues/ subscribing to queues is not desirable and should be avoided. 3. Expose Thrift API call which internally uses RabbitMQ for asynchronous communication: Exposing APIs which internally handle a queue could be a possible solution if the gateways do not require immediate confirmation that operation was successful or not, this again boils down to consistency vs availability in CAP. With option 2 being eliminated due to the reason described above. The decision is between synchronous(option 1) vs asynchronous(option 3). I would prefer option3 if immediate database consistency is not required. But if the gateway needs to know the success or failure of transaction immediately we can go with the synchronous approach. This is my understanding, please correct me if I am wrong or missing something. Thanks & Regards, Aarushi Bisht ________________________________________ From: Thejaka Amila J Kanewala <[email protected]> Sent: Wednesday, June 12, 2019 7:04 PM To: [email protected] Subject: Re: [External] Migration of sharing registry service Hello Aarushi, Nice analysis. To answer your specific question: i.e., whether to make API calls synchronous or asynchronous in future Custos: Custos is going to provide a client API to manage to "resource sharing" (as per our last discussion). Therefore, it is important to know whether a particular operation got successful or not. It seems most of the operations in SharingRegistryService are database operations and my perception is these operations are not going to consume much time. Therefore, I believe there is no harm in making those API operations synchronous (I prefer thrift API calls). Synchronous communication enables to handle errors easily also. In synchronous API invocations, we can just throw an exception. However, in asynchronous communication error handling becomes more manual. It makes sense to make an API call asynchronous if it consumes more time (e.g., works on a bulk data and execute a number of queries, etc.) My guess is we do not have such calls in SharingRegistry service. If there is such call, I like your option 3 (Expose Thrift API call which internally uses RabbitMQ for asynchronous communication), but internal queue does not need to be a RabbitMQ, you can use plain Java concurrent queue. Further, make sure to return some form an identifier, so that caller of the API can query the status of the operation later by invoking another API method. Hope this helps. Thanks -Thejaka On Tue, Jun 11, 2019 at 12:13 PM Suresh Marru <[email protected]> wrote: > Hi Aarushi, > > For keeping the user profiles coherent across Airavata and Custos (and > other gateways in future), you may want to also look at SCIM - > http://www.simplecloud.info/ <http://www.simplecloud.info/> > > Suresh > > > On Jun 11, 2019, at 11:43 AM, Pierce, Marlon <[email protected]> wrote: > > > > To partially answer my own question, some of the thought behind the use > of RabbitMQ is captured by a class project described here: > https://github.com/airavata-courses/spring17-microservice-data-management/wiki. > We can also dig up the Airavata dev list postings by Gourav Shenoy and > Ajinkya Dhamnaskar about the Airavata implementation. > > > > Marlon > > > > > > On 6/10/19, 12:47 PM, "Pierce, Marlon" <[email protected]> wrote: > > > > Hi Aarushi, > > > > Thanks for the nice problem description. It may also be helpful to > understand why the sharing and profile services communicate over RabbitMQ > while the others are direct Thrift calls. > > > > Marlon > > > > > > On 6/10/19, 10:26 AM, "Bisht, Aarushi" <[email protected]> wrote: > > > > This message was sent from a non-IU address. Please exercise > caution when clicking links or opening attachments from external sources. > > ------- > > > > ?Hi All, > > > > > > For those of you who don't know me, I am a GA in IU Bloomington, > working on the Custos project. Since the past week I have been working on > migrating Sharing Service as a standalone Custos service. > > > > > > I have been trying to remove all dependencies of airavata from > sharing registry service. So far I am able to start it as an independent > service but there are few code dependencies on airavata's utility modules. > > > > > > On investigating these dependencies I found out few things and > would like to get advice on some decisions. > > > > > > The sharing registry service is being used by the following > services in airavata: > > > > 1. Group Manager service > > > > 2. Services-security > > > > 3. Airavata-api-Server > > > > 4.Tenant Profile service > > > > 5. User Profile service > > > > 5. Registry server > > > > > > All the above services are communicating with the sharing service > through a thrift client except for user/tenant profile service and registry > service. > > > > > > The communication between profile service and sharing service is > done asynchronously through RabbitMQ . When a new user or tenant is > registered, a message is added to the PROFILE / TENANT queue which is then > consumed by the sharing service and corresponding database entries are done > in sharing database. > > > > > > Similarly registry service consumes the message from PROFILE > queue and sends a message to PROJECT queue to create a default project for > the new user account. This message is consumed by the sharing registry > service. > > > > > > I want to seek advice on how the communication with the > standalone sharing registry service should be handled in future so that it > aligns with custos project goals. > > > > > > There are couple of approaches that I can think of: > > > > 1. Thrift API calls > > > > Any communication with sharing service will be done only through > a thrift client. This will involve making changes to registry and profile > services. This makes the call synchronous, and if required, the consuming > services will need to handle asynchronous calls and error handling. > > > > > > 2. Using RabbitMQ > > > > ?The advantage of using this approach is that communication will > be asynchronous but the services consuming sharing service will have to > manage RabbitMQ connection. > > > > > > 3. Expose Thrift API call which internally use RabbitMQ for > asynchronous communication > > > > This will involve adding API call in sharing service which can be > used by other services to add a message to the sharing service queue. The > advantage of this approach is that other services consuming sharing service > will not have to manage RabbitMQ connection and the communication will > still be asynchronous (other than the call to push request in RabbitMQ) > > > > > > The decision mainly boils down to choosing between synchronous > and asynchronous approach. Asynchronous will mean faster response time for > APIs, with each service responsible for handling and resolving its own > errors. But this also means that there might be few seconds delay in > processing of requests as these are asynchronous. On other hand, > synchronous approach will ensure that all requests are processed > immediately, which increases the API response time but also makes it tricky > to handle failure edge cases as if any request fails, the corresponding DB > entries needs to be rolled back which may include another thrift call (if > entries were made via thrift). If rollback fails due to some reason, there > is no easy way to ensure data consistency, though this is a edge case > scenario. > > > > > > Thanks & Regards, > > > > Aarushi Bisht > > > > > > > > > > -- Best Regards, Thejaka Amila Kanewala, PhD https://github.com/thejkane/agm
