Hi Isuru, What are the use cases for asynchronous communication?
Thanks Dimuthu On Wed, Nov 27, 2019 at 3:39 PM Isuru Ranawaka <[email protected]> wrote: > Hi all, > > Just to give an update on the project, > > As per the discussion had with Suresh, Custos has synchronous communication > as well as asynchronous communication. For synchronous, there is one to one > mapping between integration service endpoint to core service endpoint. For > asynchronous communication there are many services communicate > asynchronously and get work done. > > I have modeled and implemented asynchronous communication as a service > chain and each core service is invoked via service task and integration > service builds a service task pipeline which is modeled as service chain. > Responses are communicated back via callbacks. > > Moreover, gRPC zipkin interceptors for tracing and Prometheus related > configurations for metrics are added. Docker images for each service is > built and published using docker maven plugin and helm charts are packaged > using helm maven plugin. > > thanks > > Isuru > > > On Wed, Nov 20, 2019 at 3:12 PM Isuru Ranawaka <[email protected]> wrote: > > > Hi all, > > > > I have implemented a basic prototype [1] according to the architecture > > mentioned above. > > > > custos > > | > > |_ custos-core-services > > | | > > | | _ tenant-profile-core-service > > | > > |_ java > > |_proto > > |_ resources > > | > > |_custos-core-services-client-stubs > > | > > |_ tenant-profile-core-service-client-stub > > | > > |_custos-integration-services > > | > > | _ tenant-registration-service > > > > > > Basically core services are gRPC services and their stubs and clients are > > generated in client-stubs module. Integration services use those stubs to > > talk to core services. This is currently working locally and as next > step, > > need to develop docker images, k8s artifacts and run on K8s cluster to > > check whether integration services resolves core services addresses > > seamlessly. > > > > [1]https://github.com/apache/airavata-custos/pull/14 > > > > thanks > > Isuru > > > > > > > > On Mon, Nov 18, 2019 at 9:49 PM Isuru Ranawaka <[email protected]> > wrote: > > > >> Hi Suresh, > >> > >> Thanks for your feedback. I have updated the doc [1]. For observability > >> purposes we can use spring-cloud-sleuth and servers such as Zipkin and > >> those are capable of distributed tracing. For vault operations we have > >> separate audit feature. > >> > >> [1] > >> > https://docs.google.com/document/d/1QtUKTfAnIrErvtYnKo3FizCRL2GUiWikEzvBnHPXfWw/edit?usp=sharing > >> > >> thanks > >> > >> On Mon, Nov 18, 2019 at 6:47 PM Suresh Marru <[email protected]> wrote: > >> > >>> Hi Isuru, > >>> > >>> Looks like your embedded architecture diagram did not make it with the > >>> email. I will request a wiki space for Custos so you can upload there > for > >>> future discussions. For now, can you add it to the google doc? > >>> > >>> Your overall architecture proposal seems to be good. Can you also > >>> consider adding logging and detailed instrumentation as part of the > core > >>> architecture? This will help to have a detailed audit of each and > every > >>> action on who did what and when. > >>> > >>> Cheers, > >>> Suresh > >>> > >>> > On Nov 18, 2019, at 3:41 PM, Isuru Ranawaka <[email protected]> > >>> wrote: > >>> > > >>> > Hi all, > >>> > > >>> > I am initiating this thread to provide Custos initial version > >>> > implementation details. Since we are moving into cloud native > >>> development > >>> > of Custos. I have come up with the following structure for micro > >>> services > >>> > based development of Custos. > >>> > > >>> > Mainly codebase composed of the atomic services and integration > >>> services. > >>> > > >>> > Atomic services are independent services with minimal network > >>> communication > >>> > and they have their own database with local data. > >>> > > >>> > Integration services represents the actual business use case and it > >>> acts as > >>> > an orchestrator between core services and external services. This is > >>> > exposed as a rest api via API Gateway. Integration services talk to > >>> atomic > >>> > services via gRPC protocol. Integration services includes gRPC stub > of > >>> > core services. Each integration service and core service is deployed > to > >>> > Kubernetes cluster as services and have their own service addresses. > >>> > > >>> > Main problem with this architecture is transaction management and it > >>> is > >>> > planned to handle with Saga using Axon Server or Camunda like > >>> framework. > >>> > All the services are written using Spring Boot and Spring Cloud. > >>> > > >>> > All the Integration services are registered with the service registry > >>> such > >>> > as Consul. API Gateway is used for API Management capabilities such > as > >>> SSL > >>> > Termination, throttling..etc. There are Open Source API Gateways such > >>> as > >>> > Ambassador, Envoy which are compatible with Kubernetes and service > >>> > registries. API Gateway is responsible for service discovery. > >>> > > >>> > Vault and Keycloak are deployed in Kubernetes with HA[1]. > >>> > > >>> > > >>> > Let me know your thoughts. > >>> > > >>> > > >>> > > >>> > [1] > >>> > > >>> > https://docs.google.com/document/d/1zGXlstt9w1ge8oRWjiUv3LdQETcSYtwDngm9grB5sbk/edit?usp=sharing > >>> > > >>> > > >>> > Thanks > >>> > > >>> > Isuru > >>> > >>> > >> > >> -- > >> Senior Engineer > >> WSO2 Lanka (pvt) Ltd > >> Blog:- http://isurur.blogspot.com/ > >> Phone:- +94714629880 > >> > > > > > > -- > > Senior Engineer > > WSO2 Lanka (pvt) Ltd > > Blog:- http://isurur.blogspot.com/ > > Phone:- +94714629880 > > > > > -- > Senior Engineer > WSO2 Lanka (pvt) Ltd > Blog:- http://isurur.blogspot.com/ > Phone:- +94714629880 >
