+1 on striving for internal communications to be the same as external - it has advantages like forcing “us” to be clients to our own micro services (dog fooding), helping uncover weaknesses and bugs before external clients do. Also keeps things simpler for both external and internal communications - due to same-ness.
>From Supun’s email, I see… But we find it difficult to use RPC calls and achieve stateless behavior and thinking of replacing Thrift services with ZooKeeper based communication layer. Curious to understand the difficulty at a deeper level. As Eran may have mentioned we have a micro services architecture with services depending on each other in a stateless manner (we’re up to 3 depths of dependencies now). We haven’t run into any difficulties. Hence, my curiosity to understand if you’ve learnt something we haven’t hit yet. Please clarify with some examples. On ZK use, I don’t understand what it means to use ZK as a communication mechanism. As Eran pointed out, I’ve heard of ZK recipes for implementing various distributed consensus and coordination scenarios (Netlfix implemented some of these ZK recipes via http://curator.apache.org ). So, yes, a concrete example would be helpful to understand this option. Thanks, —Jijoe > On Jun 13, 2014, at 9:24 PM, Eran Chinthaka Withana > <[email protected]> wrote: > > > Hi, > > > First, it was very surprising for me to hear that ZK was used as the > communication mechanism. The closest I have heard is recipes: > http://zookeeper.apache.org/doc/trunk/recipes.html. Also, since thrift > services forces us to deploy stateless services the load balancing and > scaling is pretty easy. Its just a matter of bringing up new instances. Can > you please explain one use case where you use zookeeper as the > communication mechanism? > > > Anyway, *Suresh*, why do you want to pick something else for internal > communication? Why can't we use thrift even for that? At some point, you > won't be able to define whats internal and external. May be I'm not > understanding the requirement/usecases here. Can you please elaborate with > an example? > > > Thanks, > Eran Chinthaka Withana > > > > > On Fri, Jun 13, 2014 at 4:43 AM, Suresh Marru <[email protected]> wrote: > > >> Thanks Supun these are great thoughts and is interesting to know how you >> are using zookeeper for state management and communication. Since we are >> diverging the discussion into a different direction (this is good important >> topic though), let me start a new thread and continue to discuss zookeeper >> in other thread. >> >> >> Thrift is serving well (atleast for now, barring some workable >> limitations) for Airavata client facing API. Its helping with the >> complexity in the data model and polygot clients. For justified reasons, we >> are slowly breaking down and expanding the services within Airavata leading >> to a micro-service architectures, may be towards more reactive architecture. >> >> >> What will be the good internal communication mechanism? The architectures >> I have come across have used zookeeper for mainly load balancing, >> distributed configuration management, change notifications and so forth. >> What are the alternatives? >> >> >> Since Eran, Patanachani, Jijoe and Samir have gone through similar >> iteration, it will be great to hear their opinions. >> >> >> Suresh >> >> >> On Jun 12, 2014, at 11:50 PM, Supun Kamburugamuva <[email protected]> >> wrote: >> >> >>> Hi all, >>> >>> >>> Here is what I think about Airavata and ZooKeeper. In Airavata there are >>> >>> >>> many components and these components must be stateless to achieve >> scalability and reliability.Also there must be a mechanism to communicate >> between the components. At the moment Airavata uses RPC calls based on >> Thrift for the communication. >> >> >>> >>> >>> ZooKeeper can be used both as a place to hold state and as a >>> >>> >>> communication layer between the components. I'm involved with a project >> that has many distributed components like AIravata. Right now we use Thrift >> services to communicate among the components. But we find it difficult to >> use RPC calls and achieve stateless behaviour and thinking of replacing >> Thrift services with ZooKeeper based communication layer. So I think it is >> better to explore the possibility of removing the Thrift services between >> the components and use ZooKeeper as a communication mechanism between the >> services. If you do this you will have to move the state to ZooKeeper and >> will automatically achieve the stateless behaviour in the components. >> >> >>> >>> >>> Also I think trying to make ZooKeeper optional is a bad idea. If we are >>> >>> >>> trying to integrate something fundamentally important to architecture as >> how to store state, we shouldn't make it optional. >> >> >>> >>> >>> Thanks, >>> Supun.. >>> >>> >>> >>> >>> On Thu, Jun 12, 2014 at 10:57 PM, Shameera Rathnayaka < >>> >>> >>> [email protected]> wrote: >> >> >>> Hi Lahiru, >>> >>> >>> As i understood, not only reliability , you are trying to achieve some >>> >>> >>> other requirement by introducing zookeeper, like health monitoring of the >> services, categorization with service implementation etc ... . In that >> case, i think we can get use of zookeeper's features but if we only focus >> on reliability, i have little bit of concern, why can't we use clustering + >> LB ? >> >> >>> >>> >>> Yes it is better we add Zookeeper as a prerequisite if user need to use >>> >>> >>> it. >> >> >>> >>> >>> Thanks, >>> Shameera. >>> >>> >>> >>> >>> On Thu, Jun 12, 2014 at 5:19 AM, Lahiru Gunathilake <[email protected]> >>> >>> >>> wrote: >> >> >>> Hi Gagan, >>> >>> >>> I need to start another discussion about it, but I had an offline >>> discussion with Suresh about auto-scaling. I will start another thread >>> about this topic too. >>> >>> >>> Regards >>> Lahiru >>> >>> >>> >>> >>> On Wed, Jun 11, 2014 at 4:10 PM, Gagan Juneja <[email protected] >>> >>> >>> wrote: >>> >>> >>>> Thanks Lahiru for pointing to nice library, added to my dictionary :). >>>> >>>> >>>> I would like to know how are we planning to start multiple servers. >>>> 1. Spawning new servers based on load? Some times we call it as auto >>>> scalable. >>>> 2. To make some specific number of nodes available such as we want 2 >>>> servers to be available at any time so if one goes down then I need to >>>> spawn one new to make available servers count 2. >>>> 3. Initially start all the servers. >>>> >>>> >>>> In scenario 1 and 2 zookeeper does make sense but I don't believe >>>> >>>> >>>> existing >> >> >>>> architecture support this? >>>> >>>> >>>> Regards, >>>> Gagan >>>> On 12-Jun-2014 1:19 am, "Lahiru Gunathilake" <[email protected]> >>>> >>>> >>>> wrote: >> >> >>>> >>>> >>>>> Hi Gagan, >>>>> >>>>> >>>>> Thanks for your response. Please see my inline comments. >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Jun 11, 2014 at 3:37 PM, Gagan Juneja < >>>>> >>>>> >>>>> [email protected]> >> >> >>>>> wrote: >>>>> >>>>> >>>>>> Hi Lahiru, >>>>>> Just my 2 cents. >>>>>> >>>>>> >>>>>> I am big fan of zookeeper but also against adding multiple hops in >>>>>> >>>>>> >>>>>> the >> >> >>>>>> system which can add unnecessary complexity. Here I am not able to >>>>>> understand the requirement of zookeeper may be I am wrong because of >>>>>> >>>>>> >>>>>> less >> >> >>>>>> knowledge of the airavata system in whole. So I would like to discuss >>>>>> following point. >>>>>> >>>>>> >>>>>> 1. How it will help us in making system more reliable. Zookeeper is >>>>>> >>>>>> >>>>>> not >> >> >>>>>> able to restart services. At max it can tell whether service is up >>>>>> >>>>>> >>>>>> or not >> >> >>>>>> which could only be the case if airavata service goes down >>>>>> >>>>>> >>>>>> gracefully and >> >> >>>>>> we have any automated way to restart it. If this is just matter of >>>>>> >>>>>> >>>>>> routing >> >> >>>>>> client requests to the available thrift servers then this can be >>>>>> >>>>>> >>>>>> achieved >> >> >>>>>> with the help of load balancer which I guess is already there in >>>>>> >>>>>> >>>>>> thrift >> >> >>>>>> wish list. >>>>>> >>>>>> >>>>>> We have multiple thrift services and currently we start only one >>>>> >>>>> >>>>> instance >> >> >>>>> of them and each thrift service is a stateless service. To keep the >>>>> >>>>> >>>>> high >> >> >>>>> availability we have to start multiple instances of them in production >>>>> scenario. So for clients to get an available thrift service we can use >>>>> zookeeper znodes to represent each available service. There are some >>>>> libraries which is doing similar[1] and I think we can use them >>>>> >>>>> >>>>> directly. >> >> >>>>> >>>>> >>>>>> 2. As far as registering of different providers is concerned do you >>>>>> think for that we really need external store. >>>>>> >>>>>> >>>>>> Yes I think so, because its light weight and reliable and we have to >>>>> >>>>> >>>>> do >> >> >>>>> very minimal amount of work to achieve all these features to Airavata >>>>> because zookeeper handle all the complexity. >>>>> >>>>> >>>>>> I have seen people using zookeeper more for state management in >>>>>> distributed environments. >>>>>> >>>>>> >>>>>> +1, we might not be the most effective users of zookeeper because all >>>>> >>>>> >>>>> of >> >> >>>>> our services are stateless services, but my point is to achieve >>>>> fault-tolerance we can use zookeeper and with minimal work. >>>>> >>>>> >>>>>> I would like to understand more how can we leverage zookeeper in >>>>>> airavata to make system reliable. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> [1]https://github.com/eirslett/thrift-zookeeper >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Regards, >>>>>> Gagan >>>>>> On 12-Jun-2014 12:33 am, "Marlon Pierce" <[email protected]> wrote: >>>>>> >>>>>> >>>>>>> Thanks for the summary, Lahiru. I'm cc'ing the Architecture list for >>>>>>> additional comments. >>>>>>> >>>>>>> >>>>>>> Marlon >>>>>>> >>>>>>> >>>>>>> On 6/11/14 2:27 PM, Lahiru Gunathilake wrote: >>>>>>> >>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> >>>>>>>> I did little research about Apache Zookeeper[1] and how to use it >>>>>>>> >>>>>>>> >>>>>>>> in >> >> >>>>>>>> airavata. Its really a nice way to achieve fault tolerance and >>>>>>>> >>>>>>>> >>>>>>>> reliable >>>>>>> >>>>>>> >>>>>>>> communication between our thrift services and clients. Zookeeper >>>>>>>> >>>>>>>> >>>>>>>> is a >> >> >>>>>>>> distributed, fault tolerant system to do a reliable communication >>>>>>>> >>>>>>>> >>>>>>>> between >>>>>>> >>>>>>> >>>>>>>> distributed applications. This is like an in-memory file system >>>>>>>> >>>>>>>> >>>>>>>> which >> >> >>>>>>> has >>>>>>> >>>>>>> >>>>>>>> nodes in a tree structure and each node can have small amount of >>>>>>>> >>>>>>>> >>>>>>>> data >> >> >>>>>>>> associated with it and these nodes are called znodes. Clients can >>>>>>>> >>>>>>>> >>>>>>>> connect >>>>>>> >>>>>>> >>>>>>>> to a zookeeper server and add/delete and update these znodes. >>>>>>>> >>>>>>>> >>>>>>>> In Apache Airavata we start multiple thrift services and these >>>>>>>> >>>>>>>> >>>>>>>> can >> >> >>>>>>> go >>>>>>> >>>>>>> >>>>>>>> down for maintenance or these can crash, if we use zookeeper to >>>>>>>> >>>>>>>> >>>>>>>> store >> >> >>>>>>> these >>>>>>> >>>>>>> >>>>>>>> configuration(thrift service configurations) we can achieve a very >>>>>>>> >>>>>>>> >>>>>>>> reliable >>>>>>> >>>>>>> >>>>>>>> system. Basically thrift clients can dynamically discover >>>>>>>> >>>>>>>> >>>>>>>> available >> >> >>>>>>> service >>>>>>> >>>>>>> >>>>>>>> by using ephemeral znodes(Here we do not have to change the >>>>>>>> >>>>>>>> >>>>>>>> generated >> >> >>>>>>>> thrift client code but we have to change the locations we are >>>>>>>> >>>>>>>> >>>>>>>> invoking >> >> >>>>>>>> them). ephemeral znodes will be removed when the thrift service >>>>>>>> >>>>>>>> >>>>>>>> goes >> >> >>>>>>> down >>>>>>> >>>>>>> >>>>>>>> and zookeeper guarantee the atomicity between these operations. >>>>>>>> >>>>>>>> >>>>>>>> With >> >> >>>>>>> this >>>>>>> >>>>>>> >>>>>>>> approach we can have a node hierarchy for multiple of airavata, >>>>>>>> orchestrator,appcatalog and gfac thrift services. >>>>>>>> >>>>>>>> >>>>>>>> For specifically for gfac we can have different types of services >>>>>>>> >>>>>>>> >>>>>>>> for >> >> >>>>>>> each >>>>>>> >>>>>>> >>>>>>>> provider implementation. This can be achieved by using the >>>>>>>> >>>>>>>> >>>>>>>> hierarchical >>>>>>> >>>>>>> >>>>>>>> support in zookeeper and providing some logic in gfac-thrift >>>>>>>> >>>>>>>> >>>>>>>> service >> >> >>>>>>> to >>>>>>> >>>>>>> >>>>>>>> register it to a defined path. Using the same logic orchestrator >>>>>>>> >>>>>>>> >>>>>>>> can >> >> >>>>>>>> discover the provider specific gfac thrift service and route the >>>>>>>> >>>>>>>> >>>>>>>> message to >>>>>>> >>>>>>> >>>>>>>> the correct thrift service. >>>>>>>> >>>>>>>> >>>>>>>> With this approach I think we simply have write some client code >>>>>>>> >>>>>>>> >>>>>>>> in >> >> >>>>>>> thrift >>>>>>> >>>>>>> >>>>>>>> services and clients and zookeeper server installation can be >>>>>>>> >>>>>>>> >>>>>>>> done as >> >> >>>>>>> a >>>>>>> >>>>>>> >>>>>>>> separate process and it will be easier to keep the Zookeeper >>>>>>>> >>>>>>>> >>>>>>>> server >> >> >>>>>>>> separate from Airavata because installation of Zookeeper server >>>>>>>> >>>>>>>> >>>>>>>> little >> >> >>>>>>>> complex in production scenario. I think we have to make sure >>>>>>>> >>>>>>>> >>>>>>>> everything >>>>>>> >>>>>>> >>>>>>>> works fine when there is no Zookeeper running, ex: >>>>>>>> >>>>>>>> >>>>>>>> enable.zookeeper=false >>>>>>> >>>>>>> >>>>>>>> should works fine and users doesn't have to download and start >>>>>>>> >>>>>>>> >>>>>>>> zookeeper. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [1]http://zookeeper.apache.org/ >>>>>>>> >>>>>>>> >>>>>>>> Thanks >>>>>>>> Lahiru >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> System Analyst Programmer >>>>> PTI Lab >>>>> Indiana University >>>>> >>>>> >>>>> >>>> >>>> >>> >>> >>> >>> -- >>> System Analyst Programmer >>> PTI Lab >>> Indiana University >>> >>> >>> >>> >>> >>> >>> -- >>> Best Regards, >>> Shameera Rathnayaka. >>> >>> >>> email: shameera AT apache.org , shameerainfo AT gmail.com >>> Blog : http://shameerarathnayaka.blogspot.com/ >>> >>> >>> >>> >>> >>> >>> -- >>> Supun Kamburugamuva >>> Member, Apache Software Foundation; http://www.apache.org >>> E-mail: [email protected]; Mobile: +1 812 369 6762 >>> Blog: http://supunk.blogspot.com >>> >>> >>> >> >> >> >> >>
