Hi Gagan, I need to start another discussion about it, but I had an offline discussion with Suresh about auto-scaling. I will start another thread about this topic too.
Regards Lahiru On Wed, Jun 11, 2014 at 4:10 PM, Gagan Juneja <[email protected]> wrote: > Thanks Lahiru for pointing to nice library, added to my dictionary :). > > I would like to know how are we planning to start multiple servers. > 1. Spawning new servers based on load? Some times we call it as auto > scalable. > 2. To make some specific number of nodes available such as we want 2 > servers to be available at any time so if one goes down then I need to > spawn one new to make available servers count 2. > 3. Initially start all the servers. > > In scenario 1 and 2 zookeeper does make sense but I don't believe existing > architecture support this? > > Regards, > Gagan > On 12-Jun-2014 1:19 am, "Lahiru Gunathilake" <[email protected]> wrote: > >> Hi Gagan, >> >> Thanks for your response. Please see my inline comments. >> >> >> On Wed, Jun 11, 2014 at 3:37 PM, Gagan Juneja <[email protected]> >> wrote: >> >>> Hi Lahiru, >>> Just my 2 cents. >>> >>> I am big fan of zookeeper but also against adding multiple hops in the >>> system which can add unnecessary complexity. Here I am not able to >>> understand the requirement of zookeeper may be I am wrong because of less >>> knowledge of the airavata system in whole. So I would like to discuss >>> following point. >>> >>> 1. How it will help us in making system more reliable. Zookeeper is not >>> able to restart services. At max it can tell whether service is up or not >>> which could only be the case if airavata service goes down gracefully and >>> we have any automated way to restart it. If this is just matter of routing >>> client requests to the available thrift servers then this can be achieved >>> with the help of load balancer which I guess is already there in thrift >>> wish list. >>> >> We have multiple thrift services and currently we start only one instance >> of them and each thrift service is a stateless service. To keep the high >> availability we have to start multiple instances of them in production >> scenario. So for clients to get an available thrift service we can use >> zookeeper znodes to represent each available service. There are some >> libraries which is doing similar[1] and I think we can use them directly. >> >>> 2. As far as registering of different providers is concerned do you >>> think for that we really need external store. >>> >> Yes I think so, because its light weight and reliable and we have to do >> very minimal amount of work to achieve all these features to Airavata >> because zookeeper handle all the complexity. >> >>> I have seen people using zookeeper more for state management in >>> distributed environments. >>> >> +1, we might not be the most effective users of zookeeper because all of >> our services are stateless services, but my point is to achieve >> fault-tolerance we can use zookeeper and with minimal work. >> >>> I would like to understand more how can we leverage zookeeper in >>> airavata to make system reliable. >>> >>> >> [1]https://github.com/eirslett/thrift-zookeeper >> >> >> >>> Regards, >>> Gagan >>> On 12-Jun-2014 12:33 am, "Marlon Pierce" <[email protected]> wrote: >>> >>>> Thanks for the summary, Lahiru. I'm cc'ing the Architecture list for >>>> additional comments. >>>> >>>> Marlon >>>> >>>> On 6/11/14 2:27 PM, Lahiru Gunathilake wrote: >>>> > Hi All, >>>> > >>>> > I did little research about Apache Zookeeper[1] and how to use it in >>>> > airavata. Its really a nice way to achieve fault tolerance and >>>> reliable >>>> > communication between our thrift services and clients. Zookeeper is a >>>> > distributed, fault tolerant system to do a reliable communication >>>> between >>>> > distributed applications. This is like an in-memory file system which >>>> has >>>> > nodes in a tree structure and each node can have small amount of data >>>> > associated with it and these nodes are called znodes. Clients can >>>> connect >>>> > to a zookeeper server and add/delete and update these znodes. >>>> > >>>> > In Apache Airavata we start multiple thrift services and these can >>>> go >>>> > down for maintenance or these can crash, if we use zookeeper to store >>>> these >>>> > configuration(thrift service configurations) we can achieve a very >>>> reliable >>>> > system. Basically thrift clients can dynamically discover available >>>> service >>>> > by using ephemeral znodes(Here we do not have to change the generated >>>> > thrift client code but we have to change the locations we are invoking >>>> > them). ephemeral znodes will be removed when the thrift service goes >>>> down >>>> > and zookeeper guarantee the atomicity between these operations. With >>>> this >>>> > approach we can have a node hierarchy for multiple of airavata, >>>> > orchestrator,appcatalog and gfac thrift services. >>>> > >>>> > For specifically for gfac we can have different types of services for >>>> each >>>> > provider implementation. This can be achieved by using the >>>> hierarchical >>>> > support in zookeeper and providing some logic in gfac-thrift service >>>> to >>>> > register it to a defined path. Using the same logic orchestrator can >>>> > discover the provider specific gfac thrift service and route the >>>> message to >>>> > the correct thrift service. >>>> > >>>> > With this approach I think we simply have write some client code in >>>> thrift >>>> > services and clients and zookeeper server installation can be done as >>>> a >>>> > separate process and it will be easier to keep the Zookeeper server >>>> > separate from Airavata because installation of Zookeeper server little >>>> > complex in production scenario. I think we have to make sure >>>> everything >>>> > works fine when there is no Zookeeper running, ex: >>>> enable.zookeeper=false >>>> > should works fine and users doesn't have to download and start >>>> zookeeper. >>>> > >>>> > >>>> > >>>> > [1]http://zookeeper.apache.org/ >>>> > >>>> > Thanks >>>> > Lahiru >>>> >>>> >> >> >> -- >> System Analyst Programmer >> PTI Lab >> Indiana University >> > -- System Analyst Programmer PTI Lab Indiana University
