Thanks for the summary, Lahiru. I'm cc'ing the Architecture list for additional comments.
Marlon On 6/11/14 2:27 PM, Lahiru Gunathilake wrote: > Hi All, > > I did little research about Apache Zookeeper[1] and how to use it in > airavata. Its really a nice way to achieve fault tolerance and reliable > communication between our thrift services and clients. Zookeeper is a > distributed, fault tolerant system to do a reliable communication between > distributed applications. This is like an in-memory file system which has > nodes in a tree structure and each node can have small amount of data > associated with it and these nodes are called znodes. Clients can connect > to a zookeeper server and add/delete and update these znodes. > > In Apache Airavata we start multiple thrift services and these can go > down for maintenance or these can crash, if we use zookeeper to store these > configuration(thrift service configurations) we can achieve a very reliable > system. Basically thrift clients can dynamically discover available service > by using ephemeral znodes(Here we do not have to change the generated > thrift client code but we have to change the locations we are invoking > them). ephemeral znodes will be removed when the thrift service goes down > and zookeeper guarantee the atomicity between these operations. With this > approach we can have a node hierarchy for multiple of airavata, > orchestrator,appcatalog and gfac thrift services. > > For specifically for gfac we can have different types of services for each > provider implementation. This can be achieved by using the hierarchical > support in zookeeper and providing some logic in gfac-thrift service to > register it to a defined path. Using the same logic orchestrator can > discover the provider specific gfac thrift service and route the message to > the correct thrift service. > > With this approach I think we simply have write some client code in thrift > services and clients and zookeeper server installation can be done as a > separate process and it will be easier to keep the Zookeeper server > separate from Airavata because installation of Zookeeper server little > complex in production scenario. I think we have to make sure everything > works fine when there is no Zookeeper running, ex: enable.zookeeper=false > should works fine and users doesn't have to download and start zookeeper. > > > > [1]http://zookeeper.apache.org/ > > Thanks > Lahiru
