First of all thanks a lot for the detailed explanation Suresh. I agree with the fact that with the current implementations and the Data Models getting complex will require a more efficient way to handle the data at the back end.
But can you please elaborate on why you chose ZombieDB over Astyanax? Hector and DataStax are also used widely in production. One more thing, We were planning to have real time processing of data stored in the registry using Storm ( or using a better alternative). If that's the case I suggest we can use Storm Cassandra project [1] as well. [1] https://github.com/hmsonline/storm-cassandra<https://github.com/hmsonline/storm-cassandra> On Sun, Feb 23, 2014 at 5:20 PM, Suresh Marru <[email protected]> wrote: > Hi All, > > Airavata is actively migrating to use Thrift API for the RESTless design > and to facilitate various language bindings from client gateways. The > programming language support in thrift has been so far very encouraging. > The current architecture is looking like Figure 1 at [1]. > > Language specific clients will be released as thrift SDK's (similar to > evernote sdk's [1]). These clients will be integrated into gateway portals > which connect to the API Server. The API operations brokers he simple calls > into one or more backend CPI calls (Airavata internal component > interfaces). An example set of mappings are illustrated in Figure 2 at > [1]. The current draft of thrift API for version 0.12 is at [3], please pay > attention to experiment model at [4]. > > For the persistent store, we had few iterations of Airavata Registry > shifting from a legacy XRegistry to JackRabbit to now a OpenJPA based > registry. To allow the API and the associated data models to evolve, it > will be useful to explore object databases so we can store the serialized > version of thrift objects directly. But it will be nice to have all (or > most) of the fields queriable. This calls for a more column-family design > of any NoSQL approaches. > > Any recommendations for a registry architecture? > > Quickly hacking through I find the following approach a viable one: > ZombieDB[5] over astyanax[6] which talks to Cassandra. Airavata can benefit > immediately from the replication and reliability of cassandra and > scalability in near future. Some of the model objects like experiment > creation will need to have strong consistency and most of the monitoring > can live with eventual consistency. > > Critical comments please? > > Thanks for your time, > Suresh > > [1] - > https://cwiki.apache.org/confluence/display/AIRAVATA/2014/02/23/Brainstorming+Diagrams > [2] - https://dev.evernote.com/doc/ > [3] - > https://git-wip-us.apache.org/repos/asf?p=airavata.git;a=tree;f=airavata-api/thrift-interface-descriptions;hb=HEAD > [4] - > https://git-wip-us.apache.org/repos/asf?p=airavata.git;a=blob_plain;f=airavata-api/thrift-interface-descriptions/experimentModel.thrift;hb=HEAD > [5] - https://github.com/MisterTea/ZombieDB > [6] - https://github.com/Netflix/astyanax > > -- Thanks, Sachith Withana
