My regrets for missing the meeting but I was babysitting. Marlon
On 3/2/14 11:16 PM, Suresh Marru wrote: > Thank you all for taking couple of hours on a sunday evening to participate. > I think these discussions help Airavata very significantly. > > Here is the you tube link is any one would like to follow - > http://www.youtube.com/watch?v=EY6oPwqi1g4 > > Key Summary: Sachith is interested to do a GSoC project on this topic and he > will start with summarizing the challenges in current registry. Once the > problem statement is more clearer, we can take the next steps. > > Appreciate every one input on this key topic. > > Suresh > > P.S. I will be traveling for next 4 days, so I will be slow in my responses. > > On Mar 2, 2014, at 7:56 PM, Suresh Marru <[email protected]> wrote: > >> Lets use this - >> https://plus.google.com/hangouts/_/hoaevent/AP36tYdve-71oizx25DGUbTZjSX4PtLxmDsddtqnfuDYlE9SXDSB9Q?authuser=0&hl=en >> >> I will compile a set of instructions for website so any one of us can >> preschedule it for future. >> >> Sures >> >> On Mar 2, 2014, at 7:36 PM, Eran Chinthaka Withana >> <[email protected]> wrote: >> >>> Oops, in that case, Suresh, can you please create one? >>> >>> Thanks, >>> Eran Chinthaka Withana >>> >>> >>> On Sun, Mar 2, 2014 at 4:28 PM, Suresh Marru <[email protected]> wrote: >>> >>>> Hi Eran, >>>> >>>> Is this a On-Air event? Previously I had trouble changing the previously >>>> scheduled event to On-Air. >>>> >>>> If you are creating a new hangout, can you first create it on G+ Airavata >>>> Community (all PMC Members are moderators on this community). This will be >>>> easier for archival reference - >>>> https://plus.google.com/communities/100700433662281905708 >>>> >>>> Suresh >>>> >>>> On Mar 2, 2014, at 7:21 PM, Eran Chinthaka Withana < >>>> [email protected]> wrote: >>>> >>>>> Here is the link to hangout: >>>>> >>>> https://plus.google.com/hangouts/_/event/c1sgvk7dha37rkr0adktb195lgc?authuser=0&hl=en >>>>> Thanks, >>>>> Eran Chinthaka Withana >>>>> >>>>> >>>>> On Sun, Mar 2, 2014 at 12:46 PM, Suresh Marru <[email protected]> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> Since Eran has been the one who first proposed the hangout and has >>>>>> specific suggestion on this thread I prefer to postpone to 8pm (EST). >>>> But >>>>>> if others planned for 4pm, lets goahead with the plan. >>>>>> >>>>>> Any one who planned to attend now cannot make it at 8pm (EST)? If do not >>>>>> hear any objections lets shoot for 8pm. Otherwise, lets go as planned. >>>>>> >>>>>> Cheers, >>>>>> Suresh >>>>>> >>>>>> On Mar 2, 2014, at 3:31 PM, Eran Chinthaka Withana < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Suresh, >>>>>>> >>>>>>> Sorry for the late reply. I don't think I can make it at 1pm PST today. >>>>>> Can >>>>>>> we please re-schedule this to 5pm PST (8pm EST) or later? >>>>>>> >>>>>>> Thanks, >>>>>>> Eran Chinthaka Withana >>>>>>> >>>>>>> >>>>>>> On Sun, Mar 2, 2014 at 6:38 AM, Suresh Marru <[email protected]> >>>> wrote: >>>>>>>> Hi All, >>>>>>>> >>>>>>>> Great to see we have a good quorum. So how about 4pm EST (1pm PST) >>>> today >>>>>>>> with a hangout on air. It works best if we start a a hangout then >>>>>> (previous >>>>>>>> attempts to pre-schedules on-air events did not work well. So please >>>>>> check >>>>>>>> this mailing list around 4pm EST for the hangout on air link. >>>>>>>> >>>>>>>> Meanwhile, please join the Airavata Google Plus community, that might >>>> be >>>>>>>> easier to share the link - >>>>>>>> https://plus.google.com/communities/100700433662281905708 >>>>>>>> >>>>>>>> Thanks all for willing to take time on a sunday, >>>>>>>> Suresh >>>>>>>> >>>>>>>> On Feb 28, 2014, at 9:15 PM, Supun Kamburugamuva <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> +1 for Sunday afternoon. I can make it after 4 pm EST. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Supun.. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Feb 28, 2014 at 5:04 PM, Shameera Rathnayaka < >>>>>>>> [email protected] >>>>>>>>>> wrote: >>>>>>>>>> +1 >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Shameera. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sat, Mar 1, 2014 at 3:11 AM, Eran Chinthaka Withana < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> +1 for Sunday afternoon >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Eran Chinthaka Withana >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, Feb 28, 2014 at 5:17 AM, Suresh Marru <[email protected]> >>>>>>>> wrote: >>>>>>>>>>>> Hi Eran, >>>>>>>>>>>> >>>>>>>>>>>> This is a great idea. I myself owe few replies on this thread and >>>>>>>>>> unable >>>>>>>>>>>> to take time to comprehend my thoughts (and realized I should take >>>>>>>> time >>>>>>>>>>> to >>>>>>>>>>>> properly articulate the challenges otherwise we will be discussing >>>>>>>>>>>> orthogonal issues). >>>>>>>>>>>> >>>>>>>>>>>> A hangout will help us brainstorm more comprehensively. We can >>>> have >>>>>> it >>>>>>>>>> on >>>>>>>>>>>> air so we can refer back for archival purposes. How is Sunday >>>>>>>> afternoon >>>>>>>>>>> for >>>>>>>>>>>> everyone willing to join and contribute? >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Suresh >>>>>>>>>>>> >>>>>>>>>>>> On Feb 28, 2014, at 1:45 AM, Eran Chinthaka Withana < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> Is there any chance of hosting a google hangout to talk about >>>>>> this. I >>>>>>>>>>>> think >>>>>>>>>>>>> with long emails and multiple directions things are getting >>>> little >>>>>>>>>> bit >>>>>>>>>>>>> confusing in thread (I'm partly responsible for this :) ). I can >>>>>>>>>> join a >>>>>>>>>>>>> video chat during a weekend but lets make sure its convenient for >>>>>>>>>> both >>>>>>>>>>>> east >>>>>>>>>>>>> and west coasts :) >>>>>>>>>>>>> >>>>>>>>>>>>> WDYT? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Eran Chinthaka Withana >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Feb 24, 2014 at 9:32 AM, Suresh Marru <[email protected] >>>>>>>>>>> wrote: >>>>>>>>>>>>>> I could respond to each thread in detail, but I see the general >>>>>>>>>> sense >>>>>>>>>>> is >>>>>>>>>>>>>> inquiring on the use case, so let me try and explain this and >>>> see >>>>>> if >>>>>>>>>>> it >>>>>>>>>>>>>> comes across. I am fully onboard with perceptions of relational >>>> vs >>>>>>>>>>> nosql >>>>>>>>>>>>>> and also agree current Airavata needs are not a direct map for >>>>>> NoSQL >>>>>>>>>>>>>> migration. I will summarize the driving motivation: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Background: The key problem Airavata needs to solve is getting >>>> the >>>>>>>>>> API >>>>>>>>>>>> and >>>>>>>>>>>>>> associated data model right. The problem is current relational >>>>>>>>>>> database >>>>>>>>>>>>>> (with OpenJPA overlay) is severely limiting the API evolution. >>>>>>>>>> Science >>>>>>>>>>>>>> Gateways by nature are very science domain and use-case >>>> specific. >>>>>>>>>> But >>>>>>>>>>>>>> Airavata is tackling this challenging problem of providing a >>>>>> generic >>>>>>>>>>> API >>>>>>>>>>>>>> which will meet and enable these use case centric integration. >>>> The >>>>>>>>>>> issue >>>>>>>>>>>>>> here is, we are designing an API to handle a wide range of known >>>>>>>>>> (and >>>>>>>>>>>> some >>>>>>>>>>>>>> foreseen) use cases. But at the same time trying to keep it >>>> simple >>>>>>>>>> and >>>>>>>>>>>> yet >>>>>>>>>>>>>> flexible. The only way we can get through a reasonable, >>>> normalized >>>>>>>>>>>> version >>>>>>>>>>>>>> of API is by hands-on programming against the API. Within the >>>>>>>>>> Airavata >>>>>>>>>>>> PMC >>>>>>>>>>>>>> itself, we can solicit a half-a-dozen different ways on how to >>>>>>>>>>> visualize >>>>>>>>>>>>>> the data model. And we need few hackethon's with real-end users >>>> of >>>>>>>>>>>> Airavata >>>>>>>>>>>>>> until we find a common ground. All of this needs rapid >>>>>> prototyping. >>>>>>>>>>>>>> Currently a slight change in the data model is taking close to >>>> two >>>>>>>>>>>> weeks of >>>>>>>>>>>>>> re-arcitecting the Open-JPA based registry. There are many known >>>>>>>>>>>> problems >>>>>>>>>>>>>> with current draft of data model which have to be put-down in >>>> the >>>>>>>>>>>> interest >>>>>>>>>>>>>> of making over all system progress. >>>>>>>>>>>>>> >>>>>>>>>>>>>> So the driving motivation is not certainly any of the classic >>>>>> NoSQL >>>>>>>>>>>> needs. >>>>>>>>>>>>>> But a simple one, can we have registry which is schema-agnostic >>>>>> and >>>>>>>>>>> yet >>>>>>>>>>>> is >>>>>>>>>>>>>> queriable for most of the fields in the model? Can we try 10 >>>>>>>>>> different >>>>>>>>>>>>>> variants of data model (hence API) within the next 3 months with >>>>>>>>>>> focused >>>>>>>>>>>>>> hackethon's and arrive at a stable 1.0 version of API? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Part one is the discussion is successful that it raised every >>>>>> one's >>>>>>>>>>> eye >>>>>>>>>>>>>> brows. Now that we have every one's attention, what will be a >>>> good >>>>>>>>>>> data >>>>>>>>>>>>>> store for Airavata which will meet these needs? >>>>>>>>>>>>>> >>>>>>>>>>>>>> P.S: Additional background: The API has been in development for >>>>>>>>>> close >>>>>>>>>>>> to 3 >>>>>>>>>>>>>> years and is falling short of pleasing a majority. Many academic >>>>>>>>>>>>>> standardization efforts fail terribly trying to pretend to >>>>>>>>>> understand >>>>>>>>>>>> all >>>>>>>>>>>>>> use cases and proposing a standard way (which ends up >>>>>> unnecessarily >>>>>>>>>>>> complex >>>>>>>>>>>>>> and not usable). Science by nature is evolutionary, and >>>>>> restricting >>>>>>>>>>> the >>>>>>>>>>>>>> capabilities by a known set of use cases prevents the use of >>>>>>>>>>> middleware >>>>>>>>>>>> for >>>>>>>>>>>>>> real-scientific research (and gets limited to proof of concept >>>>>>>>>>>>>> demonstrations, papers, educational use). The only way meeting >>>> the >>>>>>>>>>>>>> challenges of these evolving needs is to have the framework >>>> which >>>>>>>>>> can >>>>>>>>>>>>>> evolve with minimal disruption. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Great thoughts so far, please keep 'em coming until we can find >>>> a >>>>>>>>>>>> solution >>>>>>>>>>>>>> not by the technical fancies but to address the real need. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>> Suresh >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Feb 24, 2014, at 11:53 AM, Lahiru Gunathilake < >>>>>> [email protected] >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Feb 24, 2014 at 11:20 AM, Milinda Pathirage < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I also think that moving to Cassandra or any other NoSQL will >>>>>> add >>>>>>>>>>>>>>>> unneccessary complexity to your solution. Also designing >>>> proper >>>>>>>>>>> (easy >>>>>>>>>>>> to >>>>>>>>>>>>>>>> manage changes, easy to query) NoSQL data models are hard >>>>>> (AFAIK, >>>>>>>>>>>>>> require >>>>>>>>>>>>>>>> lots of experience and understanding about data structures and >>>>>>>>>>>> queries). >>>>>>>>>>>>>>>> Also migrating from one NoSQL technology to other can require >>>>>>>>>>> complete >>>>>>>>>>>>>>>> re-write. And current relational databases can handle heavy >>>>>> loads >>>>>>>>>>>> except >>>>>>>>>>>>>>>> Google, Twitter, Amazon and Facebook like loads. I don't think >>>>>>>>>>>> Airavata >>>>>>>>>>>>>>>> will see Google and Amazon like loads. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> If the constant changes to the data model is the problem , I >>>>>> think >>>>>>>>>>>> best >>>>>>>>>>>>>>>> option is to abstract registry implementation to something >>>> like >>>>>>>>>>>>>> collections >>>>>>>>>>>>>>>> and resources used in WSO2 Registry [1] or something suitable >>>>>> for >>>>>>>>>>>>>> Airavata >>>>>>>>>>>>>>>> context. That will make it easy to handle changes in data >>>> model. >>>>>>>>>>>>>>>> Also don't let the technologies drive design decision. Its >>>>>> always >>>>>>>>>>>>>> better to >>>>>>>>>>>>>>>> let use cases drive the design decision. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards >>>>>>>>>>>>>>> Lahiru >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>> Milinda >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [1] http://wso2.com/products/governance-registry/ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, Feb 24, 2014 at 10:57 AM, Supun Kamburugamuva < >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I'm not trying to discourage you on your exploration to NoSQL >>>>>>>>>>>>>> databases. >>>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>>> have the following concern. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Your database schema is moderately complex - even for a RDBMS >>>>>> it >>>>>>>>>>>> seems >>>>>>>>>>>>>>>>> complex and the data size is relatively small. I'm not sure >>>>>> about >>>>>>>>>>> the >>>>>>>>>>>>>>>>> current tools available but I think you will need to write >>>> more >>>>>>>>>>> code >>>>>>>>>>>> to >>>>>>>>>>>>>>>>> support all your requirements in a NoSQL database. So writing >>>>>>>>>> more >>>>>>>>>>>> code >>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>> allow redundancy to support *relatively small* and >>>> *structured >>>>>>>>>>>>>>>>> data*doesn't seem right to me. May be I'm wrong and there are >>>>>>>>>>> better >>>>>>>>>>>>>>>>> tools in >>>>>>>>>>>>>>>>> NoSQL than RDBMS, which I doubt. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Supun.. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Sun, Feb 23, 2014 at 5:20 PM, Suresh Marru < >>>>>> [email protected] >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Airavata is actively migrating to use Thrift API for the >>>>>>>>>> RESTless >>>>>>>>>>>>>>>> design >>>>>>>>>>>>>>>>>> and to facilitate various language bindings from client >>>>>>>>>> gateways. >>>>>>>>>>>> The >>>>>>>>>>>>>>>>>> programming language support in thrift has been so far very >>>>>>>>>>>>>>>> encouraging. >>>>>>>>>>>>>>>>>> The current architecture is looking like Figure 1 at [1]. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Language specific clients will be released as thrift SDK's >>>>>>>>>>> (similar >>>>>>>>>>>> to >>>>>>>>>>>>>>>>>> evernote sdk's [1]). These clients will be integrated into >>>>>>>>>> gateway >>>>>>>>>>>>>>>>> portals >>>>>>>>>>>>>>>>>> which connect to the API Server. The API operations brokers >>>> he >>>>>>>>>>>> simple >>>>>>>>>>>>>>>>> calls >>>>>>>>>>>>>>>>>> into one or more backend CPI calls (Airavata internal >>>>>> component >>>>>>>>>>>>>>>>>> interfaces). An example set of mappings are illustrated in >>>>>>>>>>> Figure 2 >>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> [1]. The current draft of thrift API for version 0.12 is at >>>>>> [3], >>>>>>>>>>>>>> please >>>>>>>>>>>>>>>>> pay >>>>>>>>>>>>>>>>>> attention to experiment model at [4]. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> For the persistent store, we had few iterations of Airavata >>>>>>>>>>> Registry >>>>>>>>>>>>>>>>>> shifting from a legacy XRegistry to JackRabbit to now a >>>>>> OpenJPA >>>>>>>>>>>> based >>>>>>>>>>>>>>>>>> registry. To allow the API and the associated data models to >>>>>>>>>>> evolve, >>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>> will be useful to explore object databases so we can store >>>> the >>>>>>>>>>>>>>>> serialized >>>>>>>>>>>>>>>>>> version of thrift objects directly. But it will be nice to >>>>>> have >>>>>>>>>>> all >>>>>>>>>>>>>> (or >>>>>>>>>>>>>>>>>> most) of the fields queriable. This calls for a more >>>>>>>>>> column-family >>>>>>>>>>>>>>>> design >>>>>>>>>>>>>>>>>> of any NoSQL approaches. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Any recommendations for a registry architecture? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Quickly hacking through I find the following approach a >>>> viable >>>>>>>>>>> one: >>>>>>>>>>>>>>>>>> ZombieDB[5] over astyanax[6] which talks to Cassandra. >>>>>> Airavata >>>>>>>>>>> can >>>>>>>>>>>>>>>>> benefit >>>>>>>>>>>>>>>>>> immediately from the replication and reliability of >>>> cassandra >>>>>>>>>> and >>>>>>>>>>>>>>>>>> scalability in near future. Some of the model objects like >>>>>>>>>>>> experiment >>>>>>>>>>>>>>>>>> creation will need to have strong consistency and most of >>>> the >>>>>>>>>>>>>>>> monitoring >>>>>>>>>>>>>>>>>> can live with eventual consistency. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Critical comments please? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks for your time, >>>>>>>>>>>>>>>>>> Suresh >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [1] - >>>>>>>>>>>>>>>>>> >>>> https://cwiki.apache.org/confluence/display/AIRAVATA/2014/02/23/Brainstorming+Diagrams >>>>>>>>>>>>>>>>>> [2] - https://dev.evernote.com/doc/ >>>>>>>>>>>>>>>>>> [3] - >>>>>>>>>>>>>>>>>> >>>> https://git-wip-us.apache.org/repos/asf?p=airavata.git;a=tree;f=airavata-api/thrift-interface-descriptions;hb=HEAD >>>>>>>>>>>>>>>>>> [4] - >>>>>>>>>>>>>>>>>> >>>> https://git-wip-us.apache.org/repos/asf?p=airavata.git;a=blob_plain;f=airavata-api/thrift-interface-descriptions/experimentModel.thrift;hb=HEAD >>>>>>>>>>>>>>>>>> [5] - https://github.com/MisterTea/ZombieDB >>>>>>>>>>>>>>>>>> [6] - https://github.com/Netflix/astyanax >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Supun Kamburugamuva >>>>>>>>>>>>>>>>> Member, Apache Software Foundation; http://www.apache.org >>>>>>>>>>>>>>>>> E-mail: [email protected]; Mobile: +1 812 369 6762 >>>>>>>>>>>>>>>>> Blog: http://supunk.blogspot.com >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Milinda Pathirage >>>>>>>>>>>>>>>> PhD Student Indiana University, Bloomington; >>>>>>>>>>>>>>>> E-mail: [email protected] >>>>>>>>>>>>>>>> Web: http://mpathirage.com >>>>>>>>>>>>>>>> Blog: http://blog.mpathirage.com >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> System Analyst Programmer >>>>>>>>>>>>>>> PTI Lab >>>>>>>>>>>>>>> Indiana University >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Best Regards, >>>>>>>>>> Shameera Rathnayaka. >>>>>>>>>> >>>>>>>>>> email: shameera AT apache.org , shameerainfo AT gmail.com >>>>>>>>>> Blog : http://shameerarathnayaka.blogspot.com/ >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Supun Kamburugamuva >>>>>>>>> Member, Apache Software Foundation; http://www.apache.org >>>>>>>>> E-mail: [email protected]; Mobile: +1 812 369 6762 >>>>>>>>> Blog: http://supunk.blogspot.com >>>>>>>> >>>>>> >>>>
