Hi, You can find a rough draft of my proposal at http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/subhobanerjee/13001
I am now working against the clock (6 hours left) to iron out the edges and to put in any content I am missing. Any comments would be welcome. Cheers, Subho, On Fri, May 3, 2013 at 5:24 PM, Danushka Menikkumbura < [email protected]> wrote: > Please find the proposal for "AMQP Messaging protocol support for Airavata > WS-Messenger" at [1] > > Thanks, > Danushka > > [1] - > > https://google-melange.appspot.com/gsoc/proposal/review/google/gsoc2013/danushka/1# > > > On Thu, May 2, 2013 at 7:56 PM, Suresh Marru <[email protected]> wrote: > > > Ping!! No proposals yet beyond Shameera's and Danushka's place holder. > > > > Suresh > > > > On May 1, 2013, at 5:13 PM, Suresh Marru <[email protected]> wrote: > > > > > Hi Vijayendra, > > > > > > As you can see from Shameera's proposal, he proposed a JSON conversion > > in front of WS Messenger. Also Danuska has been proposing for the AMQP > and > > idea and deliberating its advantages. So given all these, I would suggest > > for you to keep focused on the UI aspects of the monitoring and write > into > > your proposal a plan for determining a good strategy for the plumbing to > > WS-Eventing based existing system. You can have the concrete deliverables > > of new UI to change colors based on executions (as it currently does in > > XBaya), double click and show error messages and so forth. And keep it > > exploratory for the actually messaging format. > > > > > > I do not have any opinion on the libraries you mentioned, but yaa a > ajax > > based pub system might be the right way to go. Pending the content format > > (JSON or WS-Eventing or JMS or AMQP or something else) > > > > > > Suresh > > > > > > On May 1, 2013, at 4:13 PM, Vijayendra Grampurohit < > > [email protected]> wrote: > > > > > >> Hi Suresh > > >> > > >> I am writing proposal for monitoring tool . The monitoring tool is > > based on > > >> pub-sub model (ws-messenger). > > >> While writing proposal , I have to back it by technical stuff that > tells > > >> how can we achieve our purpose. > > >> As this monitoring tool is supposed to be a web based , and we are > > thinking > > >> in the lines of > > >> developing it in javascript. > > >> I was looking into javascript libraries that can we used with > > ws-messenger > > >> in the monitoring module. > > >> Please correct me if I am wrong. > > >> I came across some of the libraries > > >> > > >> - jQuery custom > > >> events< > > http://weblog.bocoup.com/publishsubscribe-with-jquery-custom-events> > > >> - AmplifyJS Pub/Sub <http://amplifyjs.com/api/pubsub/> > > >> - PubSubJS <https://github.com/mroderick/PubSubJS> > > >> - js-signals <http://millermedeiros.github.com/js-signals/> > > >> > > >> please tell me am I thinking in right direction? > > >> > > >> Regards > > >> Vijayendra > > >> > > >> > > >> > > >> > > >> On Wed, May 1, 2013 at 5:30 PM, Suresh Marru <[email protected]> > wrote: > > >> > > >>> Hi Shameera, > > >>> > > >>> This is great, I appreciate you sharing it, I realize this is still > > >>> working document, but I want other students to start seeing it and > > model > > >>> their proposals in a similar way. > > >>> > > >>> Airavata Mentors, > > >>> > > >>> Please provide feedback directly on the melange site and uncheck the > > >>> "private" box when you comment. > > >>> > > >>> Suresh > > >>> > > >>> On May 1, 2013, at 7:52 AM, Shameera Rathnayaka < > > [email protected]> > > >>> wrote: > > >>> > > >>>> Hi Suresh and All, > > >>>> > > >>>> Of course I am very much happy to share my proposal with everybody, > > >>>> actually i was going to update this thread with the melange link in > > few > > >>>> hours once i have done writing all the sections in the proposal. I > > >>> haven't > > >>>> yet added the milestone plan into it and now working on it. > > >>>> > > >>>> The sub area i am going to work from the Master project is ' > > >>> Implementing > > >>>> a JSON interface to Airavata Client side and Registry component'. > > Here is > > >>>> the link > > >>>> > > >>> > > > http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/shameera/60002# > > >>>> . > > >>>> > > >>>> > > >>>> Please note that i haven't completed everything in this and still > > doing > > >>>> modifications .Therefore proposal content may be changed bit, need > to > > add > > >>>> more technical details of the approach which explains it well. > > >>>> > > >>>> I would like to know the feedback from all of you regarding the > > proposal > > >>>> and will be modifying it if there is anything to be done. Also > please > > >>>> contact me if you need any help and i am very much willing to share > my > > >>>> thoughts with all. > > >>>> > > >>>> Thanks! > > >>>> Shameera > > >>>> > > >>>> > > >>>> > > >>>> On Wed, May 1, 2013 at 4:51 PM, Suresh Marru <[email protected]> > > wrote: > > >>>> > > >>>>> Hi Shameera, > > >>>>> > > >>>>> Excellent proposal, great job. Would you mind to make your > proposal > > >>>>> public and post the link here? Your proposal should help others to > > look > > >>> at > > >>>>> it and learn from. > > >>>>> > > >>>>> Again I emphasize to all students, please don't feel you will be > > >>> competing > > >>>>> with each others. If all of you write good proposals, there is a > good > > >>>>> chance all of you will be selected. But without a good proposal, we > > >>> cannot > > >>>>> help. > > >>>>> > > >>>>> Suresh > > >>>>> > > >>>>> > > >>>>> On Apr 23, 2013, at 1:22 PM, Shameera Rathnayaka < > > >>> [email protected]> > > >>>>> wrote: > > >>>>> > > >>>>>> Hi, > > >>>>>> > > >>>>>> Yes it is not easy to solve all problems, But defining our own > > standard > > >>>>> or > > >>>>>> adhere to any standard > > >>>>>> provided by third party library will solve the problem to some > > extend. > > >>>>>> > > >>>>>> Here i see two possible approaches, > > >>>>>> > > >>>>>> 1. Use existing third party library(we can find which is best) > > adhere > > >>> to > > >>>>> it > > >>>>>> standard and see how we change the > > >>>>>> backend to be inline with it. > > >>>>>> > > >>>>>> 2. Use our own convention with help of XMLSchema (The way i > > suggest). > > >>>>>> > > >>>>>> As Suresh mentioned we can do a POC with both approaches to > compare > > >>>>>> performance > > >>>>>> and changes need to be done in server side. Then select the best > > one. > > >>>>>> > > >>>>>> Another question was, can we works with graph data in JSON format. > > >>>>>> There are few JS graph framworks[1] which provide that > > functionality. > > >>>>>> we can use one of them to show airavata monitoring data as graphs > > >>>>>> > > >>>>>> Thanks, > > >>>>>> Shameera. > > >>>>>> > > >>>>>> [1] jqPlot <http://www.jqplot.com/index.php> , D3 < > http://d3js.org/> > > , > > >>>>>> Processing.js <http://processingjs.org> , Sencha > > >>>>>> Charts<http://www.sencha.com/products/extjs/> > > >>>>>> > > >>>>>> > > >>>>>> On Tue, Apr 23, 2013 at 5:44 PM, Suresh Marru <[email protected]> > > >>> wrote: > > >>>>>> > > >>>>>>> Hi Vijeyandra, > > >>>>>>> > > >>>>>>> Airavata Messaging is based on a pub-sub model and the events > > >>> themselves > > >>>>>>> are xml (WS-Eventing [1]). > > >>>>>>> > > >>>>>>> The Messenger paper [2] should give you more information. > > >>>>>>> > > >>>>>>> Hi All (Especially those at WS02): > > >>>>>>> > > >>>>>>> Here is an old effort from a Morotuwa undergrad project, you may > > want > > >>> to > > >>>>>>> read through these papers and chat with the authors to get > > >>> experiences: > > >>>>>>> > > >>>>>>> http://dl.acm.org/citation.cfm?id=1890807 > > >>>>>>> > > http://mgc2010.lncc.br/slides-pdf/Mooshabaya_Final_Presentation.pdf > > >>>>>>> > > http://kkpradeeban.blogspot.com/2010/09/mooshabaya-story-behind.html > > >>>>>>> http://mooshabaya.wikidot.com/ > > >>>>>>> > > >>>>>>> > > >>>>> > > >>> > > > http://chamibuddhika.wordpress.com/2009/10/06/mooshabaya-generates-mashups-from-workflows/ > > >>>>>>> > > >>>>>>> Suresh > > >>>>>>> [1] - http://www.w3.org/Submission/WS-Eventing/ > > >>>>>>> [2] - > > >>>>>>> > > >>>>> > > >>> > > http://www.extreme.indiana.edu/xgws/messenger/doc/HuangY-WSMessenger.pdf > > >>>>>>> > > >>>>>>> On Apr 23, 2013, at 6:20 AM, Vijayendra Grampurohit < > > >>>>>>> [email protected]> wrote: > > >>>>>>> > > >>>>>>>> Hi Suresh > > >>>>>>>> > > >>>>>>>> I wanted to know more about the monitoring tool . > > >>>>>>>> Currently from where does the monitoring tool gets data . Is it > > from > > >>>>>>>> workflow interpreter ? or Is it from the WS Messenger ( that > might > > >>>>>>> continuously > > >>>>>>>> send messages to monitoring tool as to tell how much is the > > progress > > >>>>>>>> and what are the variables getting changed) ? > > >>>>>>>> > > >>>>>>>> Again the how is the data being exchanged. I guess it must be > xml > > ? > > >>>>>>>> It must be one way data exchange . I mean the data is TO the > > >>>>>>>> monitoring module. > > >>>>>>>> Then monitoring Tool is sending back this > > >>>>>>>> data to Xbaya for displaying to the user ? Please correct me if > I > > am > > >>>>>>> wrong > > >>>>>>>> > > >>>>>>>> I have downloaded the source code from the trunk . can you > please > > >>> point > > >>>>>>>> me which part of code should I be code at for this module. > > >>>>>>>> > > >>>>>>>> Regards > > >>>>>>>> Vijayendra > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On Tue, Apr 23, 2013 at 3:16 PM, Vijayendra Grampurohit < > > >>>>>>> [email protected]> wrote: > > >>>>>>>> Hi > > >>>>>>>> > > >>>>>>>> What i am suggesting is, we send the JSON message directly to > > >>> Airavata > > >>>>>>>> Backend(or Registry) > > >>>>>>>> When the message gets build after the transport phase, convert > > JSON > > >>>>>>> message > > >>>>>>>> to SOAP(XML). > > >>>>>>>> From that point message will treated as SOAP message. > > >>>>>>>> > > >>>>>>>> If we look at the JSON <--> XML conversion there are set of > third > > >>> party > > >>>>>>>> libraries we > > >>>>>>>> can use for. But before selecting a one we need to think about > > >>> problems > > >>>>>>>> having > > >>>>>>>> > > >>>>>>>> with JSON <--> XML and how these libraries handle those issues. > > >>> Because > > >>>>>>> we > > >>>>>>>> need a robust > > >>>>>>>> way to do this conversions. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Shameera what you are suggesting is sending the JSON message > > directly > > >>>>> to > > >>>>>>> Registry. > > >>>>>>>> when the message gets built after the transport phase , convert > > it to > > >>>>>>> SOAP . > > >>>>>>>> > > >>>>>>>> If you are suggesting Registry will have JSON data instead of > > WSDL , > > >>>>>>> Then this might > > >>>>>>>> complicate the things for us . > > >>>>>>>> The workflow interpreter needs wsdl(xml) to interpret the > > workflows > > >>> and > > >>>>>>> for other details . > > >>>>>>>> Which means we might again have to do some changes with workflow > > >>>>>>> interpretor . > > >>>>>>>> Yesterday from what I heard in discussion is that , they do not > > want > > >>> to > > >>>>>>> mess with workflow > > >>>>>>>> interpreter atleast for GSOC projects. > > >>>>>>>> > > >>>>>>>> What I want to suggest is , why carry the JSON data till > > Regisrty . > > >>>>>>> Build a interface > > >>>>>>>> before (Apache server API) where we can do the necessary > > conversion > > >>>>>>> (JSON to SOAP). > > >>>>>>>> In this way we can avoid messing up with Airavata server as a > > whole. > > >>>>>>>> Client ( using a we browser) is interacting with JSON (web > > service) > > >>> but > > >>>>>>> the Apache server > > >>>>>>>> is interacting with SOAP. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Secondly yesterday Suresh was speaking about validating the > > >>> connections > > >>>>>>> of the workflow. > > >>>>>>>> for example , the workflow is expecting a file as input > > >>>>>>>> but user is giving a sting or int . > > >>>>>>>> > > >>>>>>>> Here what I suggest is , while creating wsdl in the registry > for a > > >>>>>>> particular > > >>>>>>>> workflow , we can add extra information in the form of > > >>>>>>>> annotation as the kind of input/ output the workflow is > > accepting. > > >>>>>>>> Then we will be able to check these against users entry during > > >>>>> execution. > > >>>>>>>> Please correct me if I am wrong. > > >>>>>>>> > > >>>>>>>> Regards > > >>>>>>>> Vijayendra > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On Tue, Apr 23, 2013 at 1:13 PM, Subho Banerjee < > > [email protected] > > >>>> > > >>>>>>> wrote: > > >>>>>>>> Well exactly, as long as you can define standard way of > > >>> communication. > > >>>>>>> That > > >>>>>>>> is, you can define in advance what should be a string, array and > > what > > >>>>>>>> should be a integer etc. We have no problem. > > >>>>>>>> > > >>>>>>>> So, when you look at problems, with JSON <-> XML or the other > way > > >>>>> round, > > >>>>>>>> they talk of the very general case (where you no nothing about > the > > >>> data > > >>>>>>> you > > >>>>>>>> are converting other than it is valid XML/JSON). There are a > > myriad > > >>> of > > >>>>>>>> problems in that case, which you pointed out. > > >>>>>>>> > > >>>>>>>> But when there is standard, there is only one way of doing > things, > > >>> and > > >>>>>>> not > > >>>>>>>> several. I think that is the way forward. So what I am proposing > > is > > >>>>> maybe > > >>>>>>>> we all discuss and define this standard within the first week of > > GSoC > > >>>>>>>> starting and then actually move into coding. So as long as we > work > > >>> with > > >>>>>>> the > > >>>>>>>> presumption that this will be done, we really dont have to > worry a > > >>> lot > > >>>>>>>> about this. > > >>>>>>>> > > >>>>>>>> Cheers, > > >>>>>>>> Subho. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On Tue, Apr 23, 2013 at 11:52 AM, Shameera Rathnayaka < > > >>>>>>>> [email protected]> wrote: > > >>>>>>>> > > >>>>>>>>> Hi, > > >>>>>>>>> > > >>>>>>>>> On Tue, Apr 23, 2013 at 2:25 AM, Subho Banerjee < > > >>> [email protected]> > > >>>>>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> Some of these problems are very specific to what the XML is > > >>>>>>>>> representing, > > >>>>>>>>>> it might not be an actual problem in Airavata, > > >>>>>>>>>> maybe some one more experienced with the codebase can point > this > > >>> out. > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> All issues pointed out in the paper is not directly valid to > our > > >>>>>>>>> conversion, I didn't list the issues actually need to address > in > > >>> this > > >>>>>>> case > > >>>>>>>>> because thought it is worth to read that introduction part > which > > >>>>>>> explain > > >>>>>>>>> the all the issues we have with this conversion and give us a > > solid > > >>>>>>>>> background of that. > > >>>>>>>>> > > >>>>>>>>>> 1. Anonymous values, Arrays, Implicit Typing, Character sets > -- > > I > > >>>>>>>>> really > > >>>>>>>>>> dont see these as problems, as long as you can agree that all > > >>>>>>> parts of > > >>>>>>>>>> airavata will treat the JSON in a standard (probably we have > to > > >>>>>>> define > > >>>>>>>>>> this) way. > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> The issue with JSON array only comes when we try to convert XML > > to > > >>>>>>> JSON not > > >>>>>>>>> the other way. If we map with JSON, inputparameters and > > >>>>>>> outputparameters in > > >>>>>>>>> the ServiceDescription.xsd will map with JSON Arrays. Therefore > > we > > >>>>>>> need to > > >>>>>>>>> solve this issue. > > >>>>>>>>> > > >>>>>>>>> JSON XML JSON > > >>>>>>>>> {"inputs":["test"]} --> <inputs>test<inputs> --> > > >>> {"inputs":["test"]} > > >>>>>>> // > > >>>>>>>>> correct one > > >>>>>>>>> --> {"inputs":"test"} // incorrect > one > > >>>>>>>>> > > >>>>>>>>> 2. Namespaces, Processing Instructions -- Is this required? > > >>>>>>>>> > > >>>>>>>>>> Are separate namespaces used in Airavata? Only place I can see > > >>>>>>> this > > >>>>>>>>>> being > > >>>>>>>>>> used is probably in the WSDL, but if we can agree on another > way > > >>>>>>>>>> of communicating registered applications' I/O parameters to > the > > >>>>>>> front > > >>>>>>>>>> end > > >>>>>>>>>> (JSON based), then maybe we can work around this (minor) > > problem. > > >>>>>>> Are > > >>>>>>>>>> custom processing instructions to the Xbaya XML parse even > used? > > >>>>>>>>>> 3. Attributes -- Again, this can be fixed easily > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> Yes,attributes convertion will not be a big issues we can solve > > it. > > >>> As > > >>>>>>>>> Lahiru mentioned in Hangout session namesapce handling is not a > > big > > >>>>>>> issue > > >>>>>>>>> with Airavata. > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> <array name="abc"> > > >>>>>>>>>> <element>1</element> > > >>>>>>>>>> <element>2</element> > > >>>>>>>>>> <element>3</element> > > >>>>>>>>>> <element>4</element> > > >>>>>>>>>> </array> > > >>>>>>>>>> > > >>>>>>>>>> Can become > > >>>>>>>>>> > > >>>>>>>>>> { > > >>>>>>>>>> > > >>>>>>>>>> abc : ['1', '2', '3', '4'] > > >>>>>>>>>> > > >>>>>>>>>> } > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> With this example it show us we need to change the XML message > > >>> format > > >>>>>>> of > > >>>>>>>>> server side, which require to change the all schemas, If we are > > >>> going > > >>>>>>> to > > >>>>>>>>> change the schemas then we need to change the way it process it > > in > > >>>>>>> Ariavara > > >>>>>>>>> core. We have dropped our initial major requirement, which is > > keep > > >>> the > > >>>>>>>>> Airavata Server side as it is. > > >>>>>>>>> > > >>>>>>>>> with this conversion we only deal with json strings, yes we can > > send > > >>>>>>> JSON > > >>>>>>>>> request with other formats supported by JSON like boolen, null, > > >>> Number > > >>>>>>>>> etc.. But there is no way to get the same JSON from XML as XML > > only > > >>>>>>> deal > > >>>>>>>>> only with Strings. I think it is good if we can consume a this > > >>>>> features > > >>>>>>>>> with JSON. > > >>>>>>>>> > > >>>>>>>>> let say i need to send a integer or float to the server using > > JSON > > >>>>> then > > >>>>>>>>> proper way is to send {"<name>":123.45} this will works fine > but > > >>>>>>> problem is > > >>>>>>>>> how we get the same output ? > > >>>>>>>>> > > >>>>>>>>> Thanks, > > >>>>>>>>> Shameera. > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> Cheers, > > >>>>>>>>>> Subho. > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> -- > > >>>>>>>>> Best Regards, > > >>>>>>>>> Shameera Rathnayaka. > > >>>>>>>>> > > >>>>>>>>> Blog : http://shameerarathnayaka.blogspot.com/ > > >>>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>> > > >>>>>> > > >>>>>> -- > > >>>>>> Best Regards, > > >>>>>> Shameera Rathnayaka. > > >>>>>> > > >>>>>> Blog : http://shameerarathnayaka.blogspot.com/ > > >>>>> > > >>>>> > > >>>> > > >>>> > > >>>> -- > > >>>> Best Regards, > > >>>> Shameera Rathnayaka. > > >>>> > > >>>> email: shameera AT apache.org , shameerainfo AT gmail.com > > >>>> Blog : http://shameerarathnayaka.blogspot.com/ > > >>> > > >>> > > > > > > > >
