Hi Suresh I will be writing for *workflow monitoring . *I want to know is there anybody who also wants to write for the same.* * please respond. * * Regards Vijayendra * *
On Wed, May 1, 2013 at 1:40 AM, Suresh Marru <[email protected]> wrote: > Hi All, > > Please notify what proposal you are writing against. I would say as soon > as possible create a place holder in melange and post a link. Few more > points: > > * Even though your project is focusing on a topic, do not hesitate to add > all the coding tasks and milestones related to other tasks you will be > contributing to. For example Subho was interested in workflow UI but very > much interested to contribute to the JSON-XML work with Shameera. So > Subho's proposal should include all the tasks he is interested in JSON > project as well. We very well understand all these projects are > inter-related so please do not shy away from cross-linking tasks. > > * Include a plan of development to follow Airavata's TDD approach - > https://cwiki.apache.org/confluence/display/AIRAVATA/Tests+in+Airavata > > * To make your application stronger, write briefly about how you plan to > continue working with airavata community to see your contributions can be > well integrated into the main code base and be part of the releases. Your > commitment to support your contributions will be a big plus. > > * Do not forget to add details about any technical papers you plan to > write based on this GSoC work. > > Good luck with your proposals, > Suresh > > > On Apr 29, 2013, at 11:20 PM, Suresh Marru <[email protected]> wrote: > > > Hi All, > > > > Many of you have asked about the gsoc template, I do not think we want > to have any prescriptive format and each of you can follow your style of > the proposal. I just created one if its helps - > https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+2013+Student+Application+Templatebut > don't stick to it, you can free flow yours. > > > > But please make your proposal is public and discuss with each other. Few > important points: > > > > * You are not competing against each other. The goal is to solve a > collective problem. If one of you falls behind, it might impact the project > entirely. So if you know your fellow student will not be able to spend > enough time, its equally your responsibility to make they write those times > as unavailable now. > > * Make sure all of you align the project goals. So please discuss as > much as possible within the next three days. Please do not wait for me or > others on the PMC to respond. You guys should be able to interact with each > other. > > * Directly write the proposal and edit it in melange directly. > > * The melange proposal titles need not match the JIRA ideas page, so > feel free to make your own titles. > > * Create public proposals and post the link here so other students can > see your progress. > > * Leave the mentor blank to start with, we will assign ourselves within > the next day. > > > > Cheers, > > Suresh > > > > On Apr 28, 2013, at 6:46 AM, Vijayendra Grampurohit < > [email protected]> wrote: > > > >> Hi Suresh > >> > >> I would like to concentrate on Workflow monitoring . > >> > >> Regards > >> Vijayendra > >> > >> > >> On Sun, Apr 28, 2013 at 12:09 PM, Shameera Rathnayaka < > >> [email protected]> wrote: > >> > >>> Hi All, > >>> > >>> I think , defining a standard JSON message format for communication in > the > >>> first place will > >>> solve the problem of projects inter dependency for a great extend. > Front > >>> end developer > >>> can assume that if he generate the correct JSON format message on their > >>> side then rest > >>> will be handle correctly. By the time the person who are dealing with > JSON > >>> thing need to > >>> handle it correctly. > >>> > >>> As Suresh mentioned we need to works as team while taking > responsibility of > >>> part of > >>> the master project. I would like to suggest for a IRC chanel to > Airavata > >>> then we can > >>> easily talk with each others through the mailing list as well as IRC > >>> channel too, to resolve > >>> problems arise in development phase. > >>> > >>> JSON <--> XML conversion is a huge part of this project because > successful > >>> of this whole > >>> project depend on this. Therefore It is necessary to has a solid > solution > >>> for this. As i explained > >>> in above we need to handle issues comes with this conversion like data > loss > >>> in bidirectional > >>> transformation. > >>> > >>> With this model we can replace whole Registry data format to JSON, > Registry > >>> will only deal with > >>> JSON data format and will send JSON Schema as application description > >>> instead of WSDL > >>> to Front end(web UI). When workflow interpreter or GFac need to talk to > >>> Registry, It will > >>> happen through JSON messages. To do that we need to convert XML > message to > >>> JSON > >>> in the GFac/Workflow Interpreter pre transport phase. > >>> > >>> I would be good to get responsibility of this whole JSON thing by one > >>> person. Like this we > >>> can divide responsibility of Workflow graph interpretation , Workflow > >>> composition and Workflow > >>> monitoring parts among us. In this case I would like to work on JSON > part. > >>> > >>> Cheers, > >>> Shameera. > >>> > >>> > >>> On Sun, Apr 28, 2013 at 10:38 AM, Subho Banerjee <[email protected] > >>>> wrote: > >>> > >>>> Hi Suresh, > >>>> > >>>> I think this decomposition might be a very good way to go ahead. We > can > >>>> apply to GSoC using these definitions and probably expand on the > scope of > >>>> each of the projects as time progresses. However, the only problem I > can > >>>> see is that from an implementation point of view, this might be a > little > >>>> difficult, because none of the frontend based projects will be able > test > >>>> themselves until the datamodel and the API is patched up to use JSON > (as > >>> we > >>>> had discussed earlier). > >>>> > >>>> I would like to suggest an alternate approach (which too has certain > >>>> problems) where, one project deals with multiple functional units. > Each > >>>> functional unit will try to solve a task, and will deal with the > >>> frontend, > >>>> the JSON bridge, the API as well as the data model. In this way, the > >>>> correctness of the code can be tested right away. Instead of waiting > for > >>>> another project to finish something before you can move ahead. The > >>> problem > >>>> here is the enumeration of all these functional units will be a > >>>> very difficult and time consuming task. > >>>> > >>>> Personally I am interested in working on the (frontend) application > >>>> registration and workflow composition interface as well as XML <--> > JSON > >>>> bridge which might come as a part of the workflow composition project > or > >>>> the data-models project. > >>>> > >>>> Could you also send us a template proposal for Airavata. > >>>> > >>>> Cheers, > >>>> Subho. > >>>> > >>>> > >>>> On Sun, Apr 28, 2013 at 12:06 AM, Suresh Marru <[email protected]> > >>> wrote: > >>>> > >>>>> Hi Guys, > >>>>> > >>>>> Does any one have any proposal on how to break the project into > smaller > >>>>> gsoc projects? > >>>>> > >>>>> I think you all should work in parallel and as a team. There may be > >>>>> projects two of you can implement the same idea. Example, the > workflow > >>>>> composer can be implemented in parallel by two students using > different > >>>>> libraries. But we need to make sure the JSON schema and workflow > model > >>>>> should be identical by both of them and so on. > >>>>> > >>>>> Here is one idea for the split (don't take it literally and propose > >>> other > >>>>> ideas as well): > >>>>> > >>>>> * A project fully focused on Airavata data models for application and > >>>>> workflows. So this person can work with rest of team to ensure a > common > >>>>> agreed upon JSON schema is drafted and works on the conversion as > >>> needed > >>>>> with existing XML Schema. > >>>>> * A project focused on the Airavata API and supporting the necessary > >>>>> functionality needed to construct the web based interfaces. This > >>> project > >>>>> basically have to understand the current Airavata Client API and > >>> existing > >>>>> service interfaces of internal components (the REST and SOAP > interfaces > >>>> of > >>>>> registry, interpreter, and messaging system -- GFac is invoked by > >>>>> interpreter so it is not exposed outside). > >>>>> * 2 projects on workflow composition interfaces themselves. These > >>>> projects > >>>>> have to interact with Airavata REST Service, parse the JSON messages, > >>>>> construct the graph, generate the workflow xml (XWF) at the client > or a > >>>>> intermediary service. > >>>>> * A project on workflow execution interfaces. This project has to > >>>>> understand the current workflow inputs and workflow execution context > >>> and > >>>>> workflow representation. > >>>>> * A project on workflow monitoring - this project has to understand > the > >>>>> workflow tracking schema and ws messenger services. > >>>>> > >>>>> I am not sure if this is a good way to split the projects, but I am > >>> just > >>>>> throwing an idea. Please don't look at them as separate projects. As > we > >>>>> have been discussing right from the beginning, these interleaved > tasks > >>>>> requires every one has to work on everything. > >>>>> > >>>>> Cheers, > >>>>> Suresh > >>>>> > >>>>> On Apr 27, 2013, at 1:49 PM, Andun Sameera <[email protected]> > wrote: > >>>>> > >>>>>> Hi, > >>>>>> > >>>>>> I will bring some comments which I made earlier and which will help > >>> in > >>>>> the > >>>>>> discussion. > >>>>>> > >>>>>> On Fri, Apr 26, 2013 at 2:09 AM, Vijayendra Grampurohit < > >>>>>> [email protected]> wrote: > >>>>>> > >>>>>>> Hi > >>>>>>> > >>>>>>> 1] Currently the workflow description has the graph representation > >>>> form > >>>>>>> xbaya . > >>>>>>> The data for the Workflow Inputs come from the Registry .This data > >>> is > >>>> in > >>>>>>> wsdl form which is stored > >>>>>>> in the Registry. > >>>>>>> There is also some metadata associated with wsdl . So that the > >>> client > >>>>>>> which has to bind the input values can use it. > >>>>>>> > >>>>>>> As we are discussing in the line of developing a browser based > >>>>>>> application. > >>>>>>> For constructing workflow we can make use of below graph library's > >>>>>>> http://jsplumbtoolkit.com/jquery/demo.html > >>>>>>> http://raphaeljs.com/ > >>>>>>> In our project( Openshift workflows) we had used jsplumb library > >>>>>>> which worked well with Angularjs . > >>>>>>> > >>>>>>> There are many more graph libraries which also can be explored. > >>>>>>> > >>>>>> > >>>>>> I have digged deep in to the source code via debugging to see how > the > >>>>> XBaya > >>>>>> create Workflows, persists them local file system, persists them in > >>>>>> registry. I understood following things, > >>>>>> > >>>>>> - All the workflow design happens on top of the > >>>>>> org.apache.airavata.xbaya.ui.graph.GraphCanvas > >>>>>> - The workflow data represented via the > >>>>>> org.apache.airavata.workflow.model.wf.Workflow ( I think this is the > >>>> DAG > >>>>>> representation which you are talking about in Airavata Workflow > >>>>>> Architecture in [1]) > >>>>>> - Local file system persistence, retrieving done via > >>>>>> org.apache.airavata.xbaya.core.generators.WorkflowFiler > >>>>>> - Registry based persistence, retrieving has done via the REST > client > >>>>>> org.apache.airavata.rest.clientUserWorkflowResourceClient > >>>>>> > >>>>>> So If I focus on the task of implementing Web based Workflow > Composer > >>>> for > >>>>>> Airavata, IMO we can reuse all three other than 1st one. 1st one > have > >>>> to > >>>>> be > >>>>>> replace with a JavaScript based implementation. When that handles > >>>> design > >>>>>> part all the other things related to workflow can be done reusing > the > >>>>>> others. If we use these or not, it is really valuable to understand > >>> the > >>>>>> existing way of doing the thing. > >>>>>> > >>>>>> I also looked at rapheljs and jsplumb earlier. I think they suites > >>> the > >>>>>> requirement and they comes under MIT license. > >>>>>> > >>>>>> > >>>>>>> 2] Currently the data that is stored in Registry is of the form > >>> wsdl. > >>>>>>> The workflow will use the data stored in the Registry . > >>>>>>> For the new workflow as there is a discussion going on what kind of > >>>> data > >>>>>>> are we > >>>>>>> going to store in registry (i.e wsdl or JSON ). This can be > worked > >>>>>>> accordingly. > >>>>>>> > >>>>>>> 3] Monitoring tool : I am thinking in terms of a > >>>>>>> browser plugin or a simple java script based web based monitoring > >>>> which > >>>>>>> will > >>>>>>> notify users on workflow progress in real time. > >>>>>>> This can be developed as a separate module . > >>>>>>> The Monitoring tool subscribes to a pre-specified topic to which > >>>>>>> Workflow Engine and GFac are publishing status notifications.Then > >>> the > >>>>>>> monitoring > >>>>>>> tool translates these messages and shows to to the user in the > front > >>>>> end. > >>>>>>> Please see the image inserted. > >>>>>>> > >>>>>>> [image: Inline image 1] > >>>>>>> > >>>>>>> What we can also do , When user is executing a very large workflow > >>>>>>> with hundreds of variables and doesn't want to track every thing, > >>> Then > >>>>> we > >>>>>>> can have a option to > >>>>>>> which allows customization in the monitoring tool . > >>>>>>> My point is we can have features in which user can monitor > >>>>>>> the variables or data he wants. > >>>>>>> > >>>>>>> Please correct me if I am wrong any where. > >>>>>>> > >>>>>>> Regards > >>>>>>> Vijayendra > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Thu, Apr 25, 2013 at 7:08 PM, Suresh Marru <[email protected]> > >>>>> wrote: > >>>>>>> > >>>>>>>> On Apr 23, 2013, at 8:01 AM, Suresh Marru <[email protected]> > >>> wrote: > >>>>>>>> > >>>>>>>>> Great discussion Shameera & Subho. > >>>>>>>>> > >>>>>>>>> Looks like you guys have a fair idea on what needs to be done, as > >>>> you > >>>>>>>> both state, a week or two into GSOC, you can narrow down on design > >>>> and > >>>>>>>> specifications and so forth. Given there are multiple students > >>>> tackling > >>>>>>>> these issues collectively, if there are hard decisions to make, I > >>>> would > >>>>>>>> suggest to try two approaches in parallel and quickly pick one > >>> after > >>>> a > >>>>>>>> proof of concept. > >>>>>>>> > >>>>>>>> Can you all discuss the workflow composition strategies related to > >>>> the > >>>>>>>> data model the same way we discussed application descriptions? > >>>>>>>> > >>>>>>>> We need to decompose the master project into smaller, well aligned > >>>>>>>> projects this weekend. So please start posting ideas on how to > >>>>> sub-divide > >>>>>>>> the projects. > >>>>>>>> > >>>>>>>> Suresh > >>>>>>>> > >>>>>>>>> > >>>>>>>>> Suresh > >>>>>>>>> > >>>>>>>>> On Apr 23, 2013, at 3:43 AM, Subho Banerjee <[email protected] > > > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Well exactly, as long as you can define standard way of > >>>>> communication. > >>>>>>>> That > >>>>>>>>>> is, you can define in advance what should be a string, array and > >>>> what > >>>>>>>>>> should be a integer etc. We have no problem. > >>>>>>>>>> > >>>>>>>>>> So, when you look at problems, with JSON <-> XML or the other > way > >>>>>>>> round, > >>>>>>>>>> they talk of the very general case (where you no nothing about > >>> the > >>>>>>>> data you > >>>>>>>>>> are converting other than it is valid XML/JSON). There are a > >>> myriad > >>>>> of > >>>>>>>>>> problems in that case, which you pointed out. > >>>>>>>>>> > >>>>>>>>>> But when there is standard, there is only one way of doing > >>> things, > >>>>> and > >>>>>>>> not > >>>>>>>>>> several. I think that is the way forward. So what I am proposing > >>> is > >>>>>>>> maybe > >>>>>>>>>> we all discuss and define this standard within the first week of > >>>> GSoC > >>>>>>>>>> starting and then actually move into coding. So as long as we > >>> work > >>>>>>>> with the > >>>>>>>>>> presumption that this will be done, we really dont have to worry > >>> a > >>>>> lot > >>>>>>>>>> about this. > >>>>>>>>>> > >>>>>>>>>> Cheers, > >>>>>>>>>> Subho. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Tue, Apr 23, 2013 at 11:52 AM, Shameera Rathnayaka < > >>>>>>>>>> [email protected]> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi, > >>>>>>>>>>> > >>>>>>>>>>> On Tue, Apr 23, 2013 at 2:25 AM, Subho Banerjee < > >>>>> [email protected]> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Some of these problems are very specific to what the XML is > >>>>>>>>>>> representing, > >>>>>>>>>>>> it might not be an actual problem in Airavata, > >>>>>>>>>>>> maybe some one more experienced with the codebase can point > >>> this > >>>>> out. > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> All issues pointed out in the paper is not directly valid to > our > >>>>>>>>>>> conversion, I didn't list the issues actually need to address > in > >>>>> this > >>>>>>>> case > >>>>>>>>>>> because thought it is worth to read that introduction part > which > >>>>>>>> explain > >>>>>>>>>>> the all the issues we have with this conversion and give us a > >>>> solid > >>>>>>>>>>> background of that. > >>>>>>>>>>> > >>>>>>>>>>>> 1. Anonymous values, Arrays, Implicit Typing, Character sets > >>> -- I > >>>>>>>>>>> really > >>>>>>>>>>>> dont see these as problems, as long as you can agree that all > >>>> parts > >>>>>>>> of > >>>>>>>>>>>> airavata will treat the JSON in a standard (probably we have > to > >>>>>>>> define > >>>>>>>>>>>> this) way. > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> The issue with JSON array only comes when we try to convert XML > >>> to > >>>>>>>> JSON not > >>>>>>>>>>> the other way. If we map with JSON, inputparameters and > >>>>>>>> outputparameters in > >>>>>>>>>>> the ServiceDescription.xsd will map with JSON Arrays. Therefore > >>> we > >>>>>>>> need to > >>>>>>>>>>> solve this issue. > >>>>>>>>>>> > >>>>>>>>>>> JSON XML JSON > >>>>>>>>>>> {"inputs":["test"]} --> <inputs>test<inputs> --> > >>>>> {"inputs":["test"]} > >>>>>>>> // > >>>>>>>>>>> correct one > >>>>>>>>>>> --> {"inputs":"test"} // incorrect > >>> one > >>>>>>>>>>> > >>>>>>>>>>> 2. Namespaces, Processing Instructions -- Is this required? > >>>>>>>>>>> > >>>>>>>>>>>> Are separate namespaces used in Airavata? Only place I can see > >>>> this > >>>>>>>>>>>> being > >>>>>>>>>>>> used is probably in the WSDL, but if we can agree on another > >>> way > >>>>>>>>>>>> of communicating registered applications' I/O parameters to > the > >>>>>>>> front > >>>>>>>>>>>> end > >>>>>>>>>>>> (JSON based), then maybe we can work around this (minor) > >>> problem. > >>>>>>>> Are > >>>>>>>>>>>> custom processing instructions to the Xbaya XML parse even > >>> used? > >>>>>>>>>>>> 3. Attributes -- Again, this can be fixed easily > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Yes,attributes convertion will not be a big issues we can solve > >>>> it. > >>>>> As > >>>>>>>>>>> Lahiru mentioned in Hangout session namesapce handling is not a > >>>> big > >>>>>>>> issue > >>>>>>>>>>> with Airavata. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> <array name="abc"> > >>>>>>>>>>>> <element>1</element> > >>>>>>>>>>>> <element>2</element> > >>>>>>>>>>>> <element>3</element> > >>>>>>>>>>>> <element>4</element> > >>>>>>>>>>>> </array> > >>>>>>>>>>>> > >>>>>>>>>>>> Can become > >>>>>>>>>>>> > >>>>>>>>>>>> { > >>>>>>>>>>>> > >>>>>>>>>>>> abc : ['1', '2', '3', '4'] > >>>>>>>>>>>> > >>>>>>>>>>>> } > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> With this example it show us we need to change the XML message > >>>>> format > >>>>>>>> of > >>>>>>>>>>> server side, which require to change the all schemas, If we are > >>>>> going > >>>>>>>> to > >>>>>>>>>>> change the schemas then we need to change the way it process it > >>> in > >>>>>>>> Ariavara > >>>>>>>>>>> core. We have dropped our initial major requirement, which is > >>> keep > >>>>> the > >>>>>>>>>>> Airavata Server side as it is. > >>>>>>>>>>> > >>>>>>>>>>> with this conversion we only deal with json strings, yes we can > >>>> send > >>>>>>>> JSON > >>>>>>>>>>> request with other formats supported by JSON like boolen, null, > >>>>> Number > >>>>>>>>>>> etc.. But there is no way to get the same JSON from XML as XML > >>>> only > >>>>>>>> deal > >>>>>>>>>>> only with Strings. I think it is good if we can consume a this > >>>>>>>> features > >>>>>>>>>>> with JSON. > >>>>>>>>>>> > >>>>>>>>>>> let say i need to send a integer or float to the server using > >>> JSON > >>>>>>>> then > >>>>>>>>>>> proper way is to send {"<name>":123.45} this will works fine > but > >>>>>>>> problem is > >>>>>>>>>>> how we get the same output ? > >>>>>>>>>>> > >>>>>>>>>>> Thanks, > >>>>>>>>>>> Shameera. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Cheers, > >>>>>>>>>>>> Subho. > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Best Regards, > >>>>>>>>>>> Shameera Rathnayaka. > >>>>>>>>>>> > >>>>>>>>>>> Blog : http://shameerarathnayaka.blogspot.com/ > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>>> Thanks! > >>>>>> > >>>>>> [1] - http://people.apache.org/~smarru/papers/airavata-gce11.pdf > >>>>>> -- > >>>>>> Regards > >>>>>> Andun S.L. Gunawardana > >>>>>> Undergraduate > >>>>>> Department of Computer Science And Engineering > >>>>>> University of Moratuwa > >>>>>> Sri Lanka > >>>>>> > >>>>>> Blog - http://www.insightforfuture.blogspot.com/ > >>>>>> LinkedIn - > >>>> http://www.linkedin.com/pub/andun-s-l-gunawardana/34/646/703 > >>>>>> Twitter -http://twitter.com/AndunSLG > >>>>> > >>>>> > >>>> > >>> > >>> > >>> > >>> -- > >>> Best Regards, > >>> Shameera Rathnayaka. > >>> > >>> email: shameera AT apache.org , shameerainfo AT gmail.com > >>> Blog : http://shameerarathnayaka.blogspot.com/ > >>> > > > >
