Hi Suresh, I also agree with your project decomposition. I have following comments,
On Sun, Apr 28, 2013 at 12:06 AM, Suresh Marru <[email protected]> wrote: > Hi Guys, > > Does any one have any proposal on how to break the project into smaller > gsoc projects? > > I think you all should work in parallel and as a team. There may be > projects two of you can implement the same idea. Example, the workflow > composer can be implemented in parallel by two students using different > libraries. But we need to make sure the JSON schema and workflow model > should be identical by both of them and so on. > > Here is one idea for the split (don't take it literally and propose other > ideas as well): > > * A project fully focused on Airavata data models for application and > workflows. So this person can work with rest of team to ensure a common > agreed upon JSON schema is drafted and works on the conversion as needed > with existing XML Schema. > * A project focused on the Airavata API and supporting the necessary > functionality needed to construct the web based interfaces. This project > basically have to understand the current Airavata Client API and existing > service interfaces of internal components (the REST and SOAP interfaces of > registry, interpreter, and messaging system -- GFac is invoked by > interpreter so it is not exposed outside). > * 2 projects on workflow composition interfaces themselves. These projects > have to interact with Airavata REST Service, parse the JSON messages, > construct the graph, generate the workflow xml (XWF) at the client or a > intermediary service. > There have to be a place where Application Developers can register there applications. I think these two project have to focus about that implementation also. * A project on workflow execution interfaces. This project has to > understand the current workflow inputs and workflow execution context and > workflow representation. > * A project on workflow monitoring - this project has to understand the > workflow tracking schema and ws messenger services. > > I am not sure if this is a good way to split the projects, but I am just > throwing an idea. Please don't look at them as separate projects. As we > have been discussing right from the beginning, these interleaved tasks > requires every one has to work on everything. > > Cheers, > Suresh > > On Apr 27, 2013, at 1:49 PM, Andun Sameera <[email protected]> wrote: > > > Hi, > > > > I will bring some comments which I made earlier and which will help in > the > > discussion. > > > > On Fri, Apr 26, 2013 at 2:09 AM, Vijayendra Grampurohit < > > [email protected]> wrote: > > > >> Hi > >> > >> 1] Currently the workflow description has the graph representation form > >> xbaya . > >> The data for the Workflow Inputs come from the Registry .This data is in > >> wsdl form which is stored > >> in the Registry. > >> There is also some metadata associated with wsdl . So that the client > >> which has to bind the input values can use it. > >> > >> As we are discussing in the line of developing a browser based > >> application. > >> For constructing workflow we can make use of below graph library's > >> http://jsplumbtoolkit.com/jquery/demo.html > >> http://raphaeljs.com/ > >> In our project( Openshift workflows) we had used jsplumb library > >> which worked well with Angularjs . > >> > >> There are many more graph libraries which also can be explored. > >> > > > > I have digged deep in to the source code via debugging to see how the > XBaya > > create Workflows, persists them local file system, persists them in > > registry. I understood following things, > > > > - All the workflow design happens on top of the > > org.apache.airavata.xbaya.ui.graph.GraphCanvas > > - The workflow data represented via the > > org.apache.airavata.workflow.model.wf.Workflow ( I think this is the DAG > > representation which you are talking about in Airavata Workflow > > Architecture in [1]) > > - Local file system persistence, retrieving done via > > org.apache.airavata.xbaya.core.generators.WorkflowFiler > > - Registry based persistence, retrieving has done via the REST client > > org.apache.airavata.rest.clientUserWorkflowResourceClient > > > > So If I focus on the task of implementing Web based Workflow Composer for > > Airavata, IMO we can reuse all three other than 1st one. 1st one have to > be > > replace with a JavaScript based implementation. When that handles design > > part all the other things related to workflow can be done reusing the > > others. If we use these or not, it is really valuable to understand the > > existing way of doing the thing. > > > > I also looked at rapheljs and jsplumb earlier. I think they suites the > > requirement and they comes under MIT license. > > > > > >> 2] Currently the data that is stored in Registry is of the form wsdl. > >> The workflow will use the data stored in the Registry . > >> For the new workflow as there is a discussion going on what kind of data > >> are we > >> going to store in registry (i.e wsdl or JSON ). This can be worked > >> accordingly. > >> > >> 3] Monitoring tool : I am thinking in terms of a > >> browser plugin or a simple java script based web based monitoring which > >> will > >> notify users on workflow progress in real time. > >> This can be developed as a separate module . > >> The Monitoring tool subscribes to a pre-specified topic to which > >> Workflow Engine and GFac are publishing status notifications.Then the > >> monitoring > >> tool translates these messages and shows to to the user in the front > end. > >> Please see the image inserted. > >> > >> [image: Inline image 1] > >> > >> What we can also do , When user is executing a very large workflow > >> with hundreds of variables and doesn't want to track every thing, Then > we > >> can have a option to > >> which allows customization in the monitoring tool . > >> My point is we can have features in which user can monitor > >> the variables or data he wants. > >> > >> Please correct me if I am wrong any where. > >> > >> Regards > >> Vijayendra > >> > >> > >> > >> > >> > >> On Thu, Apr 25, 2013 at 7:08 PM, Suresh Marru <[email protected]> > wrote: > >> > >>> On Apr 23, 2013, at 8:01 AM, Suresh Marru <[email protected]> wrote: > >>> > >>>> Great discussion Shameera & Subho. > >>>> > >>>> Looks like you guys have a fair idea on what needs to be done, as you > >>> both state, a week or two into GSOC, you can narrow down on design and > >>> specifications and so forth. Given there are multiple students tackling > >>> these issues collectively, if there are hard decisions to make, I would > >>> suggest to try two approaches in parallel and quickly pick one after a > >>> proof of concept. > >>> > >>> Can you all discuss the workflow composition strategies related to the > >>> data model the same way we discussed application descriptions? > >>> > >>> We need to decompose the master project into smaller, well aligned > >>> projects this weekend. So please start posting ideas on how to > sub-divide > >>> the projects. > >>> > >>> Suresh > >>> > >>>> > >>>> Suresh > >>>> > >>>> On Apr 23, 2013, at 3:43 AM, Subho Banerjee <[email protected]> > >>> wrote: > >>>> > >>>>> Well exactly, as long as you can define standard way of > communication. > >>> That > >>>>> is, you can define in advance what should be a string, array and what > >>>>> should be a integer etc. We have no problem. > >>>>> > >>>>> So, when you look at problems, with JSON <-> XML or the other way > >>> round, > >>>>> they talk of the very general case (where you no nothing about the > >>> data you > >>>>> are converting other than it is valid XML/JSON). There are a myriad > of > >>>>> problems in that case, which you pointed out. > >>>>> > >>>>> But when there is standard, there is only one way of doing things, > and > >>> not > >>>>> several. I think that is the way forward. So what I am proposing is > >>> maybe > >>>>> we all discuss and define this standard within the first week of GSoC > >>>>> starting and then actually move into coding. So as long as we work > >>> with the > >>>>> presumption that this will be done, we really dont have to worry a > lot > >>>>> about this. > >>>>> > >>>>> Cheers, > >>>>> Subho. > >>>>> > >>>>> > >>>>> On Tue, Apr 23, 2013 at 11:52 AM, Shameera Rathnayaka < > >>>>> [email protected]> wrote: > >>>>> > >>>>>> Hi, > >>>>>> > >>>>>> On Tue, Apr 23, 2013 at 2:25 AM, Subho Banerjee < > [email protected]> > >>>>>> wrote: > >>>>>> > >>>>>>> Some of these problems are very specific to what the XML is > >>>>>> representing, > >>>>>>> it might not be an actual problem in Airavata, > >>>>>>> maybe some one more experienced with the codebase can point this > out. > >>>>>>> > >>>>>> > >>>>>> All issues pointed out in the paper is not directly valid to our > >>>>>> conversion, I didn't list the issues actually need to address in > this > >>> case > >>>>>> because thought it is worth to read that introduction part which > >>> explain > >>>>>> the all the issues we have with this conversion and give us a solid > >>>>>> background of that. > >>>>>> > >>>>>>> 1. Anonymous values, Arrays, Implicit Typing, Character sets -- I > >>>>>> really > >>>>>>> dont see these as problems, as long as you can agree that all parts > >>> of > >>>>>>> airavata will treat the JSON in a standard (probably we have to > >>> define > >>>>>>> this) way. > >>>>>>> > >>>>>> > >>>>>> > >>>>>> The issue with JSON array only comes when we try to convert XML to > >>> JSON not > >>>>>> the other way. If we map with JSON, inputparameters and > >>> outputparameters in > >>>>>> the ServiceDescription.xsd will map with JSON Arrays. Therefore we > >>> need to > >>>>>> solve this issue. > >>>>>> > >>>>>> JSON XML JSON > >>>>>> {"inputs":["test"]} --> <inputs>test<inputs> --> > {"inputs":["test"]} > >>> // > >>>>>> correct one > >>>>>> --> {"inputs":"test"} // incorrect one > >>>>>> > >>>>>> 2. Namespaces, Processing Instructions -- Is this required? > >>>>>> > >>>>>>> Are separate namespaces used in Airavata? Only place I can see this > >>>>>>> being > >>>>>>> used is probably in the WSDL, but if we can agree on another way > >>>>>>> of communicating registered applications' I/O parameters to the > >>> front > >>>>>>> end > >>>>>>> (JSON based), then maybe we can work around this (minor) problem. > >>> Are > >>>>>>> custom processing instructions to the Xbaya XML parse even used? > >>>>>>> 3. Attributes -- Again, this can be fixed easily > >>>>>>> > >>>>>> > >>>>>> Yes,attributes convertion will not be a big issues we can solve it. > As > >>>>>> Lahiru mentioned in Hangout session namesapce handling is not a big > >>> issue > >>>>>> with Airavata. > >>>>>> > >>>>>> > >>>>>> > >>>>>>> > >>>>>>> <array name="abc"> > >>>>>>> <element>1</element> > >>>>>>> <element>2</element> > >>>>>>> <element>3</element> > >>>>>>> <element>4</element> > >>>>>>> </array> > >>>>>>> > >>>>>>> Can become > >>>>>>> > >>>>>>> { > >>>>>>> > >>>>>>> abc : ['1', '2', '3', '4'] > >>>>>>> > >>>>>>> } > >>>>>>> > >>>>>> > >>>>>> With this example it show us we need to change the XML message > format > >>> of > >>>>>> server side, which require to change the all schemas, If we are > going > >>> to > >>>>>> change the schemas then we need to change the way it process it in > >>> Ariavara > >>>>>> core. We have dropped our initial major requirement, which is keep > the > >>>>>> Airavata Server side as it is. > >>>>>> > >>>>>> with this conversion we only deal with json strings, yes we can send > >>> JSON > >>>>>> request with other formats supported by JSON like boolen, null, > Number > >>>>>> etc.. But there is no way to get the same JSON from XML as XML only > >>> deal > >>>>>> only with Strings. I think it is good if we can consume a this > >>> features > >>>>>> with JSON. > >>>>>> > >>>>>> let say i need to send a integer or float to the server using JSON > >>> then > >>>>>> proper way is to send {"<name>":123.45} this will works fine but > >>> problem is > >>>>>> how we get the same output ? > >>>>>> > >>>>>> Thanks, > >>>>>> Shameera. > >>>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Cheers, > >>>>>>> Subho. > >>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Best Regards, > >>>>>> Shameera Rathnayaka. > >>>>>> > >>>>>> Blog : http://shameerarathnayaka.blogspot.com/ > >>>>>> > >>>> > >>> > >>> > >> > > > > Thanks! > > > > [1] - http://people.apache.org/~smarru/papers/airavata-gce11.pdf > > -- > > Regards > > Andun S.L. Gunawardana > > Undergraduate > > Department of Computer Science And Engineering > > University of Moratuwa > > Sri Lanka > > > > Blog - http://www.insightforfuture.blogspot.com/ > > LinkedIn - http://www.linkedin.com/pub/andun-s-l-gunawardana/34/646/703 > > Twitter -http://twitter.com/AndunSLG > > Thanks! -- Regards Andun S.L. Gunawardana Undergraduate Department of Computer Science And Engineering University of Moratuwa Sri Lanka Blog - http://www.insightforfuture.blogspot.com/ LinkedIn - http://www.linkedin.com/pub/andun-s-l-gunawardana/34/646/703 Twitter -http://twitter.com/AndunSLG
