Hi Suresh I would like to concentrate on Workflow monitoring .
Regards Vijayendra On Sun, Apr 28, 2013 at 12:09 PM, Shameera Rathnayaka < [email protected]> wrote: > Hi All, > > I think , defining a standard JSON message format for communication in the > first place will > solve the problem of projects inter dependency for a great extend. Front > end developer > can assume that if he generate the correct JSON format message on their > side then rest > will be handle correctly. By the time the person who are dealing with JSON > thing need to > handle it correctly. > > As Suresh mentioned we need to works as team while taking responsibility of > part of > the master project. I would like to suggest for a IRC chanel to Airavata > then we can > easily talk with each others through the mailing list as well as IRC > channel too, to resolve > problems arise in development phase. > > JSON <--> XML conversion is a huge part of this project because successful > of this whole > project depend on this. Therefore It is necessary to has a solid solution > for this. As i explained > in above we need to handle issues comes with this conversion like data loss > in bidirectional > transformation. > > With this model we can replace whole Registry data format to JSON, Registry > will only deal with > JSON data format and will send JSON Schema as application description > instead of WSDL > to Front end(web UI). When workflow interpreter or GFac need to talk to > Registry, It will > happen through JSON messages. To do that we need to convert XML message to > JSON > in the GFac/Workflow Interpreter pre transport phase. > > I would be good to get responsibility of this whole JSON thing by one > person. Like this we > can divide responsibility of Workflow graph interpretation , Workflow > composition and Workflow > monitoring parts among us. In this case I would like to work on JSON part. > > Cheers, > Shameera. > > > On Sun, Apr 28, 2013 at 10:38 AM, Subho Banerjee <[email protected] > >wrote: > > > Hi Suresh, > > > > I think this decomposition might be a very good way to go ahead. We can > > apply to GSoC using these definitions and probably expand on the scope of > > each of the projects as time progresses. However, the only problem I can > > see is that from an implementation point of view, this might be a little > > difficult, because none of the frontend based projects will be able test > > themselves until the datamodel and the API is patched up to use JSON (as > we > > had discussed earlier). > > > > I would like to suggest an alternate approach (which too has certain > > problems) where, one project deals with multiple functional units. Each > > functional unit will try to solve a task, and will deal with the > frontend, > > the JSON bridge, the API as well as the data model. In this way, the > > correctness of the code can be tested right away. Instead of waiting for > > another project to finish something before you can move ahead. The > problem > > here is the enumeration of all these functional units will be a > > very difficult and time consuming task. > > > > Personally I am interested in working on the (frontend) application > > registration and workflow composition interface as well as XML <--> JSON > > bridge which might come as a part of the workflow composition project or > > the data-models project. > > > > Could you also send us a template proposal for Airavata. > > > > Cheers, > > Subho. > > > > > > On Sun, Apr 28, 2013 at 12:06 AM, Suresh Marru <[email protected]> > wrote: > > > > > Hi Guys, > > > > > > Does any one have any proposal on how to break the project into smaller > > > gsoc projects? > > > > > > I think you all should work in parallel and as a team. There may be > > > projects two of you can implement the same idea. Example, the workflow > > > composer can be implemented in parallel by two students using different > > > libraries. But we need to make sure the JSON schema and workflow model > > > should be identical by both of them and so on. > > > > > > Here is one idea for the split (don't take it literally and propose > other > > > ideas as well): > > > > > > * A project fully focused on Airavata data models for application and > > > workflows. So this person can work with rest of team to ensure a common > > > agreed upon JSON schema is drafted and works on the conversion as > needed > > > with existing XML Schema. > > > * A project focused on the Airavata API and supporting the necessary > > > functionality needed to construct the web based interfaces. This > project > > > basically have to understand the current Airavata Client API and > existing > > > service interfaces of internal components (the REST and SOAP interfaces > > of > > > registry, interpreter, and messaging system -- GFac is invoked by > > > interpreter so it is not exposed outside). > > > * 2 projects on workflow composition interfaces themselves. These > > projects > > > have to interact with Airavata REST Service, parse the JSON messages, > > > construct the graph, generate the workflow xml (XWF) at the client or a > > > intermediary service. > > > * A project on workflow execution interfaces. This project has to > > > understand the current workflow inputs and workflow execution context > and > > > workflow representation. > > > * A project on workflow monitoring - this project has to understand the > > > workflow tracking schema and ws messenger services. > > > > > > I am not sure if this is a good way to split the projects, but I am > just > > > throwing an idea. Please don't look at them as separate projects. As we > > > have been discussing right from the beginning, these interleaved tasks > > > requires every one has to work on everything. > > > > > > Cheers, > > > Suresh > > > > > > On Apr 27, 2013, at 1:49 PM, Andun Sameera <[email protected]> wrote: > > > > > > > Hi, > > > > > > > > I will bring some comments which I made earlier and which will help > in > > > the > > > > discussion. > > > > > > > > On Fri, Apr 26, 2013 at 2:09 AM, Vijayendra Grampurohit < > > > > [email protected]> wrote: > > > > > > > >> Hi > > > >> > > > >> 1] Currently the workflow description has the graph representation > > form > > > >> xbaya . > > > >> The data for the Workflow Inputs come from the Registry .This data > is > > in > > > >> wsdl form which is stored > > > >> in the Registry. > > > >> There is also some metadata associated with wsdl . So that the > client > > > >> which has to bind the input values can use it. > > > >> > > > >> As we are discussing in the line of developing a browser based > > > >> application. > > > >> For constructing workflow we can make use of below graph library's > > > >> http://jsplumbtoolkit.com/jquery/demo.html > > > >> http://raphaeljs.com/ > > > >> In our project( Openshift workflows) we had used jsplumb library > > > >> which worked well with Angularjs . > > > >> > > > >> There are many more graph libraries which also can be explored. > > > >> > > > > > > > > I have digged deep in to the source code via debugging to see how the > > > XBaya > > > > create Workflows, persists them local file system, persists them in > > > > registry. I understood following things, > > > > > > > > - All the workflow design happens on top of the > > > > org.apache.airavata.xbaya.ui.graph.GraphCanvas > > > > - The workflow data represented via the > > > > org.apache.airavata.workflow.model.wf.Workflow ( I think this is the > > DAG > > > > representation which you are talking about in Airavata Workflow > > > > Architecture in [1]) > > > > - Local file system persistence, retrieving done via > > > > org.apache.airavata.xbaya.core.generators.WorkflowFiler > > > > - Registry based persistence, retrieving has done via the REST client > > > > org.apache.airavata.rest.clientUserWorkflowResourceClient > > > > > > > > So If I focus on the task of implementing Web based Workflow Composer > > for > > > > Airavata, IMO we can reuse all three other than 1st one. 1st one have > > to > > > be > > > > replace with a JavaScript based implementation. When that handles > > design > > > > part all the other things related to workflow can be done reusing the > > > > others. If we use these or not, it is really valuable to understand > the > > > > existing way of doing the thing. > > > > > > > > I also looked at rapheljs and jsplumb earlier. I think they suites > the > > > > requirement and they comes under MIT license. > > > > > > > > > > > >> 2] Currently the data that is stored in Registry is of the form > wsdl. > > > >> The workflow will use the data stored in the Registry . > > > >> For the new workflow as there is a discussion going on what kind of > > data > > > >> are we > > > >> going to store in registry (i.e wsdl or JSON ). This can be worked > > > >> accordingly. > > > >> > > > >> 3] Monitoring tool : I am thinking in terms of a > > > >> browser plugin or a simple java script based web based monitoring > > which > > > >> will > > > >> notify users on workflow progress in real time. > > > >> This can be developed as a separate module . > > > >> The Monitoring tool subscribes to a pre-specified topic to which > > > >> Workflow Engine and GFac are publishing status notifications.Then > the > > > >> monitoring > > > >> tool translates these messages and shows to to the user in the front > > > end. > > > >> Please see the image inserted. > > > >> > > > >> [image: Inline image 1] > > > >> > > > >> What we can also do , When user is executing a very large workflow > > > >> with hundreds of variables and doesn't want to track every thing, > Then > > > we > > > >> can have a option to > > > >> which allows customization in the monitoring tool . > > > >> My point is we can have features in which user can monitor > > > >> the variables or data he wants. > > > >> > > > >> Please correct me if I am wrong any where. > > > >> > > > >> Regards > > > >> Vijayendra > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> On Thu, Apr 25, 2013 at 7:08 PM, Suresh Marru <[email protected]> > > > wrote: > > > >> > > > >>> On Apr 23, 2013, at 8:01 AM, Suresh Marru <[email protected]> > wrote: > > > >>> > > > >>>> Great discussion Shameera & Subho. > > > >>>> > > > >>>> Looks like you guys have a fair idea on what needs to be done, as > > you > > > >>> both state, a week or two into GSOC, you can narrow down on design > > and > > > >>> specifications and so forth. Given there are multiple students > > tackling > > > >>> these issues collectively, if there are hard decisions to make, I > > would > > > >>> suggest to try two approaches in parallel and quickly pick one > after > > a > > > >>> proof of concept. > > > >>> > > > >>> Can you all discuss the workflow composition strategies related to > > the > > > >>> data model the same way we discussed application descriptions? > > > >>> > > > >>> We need to decompose the master project into smaller, well aligned > > > >>> projects this weekend. So please start posting ideas on how to > > > sub-divide > > > >>> the projects. > > > >>> > > > >>> Suresh > > > >>> > > > >>>> > > > >>>> Suresh > > > >>>> > > > >>>> On Apr 23, 2013, at 3:43 AM, Subho Banerjee <[email protected]> > > > >>> wrote: > > > >>>> > > > >>>>> Well exactly, as long as you can define standard way of > > > communication. > > > >>> That > > > >>>>> is, you can define in advance what should be a string, array and > > what > > > >>>>> should be a integer etc. We have no problem. > > > >>>>> > > > >>>>> So, when you look at problems, with JSON <-> XML or the other way > > > >>> round, > > > >>>>> they talk of the very general case (where you no nothing about > the > > > >>> data you > > > >>>>> are converting other than it is valid XML/JSON). There are a > myriad > > > of > > > >>>>> problems in that case, which you pointed out. > > > >>>>> > > > >>>>> But when there is standard, there is only one way of doing > things, > > > and > > > >>> not > > > >>>>> several. I think that is the way forward. So what I am proposing > is > > > >>> maybe > > > >>>>> we all discuss and define this standard within the first week of > > GSoC > > > >>>>> starting and then actually move into coding. So as long as we > work > > > >>> with the > > > >>>>> presumption that this will be done, we really dont have to worry > a > > > lot > > > >>>>> about this. > > > >>>>> > > > >>>>> Cheers, > > > >>>>> Subho. > > > >>>>> > > > >>>>> > > > >>>>> On Tue, Apr 23, 2013 at 11:52 AM, Shameera Rathnayaka < > > > >>>>> [email protected]> wrote: > > > >>>>> > > > >>>>>> Hi, > > > >>>>>> > > > >>>>>> On Tue, Apr 23, 2013 at 2:25 AM, Subho Banerjee < > > > [email protected]> > > > >>>>>> wrote: > > > >>>>>> > > > >>>>>>> Some of these problems are very specific to what the XML is > > > >>>>>> representing, > > > >>>>>>> it might not be an actual problem in Airavata, > > > >>>>>>> maybe some one more experienced with the codebase can point > this > > > out. > > > >>>>>>> > > > >>>>>> > > > >>>>>> All issues pointed out in the paper is not directly valid to our > > > >>>>>> conversion, I didn't list the issues actually need to address in > > > this > > > >>> case > > > >>>>>> because thought it is worth to read that introduction part which > > > >>> explain > > > >>>>>> the all the issues we have with this conversion and give us a > > solid > > > >>>>>> background of that. > > > >>>>>> > > > >>>>>>> 1. Anonymous values, Arrays, Implicit Typing, Character sets > -- I > > > >>>>>> really > > > >>>>>>> dont see these as problems, as long as you can agree that all > > parts > > > >>> of > > > >>>>>>> airavata will treat the JSON in a standard (probably we have to > > > >>> define > > > >>>>>>> this) way. > > > >>>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> The issue with JSON array only comes when we try to convert XML > to > > > >>> JSON not > > > >>>>>> the other way. If we map with JSON, inputparameters and > > > >>> outputparameters in > > > >>>>>> the ServiceDescription.xsd will map with JSON Arrays. Therefore > we > > > >>> need to > > > >>>>>> solve this issue. > > > >>>>>> > > > >>>>>> JSON XML JSON > > > >>>>>> {"inputs":["test"]} --> <inputs>test<inputs> --> > > > {"inputs":["test"]} > > > >>> // > > > >>>>>> correct one > > > >>>>>> --> {"inputs":"test"} // incorrect > one > > > >>>>>> > > > >>>>>> 2. Namespaces, Processing Instructions -- Is this required? > > > >>>>>> > > > >>>>>>> Are separate namespaces used in Airavata? Only place I can see > > this > > > >>>>>>> being > > > >>>>>>> used is probably in the WSDL, but if we can agree on another > way > > > >>>>>>> of communicating registered applications' I/O parameters to the > > > >>> front > > > >>>>>>> end > > > >>>>>>> (JSON based), then maybe we can work around this (minor) > problem. > > > >>> Are > > > >>>>>>> custom processing instructions to the Xbaya XML parse even > used? > > > >>>>>>> 3. Attributes -- Again, this can be fixed easily > > > >>>>>>> > > > >>>>>> > > > >>>>>> Yes,attributes convertion will not be a big issues we can solve > > it. > > > As > > > >>>>>> Lahiru mentioned in Hangout session namesapce handling is not a > > big > > > >>> issue > > > >>>>>> with Airavata. > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>> > > > >>>>>>> <array name="abc"> > > > >>>>>>> <element>1</element> > > > >>>>>>> <element>2</element> > > > >>>>>>> <element>3</element> > > > >>>>>>> <element>4</element> > > > >>>>>>> </array> > > > >>>>>>> > > > >>>>>>> Can become > > > >>>>>>> > > > >>>>>>> { > > > >>>>>>> > > > >>>>>>> abc : ['1', '2', '3', '4'] > > > >>>>>>> > > > >>>>>>> } > > > >>>>>>> > > > >>>>>> > > > >>>>>> With this example it show us we need to change the XML message > > > format > > > >>> of > > > >>>>>> server side, which require to change the all schemas, If we are > > > going > > > >>> to > > > >>>>>> change the schemas then we need to change the way it process it > in > > > >>> Ariavara > > > >>>>>> core. We have dropped our initial major requirement, which is > keep > > > the > > > >>>>>> Airavata Server side as it is. > > > >>>>>> > > > >>>>>> with this conversion we only deal with json strings, yes we can > > send > > > >>> JSON > > > >>>>>> request with other formats supported by JSON like boolen, null, > > > Number > > > >>>>>> etc.. But there is no way to get the same JSON from XML as XML > > only > > > >>> deal > > > >>>>>> only with Strings. I think it is good if we can consume a this > > > >>> features > > > >>>>>> with JSON. > > > >>>>>> > > > >>>>>> let say i need to send a integer or float to the server using > JSON > > > >>> then > > > >>>>>> proper way is to send {"<name>":123.45} this will works fine but > > > >>> problem is > > > >>>>>> how we get the same output ? > > > >>>>>> > > > >>>>>> Thanks, > > > >>>>>> Shameera. > > > >>>>>> > > > >>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> Cheers, > > > >>>>>>> Subho. > > > >>>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> -- > > > >>>>>> Best Regards, > > > >>>>>> Shameera Rathnayaka. > > > >>>>>> > > > >>>>>> Blog : http://shameerarathnayaka.blogspot.com/ > > > >>>>>> > > > >>>> > > > >>> > > > >>> > > > >> > > > > > > > > Thanks! > > > > > > > > [1] - http://people.apache.org/~smarru/papers/airavata-gce11.pdf > > > > -- > > > > Regards > > > > Andun S.L. Gunawardana > > > > Undergraduate > > > > Department of Computer Science And Engineering > > > > University of Moratuwa > > > > Sri Lanka > > > > > > > > Blog - http://www.insightforfuture.blogspot.com/ > > > > LinkedIn - > > http://www.linkedin.com/pub/andun-s-l-gunawardana/34/646/703 > > > > Twitter -http://twitter.com/AndunSLG > > > > > > > > > > > > -- > Best Regards, > Shameera Rathnayaka. > > email: shameera AT apache.org , shameerainfo AT gmail.com > Blog : http://shameerarathnayaka.blogspot.com/ >
