Thanks, Eroma. For Step 1, we should also have a programmatic (API) way to create workflows. For questions 1 and 2, I think the answer is no to both. Applications in the catalog are available individually and as part of workflows. #3 is a good point. #4 is part of the larger "delete" problem that we need to discuss. I think for #5 we have one or the other (and I personally think single applications are just special cases of workflows).
Marlon On 6/9/14 11:35 AM, Eroma Abeysinghe wrote: > Hi Devs, > > I was thinking about workflows from system requirement point of view and > IMO app catalog should cover below. > (All given might be taken-care-of already; if so just ignore). > > When it comes to WFs (workflows) within App catalog we should be able to; > > 1. Create Workflows (More like adding the WF in to App catalog; as > actual WF creation is in XBAYA) > - When creating we should be able to give a name, description, ID > (generated) for the workflow. This would help with searing for > WFs as well > as searching for experiments using workflow ID, name, etc.... > - WFs can have a single node or multiple nodes. At least one node has > to have an application to execute. WFs can have one or many applications > - In WFs an output from one node can be a input to the next node > 2. Publish & Un-publish Workflows > - We need to have the option of publishing and un-publishing them. > This way we can keep the WFs un-published till they are ready > and then make > them available. > 3. Edit Workflows > - We should be able to add remove nodes, applications from workflows. > 4. Clone Workflows > - This is to assist with workflow creation. If we could clone it can > save time required to create WFs from scratch. Once cloned user should > be > able to modify the new WF. > 5. Import Workflows > - Importing from other gateways or an outside file. (Not so sure how > the importing works when it comes to XBAYA. If its imported from another > gateway the WF is already created using XBAYA. I guess then its matter > of > having it inserted into catalog. How would the import work if > the WF is not > even created in XBAYA)? > 6. Search Workflows > - We should be able to search using > - Workflow name > - Workflow description > - Application (we give an application s the key and all workflows > using the given app will get listed) > - Searching for WFs giving experiment details such as exp name, > exp description, exp status > - Created user > - ... any more? > 7. Delete Workflows > - This is to completely delete the workflow from app catalog. We may > not need this if we can un-publish > > Few questions > > 1. Can we have applications created only to be used in Workflows? > meaning we cannot use them individually in experiments. > 2. Can we make an application available in workflows but unavailable for > individual use at a given time? > 3. Currently when we search for experiments, we dont say whether we are > looking for WFs or not. but in future this may be required > 4. If we are deleting applications in the system if its is been used in > a WF are we going to allow deletion? > 5. In an experiment can we use more than one WF? can we use WF and > application in an experiment? > > Thank You, > > Best Regards, > > Eroma > > > > > > On Mon, Jun 9, 2014 at 12:02 AM, Sachith Withana <[email protected]> > wrote: > >> Thanks Raman. >> >> You are exactly correct. >> >> I have three questions. >> >> 1. We should store the DAG only. But the problem is that DAG contains >> which applications are used, and how they are connected.( which output goes >> to which input). This is a lot of data that is already available in the >> .wxf files that we use. So basically what you are proposing is shred the >> .wxf files and store them in the App Catalog? >> >> 2. If I remember correctly, the workflows we used had web services as >> nodes (SimpleMath workflow) , and they were not available in the registry >> as applications before hand. How do you propose we handle that scenario? >> >> 3. Overall, what's the purpose of having .wxf files ( WSDLS) as workflows >> since now that we will only be using it to transfer data between the >> client( XBaya) and the server correct? >> >> >> On Thu, Jun 5, 2014 at 7:32 PM, Raminder Singh <[email protected]> >> wrote: >> >>> Sachith, >>> >>> Workflow is a set of applications configured to work together. User need >>> to define its individual applications and the next step is to create a >>> workflow to configure the applications together. We should not save >>> applications used in the workflow as part of workflow. String saved in >>> workflow is a DAG (Directed acyclic graph) and it has reference to >>> applications (currently services). Other details saved in the workflow are >>> conditional blocks (if, for, while conditions), connections (input-output) >>> and defaults inputs configured by the user. Doing this it will keep the >>> structure of workflow DAG simple and application defined can be used in >>> multiple workflows or run standalone if needed. Please let me know if you >>> have more questions >>> >>> Thanks >>> Raminder >>> >>> >>> On Jun 5, 2014, at 3:59 AM, Sachith Withana <[email protected]> wrote: >>> >>> So what you are suggesting is instead of saving a one big string, we >>> should shred it and store the nodes and their relationships separately so >>> that it can be queriable? >>> >>> But XBaya and the Workflow Interpreter would still be using the string >>> right? >>> >>> >>> On Tue, Jun 3, 2014 at 9:08 PM, Saminda Wijeratne <[email protected]> >>> wrote: >>> >>>> >>>> >>>> On Sat, May 31, 2014 at 6:42 AM, Sachith Withana <[email protected]> >>>> wrote: >>>> >>>>> >>>>> >>>>> On Thu, May 29, 2014 at 10:23 PM, Saminda Wijeratne <[email protected] >>>>>> wrote: >>>>>> >>>>>> >>>>>> On Thu, May 29, 2014 at 1:15 AM, Sachith Withana <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> When designing the Application Catalog for single Applications, I ran >>>>>>> into the issue of supporting workflows in the Application Catalog. >>>>>>> >>>>>>> Should we store the workflows ( workflow files) and keep ids, which >>>>>>> would be the handles for those workflows? >>>>>>> >>>>>> Thanks for bringing this up Sachith. In your opinion how do you think >>>>>> the workflow should be represented through the CPI? Workflow string or >>>>>> Thrift object with workflow metadata distinguished or Thrift object with >>>>>> all workflow data distinguished or any other way? I suppose we need to >>>>>> figure-out what should be queried in a workflow. >>>>>> >>>>> >>>>> For the functional purposes I think it would be better to store the >>>>> workflow the way it is right now purely because we wouldn't want this to >>>>> affect the Workflow Interpreter behavior. But we can store meta-data about >>>>> the workflows to whatever the features we would want to support. Is there >>>>> a >>>>> way to get the applications residing in a workflow without explicitly >>>>> passing it to get the meta-data? >>>>> >>>> Workflow nodes specifically does directly correspond to applications but >>>> service descriptors. The current workflow model object have functions to >>>> return these service descriptors. >>>> >>>> >>>>> >>>>>>> We would have to keep track of the applications that are in that >>>>>>> workflow so that the deployment related data would be reused. >>>>>>> >>>>>>> Similarly to the single Applications, the sharing and other features >>>>>>> would be available for the workflows as well. >>>>>>> >>>>>>> Is there any more details that I should be concerned about when >>>>>>> implementing the aforementioned approach? >>>>>>> >>>>>> IMO we can generalize workflows as a composite application with >>>>>> special properties. For example a workflow could simply be another >>>>>> deployment for an Application interface. wdyt? >>>>>> >>>>> >>>>> I agree. I'll send you the current design I have which generalize this. >>>>> >>>>> >>>>>> >>>>>>> If we are planning to add searching capability with the workflows as >>>>>>> well, then we'd have to store the applications that are used in the >>>>>>> workflows, separately in the database as well instead of storing it as a >>>>>>> whole. >>>>>>> >>>>>> You mean node information here when you say "applications" right? >>>>>> >>>>> yep. wdyt? >>>>> >>>> I think it brings us back to whether want to save the workflow as a >>>> single blob or as separate data for the graph. IMO what we send to the >>>> registry should be a complex workflow object and we should be able to query >>>> those objects from the registry based on their properties. How registry >>>> saves this complex object (as a single blob, metadata+single blob or >>>> metadata+graph data) is immaterial for the user of the registry/app >>>> catalog. >>>> >>>> >>>> >>>>>>> Any suggestions/comments on the matter is highly appreciated. >>>>>>> >>>>>>> -- >>>>>>> Thanks, >>>>>>> Sachith Withana >>>>>>> >>>>>>> >>>>> >>>>> -- >>>>> Thanks, >>>>> Sachith Withana >>>>> >>>>> >>> >>> -- >>> Thanks, >>> Sachith Withana >>> >>> >>> >> >> -- >> Thanks, >> Sachith Withana >> >> >
