Hi Chris, Apologies for my delay in replying. I'm still seeing this issue. I have a sequential workflow that's working fine, but when I add a parallel section I'm seeing the error that I pasted into my previous email. I might be missing some config steps. Is my understanding correct - that OODT is trying to generate a special task (branch redirection task) that sets up a new workflow on a batch node? What should happen after that? Should this workflow task then try to run one of the parallel tasks? I've configured a batch stub running on the same machine as the master workflow to handle these requests... I'm not sure if it can access the same workflow manager configs though... I checked the workflow manager URL and it seems fine for the branch redirection task.
Ross On 15 May 2015 at 18:11, Mattmann, Chris A (3980) < [email protected]> wrote: > Ross you still having this issue? > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Chris Mattmann, Ph.D. > Chief Architect > Instrument Software and Science Data Systems Section (398) > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA > Office: 168-519, Mailstop: 168-527 > Email: [email protected] > WWW: http://sunset.usc.edu/~mattmann/ > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Adjunct Associate Professor, Computer Science Department > University of Southern California, Los Angeles, CA 90089 USA > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > > > > -----Original Message----- > From: Ross Laidlaw <[email protected]> > Reply-To: "[email protected]" <[email protected]> > Date: Wednesday, May 13, 2015 at 2:37 AM > To: "[email protected]" <[email protected]> > Subject: Re: Request for advice on workflows > > >Hi Chris, > > > >As a first step I got an XMLWorkflowRepository workflow running. I then > >tried using the PackagedWorkflowRepository to create a workflow with the > >same tasks. I got a sequential workflow up and running, similar to the > >following: > > > ><cas:workflows ...> > > <sequential id="urn:test:MainWorkflow" name="MainWorkflow"> > > <task id-ref="urn:test:TaskA"/> > > <task id-ref="urn:test:TaskB"/> > > <task id-ref="urn:test:TaskC"/> > > <task id-ref="urn:test:TaskD"/> > > <task id-ref="urn:test:TaskE"/> > > <task id-ref="urn:test:TaskF"/> > > <task id-ref="urn:test:TaskG"/> > > </sequential> > > > > <!-- task definitions --> > > <task>...</task> > > > ></cas:workflows> > > > > > >I found that I could run the workflow using ./wmgr-client --url > >[WorkflowManagerUrl] --sendEvent --eventName [WorkflowId] > >e.g. ./wmgr-client --url http://localhost:9001 --sendEvent --eventName > >urn:test:MainWorkflow > > > > > >After getting the sequential workflow running, I wanted to run tasks C and > >D in parallel so I then tried adding a <parallel> section as follows: > > > ><cas:workflows ...> > > <sequential id="urn:test:MainWorkflow" name="MainWorkflow"> > > <task id-ref="urn:test:TaskA"/> > > <task id-ref="urn:test:TaskB"/> > > <parallel id="urn:test:ParallelWorkflow" name="ParallelWorkflow"> > > <task id-ref="urn:test:TaskC"/> > > <task id-ref="urn:test:TaskD"/> > > </parallel> > > <task id-ref="urn:test:TaskE"/> > > <task id-ref="urn:test:TaskF"/> > > <task id-ref="urn:test:TaskG"/> > > </sequential> > > ... > ></cas:workflows> > > > > > >The parallel section generates an additional task to be submitted called > >'Redirector Task' that wants to be executed in the 'high' queue using the > >BranchRedirector class: > > > >INFO: Task: [Redirector Task] has no required metadata fields > >May 13, 2015 1:27:29 AM > >org.apache.oodt.cas.resource.system.XmlRpcResourceManagerClient submitJob > >FINEST: > >[ > > { > > job.queueName=high, > > job.instanceClassName=org.apache.oodt.cas.workflow.structs.TaskJob, > > job.name=redirector-322db9d2-4663-4b89-9af0-f8f97ad74c80, > > job.id=, job.status=, job.load=2, > > job.inputClassName=org.apache.oodt.cas.workflow.structs.TaskJobInput > > }, > > { > > > >task.instance.class=org.apache.oodt.cas.workflow.examples.BranchRedirector > >, > > task.config={}, > > task.metadata={...} > > } > >] > > > > > >I wasn't sure how to configure my setup to deal with this task. From > >reading OODT-70 I gather that it might be trying to start up other > >wmgr-clients on batch stubs to run these tasks in separate workflows. I > >tried setting up a local batch stub that accepts jobs from the 'high' > >queue. When running the full workflow I get the following exception from > >the batch stub when it tries to execute the redirector task: > > > >INFO: Assigning job: [redirector-8561de94-2681-477b-8bcb-331d83224d5b] to > >node: [local-redirector] > >May 13, 2015 1:36:51 AM > >org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub > >genericExecuteJob > >INFO: stub attempting to execute class: > >[org.apache.oodt.cas.workflow.structs.TaskJob] > >org.apache.xmlrpc.XmlRpcClientException: Failure writing request > > at > >org.apache.xmlrpc.XmlRpcClientRequestProcessor.encodeRequest(XmlRpcClientR > >equestProcessor.java:76) > > at > >org.apache.xmlrpc.XmlRpcClientRequestProcessor.encodeRequestBytes(XmlRpcCl > >ientRequestProcessor.java:102) > > at > >org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:70) > > at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194) > > at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185) > > at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178) > > at > >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.sendEvent( > >XmlRpcWorkflowManagerClient.java:477) > > at > >org.apache.oodt.cas.workflow.examples.BranchRedirector.run(BranchRedirecto > >r.java:62) > > at > >org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102) > > at > >org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run > >(XmlRpcBatchStub.java:229) > > at java.lang.Thread.run(Thread.java:745) > >Caused by: org.apache.xmlrpc.XmlRpcException: null values not supported by > >XML-RPC > > at org.apache.xmlrpc.XmlWriter.writeObject(XmlWriter.java:231) > > at > >org.apache.xmlrpc.XmlRpcClientRequestProcessor.encodeRequest(XmlRpcClientR > >equestProcessor.java:72) > > ... 10 more > >org.apache.oodt.cas.resource.structs.exceptions.JobInputException: Failed > >to run task > > at > >org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:104) > > at > >org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run > >(XmlRpcBatchStub.java:229) > > at java.lang.Thread.run(Thread.java:745) > >Caused by: > >org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceExcept > >ion: > >Failure writing request > > at > >org.apache.oodt.cas.workflow.examples.BranchRedirector.run(BranchRedirecto > >r.java:64) > > at > >org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102) > > ... 2 more > >May 13, 2015 1:36:51 AM > >org.apache.oodt.cas.resource.batchmgr.XmlRpcBatchMgrProxy run > >SEVERE: Job execution failed for jobId '2015-05-13T01:36:38.746-07:00' : > >batchstub.executeJob returned false > >java.lang.Exception: batchstub.executeJob returned false > > at > >org.apache.oodt.cas.resource.batchmgr.XmlRpcBatchMgrProxy.run(XmlRpcBatchM > >grProxy.java:125) > > > > > >It looks like my sequential and parallel workflows have the required names > >and IDs. Do you know what could be going wrong? Maybe I'm missing a few > >vital configuration steps. > > > >Many thanks for your help, > > > > > >Ross > > > > > > > > > > > > > > > >On 8 May 2015 at 19:31, Ross Laidlaw <[email protected]> wrote: > > > >> Hi Chris, > >> > >> Many thanks for your advice. I'll check out the > >> PackagedWorkflowRepository and will let you know how it goes. > >> > >> Ross > >> > >> > >> > >> > On May 8, 2015, at 12:58 PM, Mattmann, Chris A (3980) < > >> [email protected]> wrote: > >> > > >> > Hey Ross: > >> > > >> > Great work! Try the PackagedWorkflowRepository. The best documentation > >> > that exists of it is in OODT-70 [1]. > >> > > >> > I would use that to support the type of workflows that you’re doing. > >> > If this works we should make it the default since it’s supposed to > >> > be back compat with the XMLWorkflowRepositoryManager. > >> > > >> > Cheers, > >> > Chris > >> > > >> > P.S. see some of the ASO scripts I wrote to check on specific task/ > >> > PGE status, etc. > >> > > >> > > >> > [1]https://issues.apache.org/jira/browse/OODT-70 > >> > > >> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> > Chris Mattmann, Ph.D. > >> > Chief Architect > >> > Instrument Software and Science Data Systems Section (398) > >> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA > >> > Office: 168-519, Mailstop: 168-527 > >> > Email: [email protected] > >> > WWW: http://sunset.usc.edu/~mattmann/ > >> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> > Adjunct Associate Professor, Computer Science Department > >> > University of Southern California, Los Angeles, CA 90089 USA > >> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> > > >> > > >> > > >> > > >> > > >> > > >> > -----Original Message----- > >> > From: Ross Laidlaw <[email protected]> > >> > Reply-To: "[email protected]" <[email protected]> > >> > Date: Friday, May 8, 2015 at 8:14 AM > >> > To: "[email protected]" <[email protected]> > >> > Subject: Request for advice on workflows > >> > > >> >> Hi All, > >> >> > >> >> I'm working on an OODT setup (v0.8.1) with several workflow tasks > >> defined > >> >> and working well individually. I can use the wmgr-client to fire off > >> jobs > >> >> separately and all appears to be working as planned. I'd like to > >> combine > >> >> these tasks into a workflow so that I don't have to run each set of > >> jobs. > >> >> For example, in the workflow I'd like to run several 'task A' jobs > >>and > >> >> when > >> >> all these have finished I'd like the workflow to automatically start > >> >> several 'task B' jobs, etc. > >> >> > >> >> I found some example workflows in the distribution, plus several > >>guides > >> up > >> >> on the wiki. Is there a current recommended way of setting up a > >>simple > >> >> workflow? Should I use workflow v2 or is it best to stick with v1? > >> Once > >> >> I've set up a workflow, how do I start it off? It looks like it > >>might > >> be > >> >> possible to nest workflows, is that functionality available yet? > >> >> > >> >> Additionally, what's the best way to check on the status of currently > >> >> running jobs? I've used the pcs_stat tool a lot and it's great for > >> >> checking on the overall status, but is there a recommended way to > >>check > >> >> e.g. if any 'task A' jobs are still running for a certain workflow > >>and > >> >> what > >> >> state they're currently in? > >> >> > >> >> Many thanks in advance for your advice, > >> >> > >> >> > >> >> Ross > >> > > >> > >
