Hi Udayanga, Airavata workflows supports both data flow and control flow executions. That means if you connect output of first step to input of second step, then execution of second step will wait until the output from step 1 is available. In control flow, you can connect the right hand top corner of the step 1 to left hand bottom corner of step 2. This will indicate the execution of step 2 will wait until step 1 is executed. This will be independent of data dependencies.
Your workflow is not working since you are using SSH provider which as you described the load leveler script is executing as a synchronous command. There are plans in upcoming 0.9 release to support asynchronous local and ssh provider, they should help you. Or if you have a Grid middleware like GRAM, then they by nature support asynchronous batch submissions. Unfortunately the load leveler version on Big Red is only partial implementation of the load leveler specification and only web services version of the GRAM (GRAM 4) used to work with it. The latest pre-ws gram (GRAM 5), does not support load leveler version on Big Red. Do you have accounts on Quarry cluster? It has a GRAM5 installation which should work with Airavata. In the near future, we plan to develop native local resource manager integration so Airavata can directly interact with batch systems through ssh and gsi-ssh protocols. Suresh On Aug 12, 2013, at 8:56 PM, Udayanga Wickramasinghe <[email protected]> wrote: > Hi , > I have a workflow (which is a simple one at the moment) that will execute on > a remote environment in several stages. Actually I have registered a command > line application on Airavata that will use SSH provider to run them in a > pipeline. For a single step this works fine but i am having trouble > connecting them to execute one after the other. > > My requirement is like this , first step of the workflow will take several > inputs and generate an output file (using a remote load leveler script OR/AND > MPI application ) and we need to block/wait until a result is generated and > we take the this result plus some more inputs and execute the second. > Likewise this process will continue for several stages on a remote grid > environment (ie:- Bigred) and final result will be copied back to our staged > servers. How can we achieve a blocking wait in a workflow with Airavata > specially with load leveler/mpi application job submission being asynchronous > ? Is there any special constuct that support this out of the box or do we > have to write extensions ? Even if we can create multiple command line > application/s easily with Airavata i don't see a way to link them in a > pipeline manner. I would very much appreciate your thoughts on this and > possible way to approach this. > > Thanks > Udayanga > > > -- > http://www.udayangawiki.blogspot.com
