Hi Suresh, Thanks a lot for your thoughts on this. I see control flow strategy would work perfect for my case , however sighting your problems at GRAM support on Bigred i may need to implement an 'synchronous' call at the Bigred side (ie:- a polling mechanism) . Unfortunately we don't have a Quarry cluster as well and we are at the moment migrating to Bigred2. I think the future Airavata support you mentioned would be the perfect case for a scenario like ours (because we have hundreds of scripts that perform molecular dynamics that we execute remotely using ssh ) and it would be a great addition to the Airavata tool set.
Regards Udayanga On Mon, Aug 12, 2013 at 9:45 PM, Suresh Marru <[email protected]> wrote: > Hi Udayanga, > > Airavata workflows supports both data flow and control flow executions. > That means if you connect output of first step to input of second step, > then execution of second step will wait until the output from step 1 is > available. In control flow, you can connect the right hand top corner of > the step 1 to left hand bottom corner of step 2. This will indicate the > execution of step 2 will wait until step 1 is executed. This will be > independent of data dependencies. > > Your workflow is not working since you are using SSH provider which as you > described the load leveler script is executing as a synchronous command. > There are plans in upcoming 0.9 release to support asynchronous local and > ssh provider, they should help you. Or if you have a Grid middleware like > GRAM, then they by nature support asynchronous batch submissions. > Unfortunately the load leveler version on Big Red is only partial > implementation of the load leveler specification and only web services > version of the GRAM (GRAM 4) used to work with it. The latest pre-ws gram > (GRAM 5), does not support load leveler version on Big Red. Do you have > accounts on Quarry cluster? It has a GRAM5 installation which should work > with Airavata. > > In the near future, we plan to develop native local resource manager > integration so Airavata can directly interact with batch systems through > ssh and gsi-ssh protocols. > > Suresh > > On Aug 12, 2013, at 8:56 PM, Udayanga Wickramasinghe < > [email protected]> wrote: > > > Hi , > > I have a workflow (which is a simple one at the moment) that will > execute on a remote environment in several stages. Actually I have > registered a command line application on Airavata that will use SSH > provider to run them in a pipeline. For a single step this works fine but i > am having trouble connecting them to execute one after the other. > > > > My requirement is like this , first step of the workflow will take > several inputs and generate an output file (using a remote load leveler > script OR/AND MPI application ) and we need to block/wait until a result is > generated and we take the this result plus some more inputs and execute the > second. Likewise this process will continue for several stages on a remote > grid environment (ie:- Bigred) and final result will be copied back to our > staged servers. How can we achieve a blocking wait in a workflow with > Airavata specially with load leveler/mpi application job submission being > asynchronous ? Is there any special constuct that support this out of the > box or do we have to write extensions ? Even if we can create multiple > command line application/s easily with Airavata i don't see a way to link > them in a pipeline manner. I would very much appreciate your thoughts on > this and possible way to approach this. > > > > Thanks > > Udayanga > > > > > > -- > > http://www.udayangawiki.blogspot.com > > -- http://www.udayangawiki.blogspot.com
