On 2 February 2012 15:33, Robert Hutton <[email protected]> wrote: > Hi Everyone, > > I've just set up a small Grid Engine cluster, but I'm new to using Grid > Engine, and would like some advice on the best way to submit jobs that > in turn submit jobs. What I'd like to do is: > > Run a regular shell script that loops over all files in a directory, and > submits a job for each file. This job will create a derived file. Once > this file is created, I'd like to submit /two/ jobs for each derived > file, which can run concurrently with each other and produce two > separate derived files from the first file. > > Is the right approach to have: > > - a regular shell script, say run.sh that does a loop over all files and > submits a job for each with "qsub process.sh $filename" > > - the job script called process.sh that runs the command to create the > derived file, then runs qsub on the two subcommands that rely on the > output file from the first command, with something like: > qsub subcommand1 $derived_file > ${derived_file}1 > qsub subcommand2 $derived_file > ${derived_file}2 > > > Or would it be better to just have a single job script that requests two > slots, and does all of the commands, calling subshells for the two > subcommands? Something like: > > for f in directory/* > do > command1 "$f" > "$f.out" > (command2 "$f.out" > "$f.out2") & > (command3 "$f.out" > "$f.out3") & > done > > Or are these just completely wrong approaches? Is there a better way? I don't know about better but a more conventional way would be to have the front end script submit three jobs per file. One regular and two using -hold_jid on the first job. To get fancier with it you could have your initial submit script generate a list of files and the intended name of the corresponding derived file. Then submit three suitably sized array jobs using -hold_jid_ad for the second one. Each would then use its $SGE_TASK_ID to pick the right file to work on from your list.
_______________________________________________ users mailing list [email protected] https://gridengine.org/mailman/listinfo/users
