Hi,
Am 02.02.2012 um 16:33 schrieb Robert Hutton:
> I've just set up a small Grid Engine cluster, but I'm new to using Grid
> Engine, and would like some advice on the best way to submit jobs that
> in turn submit jobs. What I'd like to do is:
>
> Run a regular shell script that loops over all files in a directory, and
> submits a job for each file. This job will create a derived file. Once
> this file is created, I'd like to submit /two/ jobs for each derived
> file, which can run concurrently with each other and produce two
> separate derived files from the first file.
>
> Is the right approach to have:
>
> - a regular shell script, say run.sh that does a loop over all files and
> submits a job for each with "qsub process.sh $filename"
>
> - the job script called process.sh that runs the command to create the
> derived file, then runs qsub on the two subcommands that rely on the
> output file from the first command, with something like:
> qsub subcommand1 $derived_file > ${derived_file}1
> qsub subcommand2 $derived_file > ${derived_file}2
>
>
> Or would it be better to just have a single job script that requests two
> slots, and does all of the commands, calling subshells for the two
> subcommands? Something like:
>
> for f in directory/*
> do
> command1 "$f" > "$f.out"
> (command2 "$f.out" > "$f.out2") &
> (command3 "$f.out" > "$f.out3") &
> done
As long as command1 is running it would block one slot, which is not what you
want. And there is no "wait" command to wait for the two sub processes, which
could also be started by a plain &; there is no need for a subshell.
Can you put the workflow in a loop at submission time? In each loop iteration
you do:
JOB_ID=$(qsub -terse command1)
This will give you the JOB_ID of the submitted job.
Then submit the next steps like:
qsub -hold_jid $JOB_ID command2
qsub -hold_jid $JOB_ID command3
They will only start, after command1 finished.
-- Reuti
NB: using 2 slots and putting the processes in the background while waiting for
their completion would imply that they have nearly the same runtime to avoid
any waste of resources.
> Or are these just completely wrong approaches? Is there a better way?
>
> Thanks in advance,
>
> Rob
>
> --
> Robert Hutton
> Senior Systems and Database Administrator
> Centre for Genomics and Global Health <http://cggh.org>
> The Wellcome Trust Centre for Human Genetics
> Roosevelt Drive
> Oxford
> OX3 7BN
> United Kingdom
> Tel: +44 (0)1865 287721
> _______________________________________________
> users mailing list
> [email protected]
> https://gridengine.org/mailman/listinfo/users
_______________________________________________
users mailing list
[email protected]
https://gridengine.org/mailman/listinfo/users
