HI Joshua,
I like the array job option because essentially it will still be 1 job and
it will run them in parallel.
I have one issue though. I can create an array script, but here I presented
a simple problem. Actually my individual tar.gz files are under respective
directories
For example
dir1 has file1.tar.gz
dir2 has file2.tar.gz
dir3 has file3.tar.gz
The way I was then submitting them was
for i in *
do
if [ -d "$i" ]
then cd "$i"
qsub -l h_vmem=4G -cwd -j y -b y -N tar -R y -q all.q,gpu.q "tar -xf
*.tar"
cd ..
fi
done
One way is I can pull out all the tar.gz in one folder and run array script
as you told, other wise is there a work around where everything runs and
also remains in the respective directories.
Thanks for your help.
Regards
Varun
On Thu, Jun 13, 2019 at 1:11 PM Joshua Baker-LePain <j...@salilab.org> wrote:
> On Thu, 13 Jun 2019 at 9:32am, VG wrote
>
> > I have a scripting question regarding submitting jobs to the cluster.
> > There is a limitation per user of 1000 jobs only.
> > Let's say I have 1200 tar.gz files
> >
> > I tried to submit all the jobs together but after 1000 jobs it gave me an
> > error message saying per user limit is 1000 and after that it did not
> > queued the remaining jobs.
> > I want to write a script where if the submitted jobs goes below
> > 1000(because they finished running), then next jobs are submitted in the
> > queue. How can I do that?
> > I have written something like this:
> >
> > for i in *tar.gz
> > do
> > qsub -l h_vmem=4G -cwd -j y -b y -N tar -R y -q all.q,gpu.q "tar -xzf
> $i"
> > done
>
> The right answer to this problem (not the scripting issue, but how to
> untar all 1200 files without running afoul of the 1000 job limit) is an
> array job. You can submit 1 job with 1200 tasks to untar all the files.
> The relevant bits of the job script would include (assuming bash):
>
> targzs=(0 file1.tar.gz file2.tar.gz ... file1200.tar.gz)
> tar xzf ${targzs[$SGE_TASK_ID]}
>
> To submit it:
>
> qsub -t 1-1200 job.sh
>
> --
> Joshua Baker-LePain
> QB3 Shared Cluster Sysadmin
> UCSF
>
_______________________________________________
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users
