By the way, un-tarring a file is an I/O bound process and it will usually give you no benefit to run on more than about 4 machines. Fastest and best for the network would be to log into the file server if you have access, and do it sequentially from there.
On Thu, Jun 13, 2019 at 12:44 PM Daniel Povey <dpo...@gmail.com> wrote: > > for i in *tar.gz; > do > while true; do > if [ $(qstat -u $USER | wc -l) -lt 900 ]; then break; fi; > sleep 60; > done > qsub -l h_vmem=4G -cwd -j y -b y -N tar -R y -q all.q,gpu.q "tar -xzf > $i" > done > > On Thu, Jun 13, 2019 at 12:39 PM Skylar Thompson <skyl...@uw.edu> wrote: > >> We've used resource quota sets to accomplish that on a per-queue or >> per-project basis. I don't know that you can limit on jobs in RQSs but you >> certainly can on slots; the sge_resource_quota(5) man page has some >> examples. >> >> On Thu, Jun 13, 2019 at 12:32:51PM -0400, VG wrote: >> > I have a scripting question regarding submitting jobs to the cluster. >> > There is a limitation per user of 1000 jobs only. >> > Let's say I have 1200 tar.gz files >> > >> > >> > I tried to submit all the jobs together but after 1000 jobs it gave me >> an >> > error message saying per user limit is 1000 and after that it did not >> > queued the remaining jobs. >> > I want to write a script where if the submitted jobs goes below >> > 1000(because they finished running), then next jobs are submitted in the >> > queue. How can I do that? >> > I have written something like this: >> > >> > for i in *tar.gz >> > do >> > qsub -l h_vmem=4G -cwd -j y -b y -N tar -R y -q all.q,gpu.q "tar >> -xzf $i" >> > done >> > >> > Hope to hear from you soon. >> > >> > Regards >> > Varun >> >> > _______________________________________________ >> > users mailing list >> > users@gridengine.org >> > https://gridengine.org/mailman/listinfo/users >> >> >> -- >> -- Skylar Thompson (skyl...@u.washington.edu) >> -- Genome Sciences Department, System Administrator >> -- Foege Building S046, (206)-685-7354 >> -- University of Washington School of Medicine >> _______________________________________________ >> users mailing list >> users@gridengine.org >> https://gridengine.org/mailman/listinfo/users >> >
_______________________________________________ users mailing list users@gridengine.org https://gridengine.org/mailman/listinfo/users
