HI Daniel, If I have 100 tar.gz files and I have 100 slots, submitting them in parallel would be fast since all of them are running simultaneously. I know un tarring a file is surely I/O bound process.
Regards Varun On Thu, Jun 13, 2019 at 12:51 PM Daniel Povey <dpo...@gmail.com> wrote: > By the way, un-tarring a file is an I/O bound process and it will usually > give you no benefit to run on more than about 4 machines. Fastest and best > for the network would be to log into the file server if you have access, > and do it sequentially from there. > > > On Thu, Jun 13, 2019 at 12:44 PM Daniel Povey <dpo...@gmail.com> wrote: > >> >> for i in *tar.gz; >> do >> while true; do >> if [ $(qstat -u $USER | wc -l) -lt 900 ]; then break; fi; >> sleep 60; >> done >> qsub -l h_vmem=4G -cwd -j y -b y -N tar -R y -q all.q,gpu.q "tar -xzf >> $i" >> done >> >> On Thu, Jun 13, 2019 at 12:39 PM Skylar Thompson <skyl...@uw.edu> wrote: >> >>> We've used resource quota sets to accomplish that on a per-queue or >>> per-project basis. I don't know that you can limit on jobs in RQSs but >>> you >>> certainly can on slots; the sge_resource_quota(5) man page has some >>> examples. >>> >>> On Thu, Jun 13, 2019 at 12:32:51PM -0400, VG wrote: >>> > I have a scripting question regarding submitting jobs to the cluster. >>> > There is a limitation per user of 1000 jobs only. >>> > Let's say I have 1200 tar.gz files >>> > >>> > >>> > I tried to submit all the jobs together but after 1000 jobs it gave me >>> an >>> > error message saying per user limit is 1000 and after that it did not >>> > queued the remaining jobs. >>> > I want to write a script where if the submitted jobs goes below >>> > 1000(because they finished running), then next jobs are submitted in >>> the >>> > queue. How can I do that? >>> > I have written something like this: >>> > >>> > for i in *tar.gz >>> > do >>> > qsub -l h_vmem=4G -cwd -j y -b y -N tar -R y -q all.q,gpu.q "tar >>> -xzf $i" >>> > done >>> > >>> > Hope to hear from you soon. >>> > >>> > Regards >>> > Varun >>> >>> > _______________________________________________ >>> > users mailing list >>> > users@gridengine.org >>> > https://gridengine.org/mailman/listinfo/users >>> >>> >>> -- >>> -- Skylar Thompson (skyl...@u.washington.edu) >>> -- Genome Sciences Department, System Administrator >>> -- Foege Building S046, (206)-685-7354 >>> -- University of Washington School of Medicine >>> _______________________________________________ >>> users mailing list >>> users@gridengine.org >>> https://gridengine.org/mailman/listinfo/users >>> >> _______________________________________________ > users mailing list > users@gridengine.org > https://gridengine.org/mailman/listinfo/users >
_______________________________________________ users mailing list users@gridengine.org https://gridengine.org/mailman/listinfo/users
