You can try to write the script to first scan all the files to get their full
path names and then run the Array jobs.
> On Jun 13, 2019, at 1:20 PM, VG <gupta567va...@gmail.com> wrote:
>
> HI Joshua,
> I like the array job option because essentially it will still be 1 job and it
> will run them in parallel.
>
> I have one issue though. I can create an array script, but here I presented a
> simple problem. Actually my individual tar.gz files are under respective
> directories
> For example
> dir1 has file1.tar.gz
> dir2 has file2.tar.gz
> dir3 has file3.tar.gz
>
> The way I was then submitting them was
>
> for i in *
> do
> if [ -d "$i" ]
> then cd "$i"
> qsub -l h_vmem=4G -cwd -j y -b y -N tar -R y -q all.q,gpu.q "tar -xf
> *.tar"
> cd ..
> fi
> done
>
> One way is I can pull out all the tar.gz in one folder and run array script
> as you told, other wise is there a work around where everything runs and also
> remains in the respective directories.
>
> Thanks for your help.
>
> Regards
> Varun
>
>
>
>
>
>> On Thu, Jun 13, 2019 at 1:11 PM Joshua Baker-LePain <j...@salilab.org> wrote:
>> On Thu, 13 Jun 2019 at 9:32am, VG wrote
>>
>> > I have a scripting question regarding submitting jobs to the cluster.
>> > There is a limitation per user of 1000 jobs only.
>> > Let's say I have 1200 tar.gz files
>> >
>> > I tried to submit all the jobs together but after 1000 jobs it gave me an
>> > error message saying per user limit is 1000 and after that it did not
>> > queued the remaining jobs.
>> > I want to write a script where if the submitted jobs goes below
>> > 1000(because they finished running), then next jobs are submitted in the
>> > queue. How can I do that?
>> > I have written something like this:
>> >
>> > for i in *tar.gz
>> > do
>> > qsub -l h_vmem=4G -cwd -j y -b y -N tar -R y -q all.q,gpu.q "tar -xzf $i"
>> > done
>>
>> The right answer to this problem (not the scripting issue, but how to
>> untar all 1200 files without running afoul of the 1000 job limit) is an
>> array job. You can submit 1 job with 1200 tasks to untar all the files.
>> The relevant bits of the job script would include (assuming bash):
>>
>> targzs=(0 file1.tar.gz file2.tar.gz ... file1200.tar.gz)
>> tar xzf ${targzs[$SGE_TASK_ID]}
>>
>> To submit it:
>>
>> qsub -t 1-1200 job.sh
>>
>> --
>> Joshua Baker-LePain
>> QB3 Shared Cluster Sysadmin
>> UCSF
> _______________________________________________
> users mailing list
> users@gridengine.org
> https://gridengine.org/mailman/listinfo/users
_______________________________________________
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users
