Possibly assigning a fair share to each job with -js would cause them to change priority between scheduling runs so different jobs would snaffle the reservations on each run.
On 13 December 2011 15:57, Lars van der bijl <[email protected]> wrote: > hey Reuti, > > I wrote a python api using networkx and a database layer called clue > so i've removed a few of the flags. > > qsub -r y -l mem_free=1.9G,s_vmem=2G,hbatch=1 -pe smp 1 -N > sshot3__out__rs_mantra1__ifd_gen -q *@@ifd_gen -P rnd -o > /production/tmp/gridjobs/lars/fluffy/grid_feed1.rvb9CO -e > /production/tmp/gridjobs/lars/fluffy/grid_feed1.rvb9CO -p 0 -v > GRIDJOB,ATTEMPTS,STEPFRAME,BLOCKONERROR,GRIDSHOT,SLOTS -t 1-50:5 > /tmp/gridTask_sshot3__out__rs_mantra1__ifd_gen.30851.0.sh > > qsub -r y -l mem_free=1.9G,s_vmem=2G -pe smp 4 -N > sshot3__out__rs_mantra1__rs_mantra_tile1__mantra_render_split_1_seed_0 > -q *@@atoms -P rnd -o > /production/tmp/gridjobs/lars/fluffy/grid_feed1.rvb9CO -e > /production/tmp/gridjobs/lars/fluffy/grid_feed1.rvb9CO -p 0 > -hold_jid_ad 47673 -v > SPLITTOTAL,SEEDTOTAL,GRIDJOB,ATTEMPTS,STEPFRAME,BLOCKONERROR,SEED,SPLIT,SEEDNUM,GRIDSHOT,SLOTS > -t 1-50:1 > /tmp/gridTask_sshot3__out__rs_mantra1__rs_mantra_tile1__mantra_render_split_1_seed_0.30771.0.sh > qsub -r y -l mem_free=1.9G,s_vmem=2G -pe smp 4 -N > sshot3__out__rs_mantra1__rs_mantra_tile1__mantra_render_split_3_seed_0 > -q *@@atoms -P rnd -o > /production/tmp/gridjobs/lars/fluffy/grid_feed1.rvb9CO -e > /production/tmp/gridjobs/lars/fluffy/grid_feed1.rvb9CO -p 0 > -hold_jid_ad 47673 -v > SPLITTOTAL,SEEDTOTAL,GRIDJOB,ATTEMPTS,STEPFRAME,BLOCKONERROR,SEED,SPLIT,SEEDNUM,GRIDSHOT,SLOTS > -t 1-50:1 > /tmp/gridTask_sshot3__out__rs_mantra1__rs_mantra_tile1__mantra_render_split_3_seed_0.30753.0.sh > qsub -r y -l mem_free=1.9G,s_vmem=2G -pe smp 4 -N > sshot3__out__rs_mantra1__rs_mantra_tile1__mantra_render_split_2_seed_0 > -q *@@atoms -P rnd -o > /production/tmp/gridjobs/lars/fluffy/grid_feed1.rvb9CO -e > /production/tmp/gridjobs/lars/fluffy/grid_feed1.rvb9CO -p 0 > -hold_jid_ad 47673 -v > SPLITTOTAL,SEEDTOTAL,GRIDJOB,ATTEMPTS,STEPFRAME,BLOCKONERROR,SEED,SPLIT,SEEDNUM,GRIDSHOT,SLOTS > -t 1-50:1 > /tmp/gridTask_sshot3__out__rs_mantra1__rs_mantra_tile1__mantra_render_split_2_seed_0.30657.0.sh > qsub -r y -l mem_free=1.9G,s_vmem=2G -pe smp 4 -N > sshot3__out__rs_mantra1__rs_mantra_tile1__mantra_render_split_0_seed_0 > -q *@@atoms -P rnd -o > /production/tmp/gridjobs/lars/fluffy/grid_feed1.rvb9CO -e > /production/tmp/gridjobs/lars/fluffy/grid_feed1.rvb9CO -p 0 > -hold_jid_ad 47673 -v > SPLITTOTAL,SEEDTOTAL,GRIDJOB,ATTEMPTS,STEPFRAME,BLOCKONERROR,SEED,SPLIT,SEEDNUM,GRIDSHOT,SLOTS > -t 1-50:1 > /tmp/gridTask_sshot3__out__rs_mantra1__rs_mantra_tile1__mantra_render_split_0_seed_0.30680.0.sh > > qsub -r y -l mem_free=1.9G,s_vmem=2G -pe smp 1 -N > sshot3__out__rs_mantra1__rs_mantra_tile1__joinexr -q *@@allhosts -P > rnd -o /production/tmp/gridjobs/lars/fluffy/grid_feed1.rvb9CO -e > /production/tmp/gridjobs/lars/fluffy/grid_feed1.rvb9CO -p 0 > -hold_jid_ad 47677,47674,47675,47676 -v > GRIDJOB,ATTEMPTS,STEPFRAME,BLOCKONERROR,GRIDSHOT,SLOTS -t 1-50:1 > /tmp/gridTask_sshot3__out__rs_mantra1__rs_mantra_tile1__joinexr.30998.0.sh > > now the command runs fine. but the behaviour of the queue seems to be > to want to finish a PE job before moving on to a next one where as > what i'd want is to do the splits in a round robin fashion. > > Lars > > > > > > > > > On 13 December 2011 15:48, Reuti <[email protected]> wrote: >> Hi, >> >> Am 13.12.2011 um 16:25 schrieb Lars van der bijl: >> >>> we have been running our sge for a while now but we implemented a new >>> technique and I'm having trouble figuring out how to make the grid >>> help with it. >>> >>> I have the following task / dependency structure. >>> >>> task1 >>> >>> task2_seed_0 = dependent on output of task1 >>> task2_seed_1 = dependent on output of task1 >>> task2_seed_2 = dependent on output of task1 >>> task2_seed_3 = dependent on output of task1 >> >> how did you set up the job dependency and submission in detail? >> >> -- Reuti >> >> >>> joiner = dependent on output of seed tasks. >>> >>> all of these are Parallel environments with a range from 1-100 and a batch >>> of 1 >>> >>> now what I would like to happen is that the Grid finishes doing the >>> first batch of task 1. >>> then all 4 of the task2 will get there first batch started. >>> >>> what's currently happening is that sge tries to schedule and finish >>> all of task2_seed_0 before moving on to the task2_seed_1. >>> >>> anyone have a idea on how to get this behaviour? >>> >>> Lars >>> _______________________________________________ >>> users mailing list >>> [email protected] >>> https://gridengine.org/mailman/listinfo/users >> > _______________________________________________ > users mailing list > [email protected] > https://gridengine.org/mailman/listinfo/users > > _______________________________________________ users mailing list [email protected] https://gridengine.org/mailman/listinfo/users
