Hi, Am 03.08.2012 um 11:55 schrieb Gaya Nadarajan:
> I'm relatively new to sge, however I have the task of making optimal use of 7 > multi-core VM nodes (5 in a cluster and 2 individual) for a set of tasks. The > grid engine is installed in one of the individual VMs which is serving as the > master node. > > I have a set of independent jobs (in hundreds or thousands) which each have 3 > subtasks that have to be run in sequence). I would like a node to be assigned > to the chunk of 3 jobs so that they can share the data between them. > > Reading the sge manual suggests that a resource is really a queue. Yes, you can say so. You submit a job and SGE selects a queue instance (i.e. a queue on a particular exechost) for you which fulfills the resource requests you specified. > Should I create many queues, each with the 3 subtasks so that the data and > task dependencies could be dealt with? Usually it's best to have as few queues as possible. Running the jobs in sequence is no problem, you can use -hold_jid for it. But the "problem" is the temporary data. I assume, you want the temporary data on the local disk and not a shared file space, where it can be accessed by all nodes. The best would be to assemble one job from your 3 subtasks instead. This way you can also use the $TMPDIR, which is created and removed by SGE - preferable on a local /scratch file space. Otherwise you would need to tell the other two jobs to run on the same node the first task was scheduled to (or route all by hand during the submission to individual particular exechosts). > Also I hear that array task dependencies could be good for such tasks but not > sure if I should use them. I think this is not suitable for your setup. E.g. you have two array jobs A and B, each running from [1..10]. Then B[1] should be allowed to start as soon as A[1] finished. Similar for the other 9 job indices. -- Reuti > Any advice or suggestions would be much appreciated. > > > Many Thanks, > Gaya > > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > _______________________________________________ > users mailing list > [email protected] > https://gridengine.org/mailman/listinfo/users _______________________________________________ users mailing list [email protected] https://gridengine.org/mailman/listinfo/users
