On 23 March 2012 16:47, William Hay <[email protected]> wrote: > 2012/3/23 "Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D." <[email protected]>: >> hi >> please subscribe [email protected] >> https://gridengine.org/mailman/listinfo/users >> >> regards >> >> >> On 3/23/2012 12:08 PM, Ventre, Brian D. wrote: >>> All, >>> >>> We are in the process of setting up a Rocks 5.4.3 cluster with SGE and need >>> some advice on setting up queues. Our configuration is: >>> - 1 front-end node (4 cores, Xeon X3460, Lynnfield) >>> - 1 login node (16 cores, 2x Xeon E5-2690, Sandy Bridge-EP) >>> - 4 "small" compute nodes (4 cores, Xeon X3460, Lynnfield) >>> - 5 "large" compute nodes (16 cores, 2x Xeon E5-2690, Sandy Bridge-EP) >>> All nodes are connected with 1 GbE. We made two separate purchases about >>> 1.5 years apart, hence the node disparity. This cluster is built >>> specifically to support a home-grown MPI application (with later growth to >>> other apps). The application is structured somewhat differently than I >>> think is standard, in that rank 0 is purely single threaded (a "collector" >>> type process), while the other ranks expand to fill every core on their >>> node. >>> >>> Because of security requirements, there is no shell access via SSH to any >>> of the compute nodes (except root). I think this means no mpirun, and that >>> we have to support the scheduler from the start. I want to set up a >>> special queue for our app that: >>> - Puts rank 0 on one of the small compute nodes (which conceivably could >>> live with another rank 0 from a separate instance) >>> - Spreads all other ranks across the large compute nodes (1 node == 1 >>> rank), up to the size specified by a user during job submission. Whoops missed that second requirement. Similar trick with quotas, single slot queues. Or you could just add a consumable with 1 available on each node in combination with -masterq. You might want to request an exclusive resource as well.
>>> Since the cluster is so small, I can't really afford to waste a large node >>> on rank 0. Processing speed would take a big hit if we end up with one of >>> the non-rank 0 processes on a small node. Alternatively, if we could >>> "double up" rank 0 and one other rank, that would probably work as well. >>> >>> We are already making a roll with our customizations (user authentication, >>> custom apps, etc), so we have a place to put any modifications. >>> >>> So my questions: >>> - What suggestions, if any, do people have for this type of layout? >>> - Should I make a separate appliance type for my small/large nodes (so >>> I/SGE can tell the difference)? >>> - My experience with SGE is nil, and I haven't found anything that gives a >>> good guide to heterogeneous queues. Anyone have a resource? > Grid engine queues are not its primary resource allocation mechanism. > In this instance you probably want to work mostly with parallel > environments. > > I'd probably go with the following: > 1.Create a lot(equal to the maximum number of jobs you can accomodate) > of identical PEs whose names match a pattern (eg mpi*). > 2.Use RQS to restrict each PE to 1 slot on any of the small nodes > (collectively). > 3.Submit jobs specifying the pattern as the pename and -masterq to > force the master > process on to the small nodes. If you don't trust the users to do > this correctly use a JSV or sge_request file. > If you don't want to use RQS you could achieve the same effect by > having single slot queues on the small nodes > each associated with a single PE and a single host. > > William _______________________________________________ users mailing list [email protected] https://gridengine.org/mailman/listinfo/users
