On Jun 1, 2012, at 4:28 PM, Tom Harvill wrote: > We would rather that OpenMPI use shared-mem (sm) module when running > intra-node processes.
Doesn't PSM use shared memory to communicate between peers on the same node? (that is hidden from us in Open MPI; I am *assuming* that internal to PSM, it uses shared memory for on-node communication and QLogic IB for off-node communication) > We believe that by using our scheduler's allocation policy (packing) and > considering our job mix, we might be able to add nodes to this cluster using > only one HCA per node (again, we would rather not use 'shared contexts'). Are you saying that you want Open MPI to not use PSM when the job entirely fits within a single node? If so, you might want to experiment with the pre-job hook in the job scheduler. You could try setting MCA parameters as environment variables (e.g., setenv OMPI_MCA_pml ob1 -- which would exclude the CM PML and therefore the PSM MTL) if your pre-job hook can tell if the job fits entirely on a single node. Does that help? -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
