PMIx and PRRTE both read and forward their respective default MCA parameters from default system and user-level param files:
<install>/etc/pmix-mca-params.conf <homedir>/.pmix/mca-params.conf <install>/etc/prte-mca-params.conf <homedir>/.prte/mca-params.conf PMIx will also do the same thing for OMPI default system and user-level params provided it is told the programming model. Getting the system-level defaults requires that the user have "OMPIHOME" set in their environment to point to the installation so we can find the default param file. In the case of "mpirun" and friends, we automatically do this. Our hope is that resource managers such as Slurm will update their PMIx support to include this feature - even though they auto-forward your environment, they don't know to collect the params in the default files, thus forcing every OMPI proc to open/read those files...and causing a logjam on the file system. In addition to the default params, PMIx also (if the OMPI model is declared) picks up and forwards OMPI-specific envars. By default, we capture all envars that start with "OMPI_". However, you can control that by setting the PMIX_MCA_pmdl_ompi_include and PMIX_MCA_pmdl_ompi_exclude params. There is a similar capability for OSHMEM. The default here is to forward all envars that start with "SHMEM_" or "SMA_", per the folks at Stonybrook. The question that has arisen is: what should the default params include? Obviously, the system manager as well as the user can set them to be anything they want, but it would be nice if we could provide an adequate initial set. For example, one group has suggested that the OMPI model include all "CUDA_" envars. I should note that PMIx also does this for fabrics! For example, if we see OPA is on the system, we automatically forward all "HFI_" and "PSM2_" envars. This is the only fabric with a PMIx plugin at the moment, so this feature is limited to OPA right now. Any thoughts on this? Ralph
