Yo all I've been doing further research into the modex and came across something I don't fully understand. It seems we have each process insert into the modex the name of the PML module that it selected. Once the modex has exchanged that info, it then loops across all procs in the job to check their selection, and aborts if any proc picked a different PML module.
All well and good...assuming that procs actually -can- choose different PML modules and hence create an "abort" scenario. However, if I look inside the PML's at their selection logic, I find that a proc can ONLY pick a module other than ob1 if: 1. the user specifies the module to use via -mca pml xyz or by using a module specific mca param to adjust its priority. In this case, since the mca param is propagated, ALL procs have no choice but to pick that same module, so that can't cause us to abort (we will have already returned an error and aborted if the specified module can't run). 2. the pml/cm module detects that an MTL module was selected, and that it is other than "psm". In this case, the CM module will be selected because its default priority is higher than that of OB1. In looking deeper into the MTL selection logic, it appears to me that you either have the required capability or you don't. I can see that in some environments (e.g., rsh across unmanaged collections of machines), it might be possible for someone to launch across a set of machines where some do and some don't have the required support. However, in all other cases, this will be homogeneous across the system. Given this analysis (and someone more familiar with the PML should feel free to confirm or correct it), it seems to me that this could be streamlined via one or more means: 1. at the most, we could have rank=0 add the PML module name to the modex, and other procs simply check it against their own and return an error if they differ. This accomplishes the identical functionality to what we have today, but with much less info in the modex. 2. we could eliminate this info from the modex altogether by requiring the user to specify the PML module if they want something other than the default OB1. In this case, there can be no confusion over what each proc is to use. The CM module will attempt to init the MTL - if it cannot do so, then the job will return the correct error and tell the user that CM/MTL support is unavailable. 3. we could again eliminate the info by not inserting it into the modex if (a) the default PML module is selected, or (b) the user specified the PML module to be used. In the first case, each proc can simply check to see if they picked the default - if not, then we can insert the info to indicate the difference. Thus, in the "standard" case, no info will be inserted. In the second case, we will already get an error if the specified PML module could not be used. Hence, the modex check provides no additional info or value. I understand the motivation to support automation. However, in this case, the automation actually doesn't seem to buy us very much, and it isn't coming "free". So perhaps some change in how this is done would be in order? Ralph
