On Oct 20, 2015, at 5:26 PM, Paul Hargrove <phhargr...@lbl.gov> wrote:
>
> I have multiple ports of the same type, lets say a dual-port Mellanox HCA,
> and just want to disable one of them (reserving it for Luster perhaps).
> If OMPI is hiding from me the details of the API selection, how do I
> enable/disable specific ports?
> Right now I believe that I need two distinct MCA params to instruct ibv and
> mxm both to exclude a given IB port.
> I assume I will need two more params to tell ofi and ucx not to use the port
> either, right?
> Now lets assume I've got the portals4 reference implementation (over verbs)
> installed too.
> Now that makes no less than 5 MCA params I might need to pass to tell 5
> different components to keep their hands off the reserved port.
>
> If I didn't know the HCA model/vendor, I might need a sixth and seventh MCA
> param to tell psm and psm2 not to use the port.
> However, if the port name is something like "mlx4_0.0" or "ipath0.0" then we
> can at least know if mxm or psm* are even possible.
I don't think it gets up to 7 MCA params to guarantee a specific API path is
used to get to a specific network / port, but your overall point is fair. I
think it reduces to:
Should a "higher-level abstraction" CLI set of options include the option to
select a specific network interface?
I don't know the answer to that. One could argue that the "expert" layer will
already let you do that, and that it will be difficult enough to create this
new set of "higher abstraction" CLI options without *also* allowing the
specification of interface+port. But one could also argue -- and I think you
are :-) -- that without allowing the specification of interface+port, the
"higher abstraction" CLI options are significantly less useful.
> I don't suggest that OMPI should magically discover "aliases" for a port but
> if I have IPoIB the problem grows:
> I need yet more MCA params to tell tcp, ofi (and maybe ucx?) not to use the
> corresponding ibN interface.
> And don't forget about oob_tcp_if_exclude.
I'm not sure you're adding up MCA params correctly. To guarantee that I use a
specific IB interface (and not MXM and not PSM and not TCP/IPoIB and not ...),
for example, you only need three:
mpirun --mca pml ob1 --mca btl openib,vader,self --mca
btl_openib_if_include mlx4_0:0 ...
(but don't get me wrong -- I acknowledge that you need to be an OMPI expert to
know/figure that out, and that's not desirable)
Are there other cases where the MCA param count really does explode to 10 or
more?
> To ensure that Jeff doesn't dodge the issue, lets assume this desire to
> disable one port is a *transient* need/desire by an end-user.
> In other words, I don't accept "admin should place all 11 MCA params in the
> global config file" as a valid solution.
Yes -- same point above: you're arguing that without allowing specifying
interface+port, the higher-abstraction CLI options aren't as useful.
You might be right.
--
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/
