Yevgeny Kliteynik wrote:
Besides, if Lustre sends path queries to OpenSM (via RDMACM) when
opening their own QPs, then Lustre can use other SLs too. SM can even
assign different SLs to communication to Lustre metadata servers and
object storage servers if they are running on separate hosts, all
based on the path query.
Indeed, since Lustre uses the RDMA-CM which in turn does its path query
in a <SGID, DGID, PKEY, SID, TOS> manner you can use either of the
existing mechanisms in openSM to assign SL to this QP:
1. port group based (eg based on the SGID, DGID from the query)
2. pkey based (eg based on the PKEY)
3. SID based
4. TOS based
5. you name it...
~Everything happens under the cover of the rdma-cm, that is your app
need to do nothing such that <SGID, DGID, PKEY, SID, TOS=0> would be
provided by the rdma-cm in the path query, and the rdma-cm uses the SL
from the returned path when your app calls rdma_create_qp. Only if you
want non zero TOS to be in the query, rdma_set_service_type has to be
called before rdma_resolve_route.
The approach is that the host provides the information, the IB sys admin
design a QoS setting for the fabric and apply it through the mechanisms
of the openSM, please see the cool presentation of Liran Liss from
Sonoma @
http://www.openfabrics.org/archives/spring2008sonoma/Tuesday/qos_sonoma08_ofa_v1.ppt
and let us know if you have any further questions.
For MPIs which don't use path query the IB sys admin have instruct the
users which SL to manually make the ranks of this job use, which is by
far much less convenient.
Or.
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
