Signed-off-by: Nicolas Morey-Chaisemartin <[email protected]> --- opensm/doc/current-routing.txt | 32 ++++++++++++++++++++++++++++++++ opensm/man/opensm.8.in | 27 +++++++++++++++++++++++++++ 2 files changed, 59 insertions(+), 0 deletions(-)
diff --git a/opensm/doc/current-routing.txt b/opensm/doc/current-routing.txt index 0034d0e..1302860 100644 --- a/opensm/doc/current-routing.txt +++ b/opensm/doc/current-routing.txt @@ -237,6 +237,38 @@ in the same directory where the OpenSM log resides. This ordering file provides the CN order that may be used to create efficient communication pattern, that will match the routing tables. +Routing between non-CN nodes + + +The use of the cn_guid_file option allows non-CN nodes to be located on different levels in the fat tree. +In such case, it is not guaranteed that the Fat Tree algorithm will route between two non-CN nodes. +In the scheme below, N1, N2 and N3 are non-CN nodes. Although all the CN have routes to and from them, +there will not necessarily be a route between N1,N2 and N3. +Such routes would require to use at least one of the Switch the wrong way around +(In fact, go out of one of the top Switch through a downgoing port while we are supposed to go up). + + Spine1 Spine2 Spine 3 + / \ / | \ / \ + / \ / | \ / \ + N1 Switch N2 Switch N3 + /|\ /|\ + / | \ / | \ + Going down to compute nodes + +To solve this problem, a list of non-CN nodes can be specified by \'-G\' or \'--io_guid_file\' option. +Theses nodes will be allowed to use switches the wrong way around a specific number of times (specified by \'-H\' or \'--max_reverse_hops\'. +With the proper max_reverse_hops and io_guid_file values, you can ensure full connectivity in the Fat Tree. + +In the scheme above, with a max_reverse_hop of 1, routes will be instanciated between N1<->N2 and N2<->N3. +With a max_reverse_hops value of 2, N1,N2 and N3 will all have routes between them. + +Please note that using max_reverse_hops creates routes that use the switch in a counter-stream way. +This option should never be used to connect nodes with high bandwidth traffic between them ! It should only be used +to allow connectivity for HA purposes or similar. +Also having routes the other way around can in theory cause credit loops. + +Use these options with extreme care ! + Usage: diff --git a/opensm/man/opensm.8.in b/opensm/man/opensm.8.in index 7690980..ce14c02 100644 --- a/opensm/man/opensm.8.in +++ b/opensm/man/opensm.8.in @@ -22,6 +22,8 @@ opensm \- InfiniBand subnet manager and administration (SM/SA) [\-S | \-\-sadb_file <file name>] [\-a | \-\-root_guid_file <path to file>] [\-u | \-\-cn_guid_file <path to file>] +[\-G | \-\-io_guid_file <path to file>] +[\-H | \-\-max_reverse_hops <max reverse hops allowed>] [\-X | \-\-guid_routing_order_file <path to file>] [\-m | \-\-ids_guid_file <path to file>] [\-o(nce)] @@ -183,6 +185,16 @@ algorithm to the guids provided in the given file (one to a line). Set the compute nodes for the Fat-Tree routing algorithm to the guids provided in the given file (one to a line). .TP +\fB\-G\fR, \fB\-\-io_guid_file\fR <file name> +Set the I/O nodes for the Fat-Tree routing algorithm +to the guids provided in the given file (one to a line). +I/O nodes are non-CN nodes allowed to use up to max_reverse_hops switches +the wrong way around to improve connectivity. +.TP +\fB\-H\fR, \fB\-\-max_reverse_hops\fR <file name> +Set the maximum number of reverse hops an I/O node is allowed +to make. A reverse hop is the use of a switch the wrong way around. +.TP \fB\-m\fR, \fB\-\-ids_guid_file\fR <file name> Name of the map file with set of the IDs which will be used by Up/Down routing algorithm instead of node GUIDs @@ -800,6 +812,21 @@ in the same directory where the OpenSM log resides. This ordering file provides the CN order that may be used to create efficient communication pattern, that will match the routing tables. +Routing between non-CN nodes + +The use of the cn_guid_file option allows non-CN nodes to be located on different levels in the fat tree. +In such case, it is not guaranteed that the Fat Tree algorithm will route between two non-CN nodes. +To solve this problem, a list of non-CN nodes can be specified by \'-G\' or \'--io_guid_file\' option. +Theses nodes will be allowed to use switches the wrong way round a specific number of times (specified by \'-H\' or \'--max_reverse_hops\'. +With the proper max_reverse_hops and io_guid_file values, you can ensure full connectivity in the Fat Tree. + +Please note that using max_reverse_hops creates routes that use the switch in a counter-stream way. +This option should never be used to connect nodes with high bandwidth traffic between them ! It should only be used +to allow connectivity for HA purposes or similar. +Also having routes the other way around can in theory cause credit loops. + +Use these options with extreme care ! + Activation through OpenSM Use '-R ftree' option to activate the fat-tree algorithm. -- 1.6.1 _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
