Sasha Khapyorsky wrote:
Hi Yevgeny,
On 16:42 Wed 27 Feb , Yevgeny Kliteynik wrote:
The following doc describes QoS management in OpenSM.
This doc (named QoS_management_in_OpenSM.txt) has been added to
the OFED docs, along with the QoS_in_OFED.txt.
I'd like to add this info to OpenSM man pages as well.
Yes, I think that it could be useful to have it under opensm/doc too.
I'm including the text here as is, so it will be easier to follow
possible changes. When those will be done, I'll fix the format to
match the OpenSM man pages and post a patch.
The only problem is that the whole OpenSM man has ~850 lines,
while this QoS management file has ~500 lines... :)
I would suggest to have some basic part (50-100 lines) included in the
man page and reference an entire document (under opensm/doc) for more
details.
OK, I can prepare some kind of summary that would go into the man page.
However, this means that a user would no be able to define a QoS policy
just from reading an OpenSM man pages - he will HAVE to check the full
doc under opensm/doc.
Please review.
Looks fine, few tiny nits are below.
[snip...]
==============================================================================
4. Policy File Syntax Guidelines
==============================================================================
- Empty lines are ignored.
It is mentioned on the next line too.
Right
- Leading and trailing blanks, as well as empty lines, are ignored, so
the indentation in the example is just for better readability.
- Comments are started with the pound sign (#) and terminated by EOL.
- Any keyword should be the first non-blank in the line, unless it's a
comment.
- Keywords that denote section/subsection start have matching closing
keywords.
- Having a QoS Level named "DEFAULT" is a must - it is applied to PR/MPR
requests that didn't match any of the matching rules.
- Any section/subsection of the policy file is optional.
[snip...]
==============================================================================
6. Simplified QoS Policy - Details and Examples
==============================================================================
Simplified QoS policy match rules are tailored for matching ULPs (or some
application on top of a ULP) PR/MPR requests. This section has a list of
per-ULP (or per-application) match rules and the SL that should be enforced
on the matched PR/MPR query.
Match rules include:
- Default match rule that is applied to PR/MPR query that didn't match any
of the other match rules
- SDP
- SDP application with a specific target TCP/IP port range
- SRP with a specific target IB port GUID
- RDS
- iSER
- iSER application with a specific target TCP/IP port range
- IPoIB with a default PKey
- IPoIB with a specific PKey
- any ULP/application with a specific Service ID in the PR/MPR query
- any ULP/application with a specific PKey in the PR/MPR query
- any ULP/application with a specific target IB port GUID in the PR/MPR
query
Since any section of the policy file is optional, as long as basic rules of
the file are kept (such as no referring to nonexisting port group, having
default QoS Level, etc), the simplified policy section (qos-ulps) can serve
as a complete QoS policy file.
The shortest policy file in this case would be as follows:
qos-ulps
default : 0 #default SL
end-qos-ulps
It is equivalent to the previous example of the shortest policy file, and
it
is also equivalent to not having policy file at all.
Below is an example of simplified QoS policy with all the possible
keywords:
qos-ulps
default : 0 # default SL
sdp, port-num 30000 : 0 # SL for application running on
top
# of SDP when a destination
# TCP/IPport is 30000
sdp, port-num 10000-20000 : 0
sdp : 1 # default SL for any other
# application running on top of
SDP
rds : 2 # SL for RDS traffic
iser, port-num 900 : 0 # SL for iSER with a specific
target
# port
iser : 3 # default SL for iSER
ipoib, pkey 0x0001 : 0 # SL for IPoIB on partition with
# pkey 0x0001
ipoib : 4 # default IPoIB partition,
# pkey=0x7FFF
any, service-id 0x6234 : 6 # match any PR/MPR query with a
# specific Service ID
any, pkey 0x0ABC : 6 # match any PR/MPR query with a
# specific PKey
srp, target-port-guid 0x1234 : 5 # SRP when SRP Target is located
on
# a specified IB port GUID
any, target-port-guid 0x0ABC-0xFFFFF : 6 # match any PR/MPR query
with
# a specific target port GUID
end-qos-ulps
Likely I missed this in implementation phase. But isn't it better to
have ULPs to match QoS level rather than SL? Or probably both?
The goal of this part is to give up some flexibility in order to get maximum
simplicity. If an administrator wants to define a qos-level with more than
just SL (and I think that in most cases he wouldn't want to do that), there's
another way to refer to such qos-level - trough match rules.
Similar to the full policy definition, matching of PR/MPR queries is done
in
order of appearance in the QoS policy file such as the first match takes
precedence, except for the "default" rule, which is applied only if the
query
didn't match any other rule.
All other sections of the QoS policy file take precedence over the qos-ulps
section. That is, if a policy file has both qos-match-rules and qos-ulps
sections, then any query is matched first against the rules in the
qos-match-rules section, and only if there was no match, the query is
matched
against the rules in qos-ulps section.
Note that some of these match rules may overlap, so in order to use the
simplified QoS definition effectively, it is important to understand how
each
of the ULPs is matched:
6.1 IPoIB
IPoIB query is matched by PKey. Default PKey for IPoIB partition is 0x7fff,
so
the following three match rules are equivalent:
ipoib : <SL>
ipoib, 0x7fff : <SL>
any, pkey 0x7fff : <SL>
6.2 SDP
SDP PR query is matched by Service ID. The Service-ID for SDP is
0x000000000001PPPP, where PPPP are 4 hex digits holding the remote TCP/IP
Port
Number to connect to. The following two match rules are equivalent:
sdp : <SL>
any, service-id 0x0000000000010000-0x000000000001ffff : <SL>
6.3 RDS
Similar to SDP, RDS PR query is matched by Service ID. The Service ID for
RDS
is 0x000000000106PPPP, where PPPP are 4 hex digits holding the remote
TCP/IP
Port Number to connect to. Default port number for RDS is 0x48CA, which
makes
a default Service-ID 0x00000000010648CA. The following two match rules are
equivalent:
rds : <SL>
any, service-id 0x00000000010648CA : <SL>
6.4 iSER
Similar to RDS, iSER query is matched by Service ID, where the the Service
ID
is also 0x000000000106PPPP. Default port number for iSER is 0x035C, which
makes
a default Service-ID 0x000000000106035C. The following two match rules are
equivalent:
iser : <SL>
any, service-id 0x000000000106035C : <SL>
6.5 SRP
Service ID for SRP varies from storage vendor to vendor, thus SRP query is
matched by the target IB port GUID. The following two match rules are
equivalent:
srp, target-port-guid 0x1234 : <SL>
any, target-port-guid 0x1234 : <SL>
Note that any of the above ULPs might contain target port GUID in the PR
query, so in order for these queries not to be recognized by the QoS
manager
as SRP, the SRP match rule (or any match rule that refers to the target
port
guid only) should be placed at the end of the qos-ulps match rules.
6.6 MPI
SL for MPI is manually configured by MPI admin. OpenSM is not forcing any
SL
on the MPI traffic, and that's why it is the only ULP that did not appear
in
the qos-ulps section.
==============================================================================
7. SL2VL Mapping and VL Arbitration
==============================================================================
I think here should be stated that both policies should be merged at
some point.
Agree
-- Yevgeny
[snip...]
Sasha
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
