I feel very strongly that I'd prefer to avoid the use of a driver.conf
for this, and instead handle it as a Brussels property, at least on
Solaris Nevada. (This will support administration via dladm, and
ultimately also ndd, though we don't like to say that. ;-)
If you need to use a driver.conf for Solaris 10, that's OK I suppose
(although an ndd tunable would be better there too, since it doesn't
require the driver to be unloaded and reloaded to change the setting --
which can be very challenging for administrators to figure out.)
I feel TCR-strong on this -- if it were a full case I'd insist that this
be part of the spec before I'd vote to approve.
Is the project team amenable to making this change, or do they have some
other reason why driver.conf values need to be used instead.
Also, I'd like the mtu to be set via Brussels as well, if it isn't
already handled that way.
- Garrett
Ted Kim wrote:
> Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI
> This information is Copyright 2009 Sun Microsystems
> 1. Introduction
> 1.1. Project/Component Working Name:
> IPoIB Connected Mode
> 1.2. Name of Document Author/Supplier:
> Author: Kevin Ge
> 1.3 Date of This Document:
> 30 October, 2009
> 4. Technical Description
>
> A. Overview
> -----------
>
> This case proposes changes to the Solaris kernel to provide support
> for "Connected Mode" in the IPoIB driver ibd(7D) (described in [1]
> and [2]).
>
> The Infiniband Architecture [3] defines multiple "transport service
> types", including Unreliable Datagram (UD), Reliable Connected (RC)
> and Unreliable Connected (UC). Current ibd (based on [4]) runs in
> "Datagram Mode" over the UD transport service type. Connected Mode
> (described in [5]) can use either UC and/or RC.
>
> This IPoIB-CM project uses RC, because of the desire to
> inter-operate with Linux which also uses RC. The main advantage of
> Connected Mode is better performance (higher throughput and lower
> CPU utilization) based on using very large MTUs (see below for more
> discussion). Connected Mode, though, can have the disadvantage of
> consuming more resources, especially when scaling up to a large
> cluster (due to using an InfiniBand connection to each destination).
>
> Note that this case only covers all necessary changes to support
> IPoIB driver running in Connected Mode over RC. Other enhancements
> are outside the scope of this case.
>
> A micro/patch binding is asserted for this proposal.
>
> B. Connected Mode IPoIB driver
> ------------------------------
>
> The revised ibd(7D) driver will support both Connected and Datagram
> mode. The features from the current Datagram mode ibd driver will
> be inherited. The remainder of this section discusses interface
> additions for the Connected mode capable driver.
>
>
> B.1 Switching between datagram and connected mode
>
> The existing ibd driver in OpenSolaris and Solaris 10 does not
> ship with a driver .conf file. However, the Connected Mode support
> described in this case introduces a new parameter 'enable_rc' that
> may be set via the ibd driver .conf file.
>
> This parameter specifies whether each ibd instance defaults to
> using Connected Mode over RC or not.
>
> # 1: unicast packets will be sent over Reliable Connected Mode
> # 0: unicast packets will be sent over Unreliable Datagram Mode
> #
> # Each element in the list below maps to the corresponding ibd
> # instance; the first element is for ibd instance 0, the second
> # element is for instance 1 and so on.
> #
> enable_rc=1,1,0,0;
>
> Please note that Connected Mode support in IPoIB is optional as per
> [5]. Therefore, if Connected Mode is not available for a remote
> node, the Datagram mode will automatically be used for that
> destination by the ibd driver. Therefore, the only meaning of
> 'enable_rc' is to decide whether to try Connected Mode first or
> not, and whether to advertise this as a capability supported by
> this instance or not.
>
> The default value for 'enable_rc' for each instance is 0. Hence
> without a ibd.conf file, Datagram mode will be used. We intend to
> ship a driver .conf file for ibd in ONNV (and hence OpenSolaris)
> with enable_rc set to all ones (enabling Connected Mode by
> default on all instances) for the best performance.
>
> However, for Solaris 10, we have received business guidance to have
> an "opt-in" approach due to a desire for greater stability in
> established enterprise environments. We will do this by not
> shipping the .conf file. Therefore, by default Solaris 10 will be
> Datagram mode. It will take an explicit administrator action
> (setting enable_rc) to cause Solaris 10 to use Connected Mode.
>
> OFED (Linux IB) originally made Connected Mode opt-in too. However,
> later OFED made it the default. We don't intend to change it later
> to be the default in Solaris 10. However, Solaris Next, being
> descended from ONNV, will have it as default.
>
> An edited ibd(7D) manpage documenting this change is in the
> materials directory.
>
> B.2 Change of default MTU size
>
> Connected Mode by virtue of using the RC transport service type
> offers link MTUs of up to 2^31-4 octets in length. Thus, the use of
> Connected Mode can offer benefits by supporting very large MTUs.
> Datagram Mode using UD is limited to 4092 (4K-4) octets, though
> commonly only 2044 (2K-4) is offered.
>
> Due to the limits of the TCP/IP protocol, it makes sense to only
> offer up to 65535 (64K-1) bytes. OFED (i.e. Linux IB) uses 65520
> (64K-16) byte MTU for alignment reasons. To inter-operate with
> OFED at the best performance, we also adopt 65520 as the default
> MTU of the Connected Mode.
>
>
> C. Interfaces
> -------------
> +-------------------------------------------------------------------+
> | Interfaces Exported |
> +---------------------------+------------------+--------------------+
> | Interface Name | Classification | Comment |
> +---------------------------+------------------+--------------------+
> |/kernel/drv/ibd.conf* | Uncommitted | Configuration file |
> +---------------------------+------------------+--------------------+
> * = only for OpenSolaris
>
>
> D. References
> -------------
>
> [1] IP over InfiniBand, PSARC/2001/289
>
> [2] IPoIB Conversion to GLDv3, PSARC/2007/636
>
> [3] InfiniBand Architecture Specification Volume 1, Release 1.2.1,
> InfiniBand Trade Association, 2007.
> http://www.infinibandta.org/content/pages.php?pg=technology_download
>
> [4] Transmission of IP over InfiniBand (IPoIB), RFC 4391, IETF,
> http://www.ietf.org/rfc/rfc4391.txt
>
> [5] IP over InfiniBand: Connected Mode, RFC 4755, IETF,
> http://www.ietf.org/rfc/rfc4755.txt
>
> 6. Resources and Schedule
> 6.4. Steering Committee requested information
> 6.4.1. Consolidation C-team Name:
> ON
> 6.5. ARC review type: FastTrack
> 6.6. ARC Exposure: open
>
>
