As a general architectural comment here, I want to point out that I feel
that this case introduces additional complexity, that should be
unnecessary -- performance critical NICs should be written to GLDv3.
(The fact that GLDv3's API is still private is another matter altogether
-- but I believe introducing this code to address a "performance issue"
with a certain NIC product, is the wrong way to go here.)
Far far better, IMO, is to "fix" the main driver this case is for
(cassini), which architecturally could easily be converted to GLDv3 and
bundled into Nevada, and then spend effort to wrap up GLDv3 enough to
publish it for 3rd parties.
-- Garrett
Sebastien Roy wrote:
> I'm submitting this fast-track for Cathy Zhou. The timer is set for
> April 14th, 2008.
>
> Data Fast-Path for Softmac
> ==========================
>
> Release binding: Patch
>
> Summary
> =======
>
> This case introduces a GLDv3 fast-path architecture for softmac
> devices.
> The softmac architecture defined by PSARC/2006/499 initially
> contained
> a fast-path which was not delivered with Clearview UV as discussed in
> section 2.1 of PSARC/2008/002. PSARC/2008/002 states that an
> alternative
> fast-path architecture would be defined in a future case, and that is
> this case.
>
> Details
> =======
>
> - Data fast-path and slow-path
>
> Currently, the softmac module discovers each legacy network device
> and registers (soft)MACs to the GLDv3 framework on behalf of these
> legacy devices. Whenever a softMAC is used, the softmac module
> opens
> a DLIOCRAW stream over the legacy device, and enables
> DL_PROMISC_SAP
> promiscuous mode on the stream. This DLIOCRAW stream is then shared
> by all data packets sent and received over this softMAC. We call
> this
> DLIOCRAW stream "shared lower stream".
>
> Although the above is consistent with the existing GLDv3 model and
> requires less changes to the GLDv3 framework, it introduces
> unacceptable performance regression over legacy devices, because of
> the extra data demultiplexing and filtering processing in the GLDv3
> DLS/MAC layer.
>
> Therefore, this case proposes a softmac data fast-path architecture
> which is less intrusive to the current Solaris network architecture
> and will address the performance issue of the above approach:
>
> Specifically, when the fast-path is used, a dedicated lower-stream
> will be opened over the legacy device for each IP/ARP
> (upper-)stream
> over the softMAC, and all DLPI messages (including control messages
> and data messages) will be exchanged between the upper-stream and
> the corresponding lower-stream directly. Therefore, the data
> demultiplexing, filtering and classification processing will be
> done
> by the lower-stream, and the GLDv3 DLS/MAC layer processing will be
> no longer needed.
>
> Note that some GLDv3 features requires the GLDv3 DLS/MAC layer
> processing to not be bypassed to assure its function correctness.
> For example, softmac fast-path must be disabled to support GLDv3
> VNIC functionality. As the result, we propose: By default, the
> softmac fast-path mode will be used to assure the performance; MAC
> clients will be able to request to disable the softmac fast-path
> mode to support certain features, and if that succeeds, the system
> will fallback to the existing softmac data-path model (called
> slow-path).
>
> The details of the new softmac data fast-path model is stated as
> below
> (note that the softmac fast-path will only be used for the IP/ARP
> streams over softMACs, which is the case performance matters most):
>
> 1. When a stream (including a VLAN stream) is opened on a softMAC,
> the softmac module will takes over the DLPI processing on this
> stream;
>
> 2. For IP/ARP streams over a softMAC, softmac data fast-path
> will be
> used by default, unless fast-path is disabled by any MAC client
> explicitly. The softmac module first identifies an IP/ARP stream
> by seeing whether there is a SIOCSLIFNAME ioctl sent from
> upstream,
> if there is one, this stream is either an IP or an ARP stream
> and will use fast-path potentially;
>
> 3. When the softmac fast-path is used, an dedicated lower-stream
> will
> be setup for each IP/ARP stream (1-1 mapping). From that
> point on,
> all control and data messages will be exchanged between the
> IP/ARP
> upper-stream and the legacy device through this dedicated
> lower-stream. As a result, the DLS/MAC layer processing in GLDv3
> will be skipped, and this greatly improves the performance;
>
> 4. When the softmac data fast-path is disabled by a MAC client
> (e.g.,
> by a VNIC), all the IP/ARP upper streams will try to switch from
> the fast-path to the slow-path. The dedicated lower-stream
> will be
> destroyed, and all the control and data-messages will go through
> the existing GLDv3 code path and (in the end) the shared
> lower-stream;
>
> 5. On the other hand, when the last MAC client cancels its
> fast-path
> disable request, all the IP/ARP streams will try to switch
> back to
> the fast-path mode;
>
> Step 5 and 6 both rely on the data-path mode switching process
> described below:
>
> 1) To switch the softmac data-path mode (between fast-path and
> slow-path), softmac will first send a DL_NOTE_REPLUMB
> DL_NOTIFY_IND message upstream over each IP/ARP streams that
> needs data-path mode switching;
>
> 2) When IP receives this DL_NOTE_REPLUMB message, it will bring
> down all the IP interfaces on the corresponding ill (IP Lower
> level structure), and bring up those interfaces over again;
> this will in turn cause the ARP to "replumb" the interface.
>
> During the replumb process, both IP and ARP will send
> downstream the necessary DL_DISABMULTI_REQ and DL_UNBIND_REQ
> messages and cleanup the old state of the underlying softMAC,
> following with the necessary DL_BIND_REQ and DL_ENABMULTI_REQ
> messages to setup the new state. Between the cleanup and
> re-setup
> process, IP/ARP will also send down a DL_NOTE_REPLUMB_DONE
> DL_NOTIFY_CONF messages to the softMAC to indicate the
> *switching point*;
>
> 3) When softmac receives the DL_NOTE_REPLUMB_DONE message, it
> either creates or destroys the dedicated lower-stream
> (depending
> on which data-path mode the softMAC switches to), and change
> the softmac data-path mode. From then on, softmac will
> process
> all the succeeding control messages (including the
> DL_BIND_REQ and
> DL_ENABMULTI_REQ messages) and data messages based on new
> data-path mode.
>
> Interfaces
> ==========
>
> - mac_fastpath_disable()/mac_fastpath_enable() (Consolidation
> Private)
>
> Two MAC client interfaces will be added:
>
> int mac_fastpath_disable(mac_handle_t);
>
> This function will be called when a mac client requires to disable
> the softmac data fast-path over a specific mac. It will return
> errno
> in case that the fast-path is failed to be disabled. If the
> given mac
> is a mac over a GLDv3 device, 0 (success) will be directly
> returned.
>
> void mac_fastpath_enable(mac_handle_t);
>
> The mac_fastpath_enable() function will be called when a mac client
> cancels its softmac fast-path disabling requests over a specific
> mac.
> When the last fast-path disabling request is canceled, the
> underlying
> softmac will try to reenable the data fast-path. It is not
> considered
> as a failure if fast-path fails to be reenabled, as that will only
> have performance impact.
>
> - dld_str_open()/dld_str_close()/dld_str_private() (Project Private)
>
> Three new dls interfaces will be added:
>
> int dld_str_open(queue_t *, dev_t *, void *opaque);
>
> This function will be called by softmac to setup the dld stream
> related structures, which will be used when softmac fast-path is
> not used. An opaque pointer will be passed into this function
> to keep softmac specific state associated with the given stream.
>
> int dld_str_close(queue_t *);
>
> This function will be called by softmac to finish up the dld stream
> operation and destroy dld related structures.
>
> void *dld_str_private(queue_t *);
>
> This function will be called to query the opaque pointer associated
> with the given stream.
>
> - the mac_capab_legacy_t structure (Project Private)
>
> The mac_capab_legacy_t structure was introduced by PSARC/2008/002,
> it is the data pointer of the MAC_CAPAB_LEGACY MAC capability.
>
> This case proposes to add the following fields to the data
> structure:
>
> /*
> * Info and callbacks of legacy devices.
> */
> typedef struct mac_capab_legacy_s {
> ...
> int (*ml_fastpath_disable)(void *);
> void (*ml_fastpath_enable)(void *);
> boolean_t (*ml_active_set)(void *);
> void (*ml_active_clear)(void *);
> } mac_capab_legacy_t;
>
> The ml_fastpath_enable() and ml_fastpath_disable() callbacks
> will be
> called as the result of first mac_fastpath_disable() and last
> mac_fastpath_enable() request on a particular softMAC.
>
> Further, to assure there is at most one exclusive active MAC
> consumer
> at a time (see [3]), we must prevent other exclusive active MAC
> consumers when there is already IP/ARP stream on a softMAC, and
> prevent
> IP/ARP streams if there is already exclusive active MAC consumers.
> This is done by the mac module calling the ml_active_set() and
> ml_active_clear() callbacks to inform softmac the existence and
> disappearance of an exclusive active MAC consumer.
>
> - the DL_NOTIFY_CONF primitive (M_PROTO type) (Project Private)
>
> The DL_NOTIFY_CONF primitive is similar to DL_NOTIFY_IND, but it is
> notification sent from upstream (from a DLPI consumer to a DLPI
> provider).
>
> DL_NOTIFY_CONF is a M_PROTO message with the following payload:
>
> typedef struct {
> t_uscalar_t dl_primitive; /* set to DL_NOTIFY_CONF */
> uint32_t dl_notification; /* Which notification? */
> } dl_notify_conf_t;
>
> The only dl_notification type we will support is
> DL_NOTE_REPLUMB_DONE.
>
> - the DL_NOTE_REPLUMB DL_NOTIFY_IND message (Project Private)
>
> As described above, the DL_NOTE_REPLUMB message is sent by softmac
> to inform the IP/ARP upstream to "replumb" the stream.
>
> References
> ==========
>
> [1] PSARC 2006/499 Clearview Nemo unification and vanity naming -
> http://sac.sfbay/PSARC/2006/499/
> [2] PSARC 2008/002 Clearview UV Updates -
> http://sac.sfbay/PSARC/2008/002/
> [3] PSARC 2005/207 Clearview for Nemo -
> http://sac.sfbay/PSARC/2005/207/
