I'm submitting this fast-track for Cathy Zhou. The timer is set for
April 14th, 2008.
Data Fast-Path for Softmac
==========================
Release binding: Patch
Summary
=======
This case introduces a GLDv3 fast-path architecture for softmac devices.
The softmac architecture defined by PSARC/2006/499 initially contained
a fast-path which was not delivered with Clearview UV as discussed in
section 2.1 of PSARC/2008/002. PSARC/2008/002 states that an alternative
fast-path architecture would be defined in a future case, and that is
this case.
Details
=======
- Data fast-path and slow-path
Currently, the softmac module discovers each legacy network device
and registers (soft)MACs to the GLDv3 framework on behalf of these
legacy devices. Whenever a softMAC is used, the softmac module opens
a DLIOCRAW stream over the legacy device, and enables DL_PROMISC_SAP
promiscuous mode on the stream. This DLIOCRAW stream is then shared
by all data packets sent and received over this softMAC. We call this
DLIOCRAW stream "shared lower stream".
Although the above is consistent with the existing GLDv3 model and
requires less changes to the GLDv3 framework, it introduces
unacceptable performance regression over legacy devices, because of
the extra data demultiplexing and filtering processing in the GLDv3
DLS/MAC layer.
Therefore, this case proposes a softmac data fast-path architecture
which is less intrusive to the current Solaris network architecture
and will address the performance issue of the above approach:
Specifically, when the fast-path is used, a dedicated lower-stream
will be opened over the legacy device for each IP/ARP (upper-)stream
over the softMAC, and all DLPI messages (including control messages
and data messages) will be exchanged between the upper-stream and
the corresponding lower-stream directly. Therefore, the data
demultiplexing, filtering and classification processing will be done
by the lower-stream, and the GLDv3 DLS/MAC layer processing will be
no longer needed.
Note that some GLDv3 features requires the GLDv3 DLS/MAC layer
processing to not be bypassed to assure its function correctness.
For example, softmac fast-path must be disabled to support GLDv3
VNIC functionality. As the result, we propose: By default, the
softmac fast-path mode will be used to assure the performance; MAC
clients will be able to request to disable the softmac fast-path
mode to support certain features, and if that succeeds, the system
will fallback to the existing softmac data-path model (called slow-path).
The details of the new softmac data fast-path model is stated as below
(note that the softmac fast-path will only be used for the IP/ARP
streams over softMACs, which is the case performance matters most):
1. When a stream (including a VLAN stream) is opened on a softMAC,
the softmac module will takes over the DLPI processing on this
stream;
2. For IP/ARP streams over a softMAC, softmac data fast-path will be
used by default, unless fast-path is disabled by any MAC client
explicitly. The softmac module first identifies an IP/ARP stream
by seeing whether there is a SIOCSLIFNAME ioctl sent from upstream,
if there is one, this stream is either an IP or an ARP stream
and will use fast-path potentially;
3. When the softmac fast-path is used, an dedicated lower-stream will
be setup for each IP/ARP stream (1-1 mapping). From that point on,
all control and data messages will be exchanged between the IP/ARP
upper-stream and the legacy device through this dedicated
lower-stream. As a result, the DLS/MAC layer processing in GLDv3
will be skipped, and this greatly improves the performance;
4. When the softmac data fast-path is disabled by a MAC client (e.g.,
by a VNIC), all the IP/ARP upper streams will try to switch from
the fast-path to the slow-path. The dedicated lower-stream will be
destroyed, and all the control and data-messages will go through
the existing GLDv3 code path and (in the end) the shared
lower-stream;
5. On the other hand, when the last MAC client cancels its fast-path
disable request, all the IP/ARP streams will try to switch back to
the fast-path mode;
Step 5 and 6 both rely on the data-path mode switching process
described below:
1) To switch the softmac data-path mode (between fast-path and
slow-path), softmac will first send a DL_NOTE_REPLUMB
DL_NOTIFY_IND message upstream over each IP/ARP streams that
needs data-path mode switching;
2) When IP receives this DL_NOTE_REPLUMB message, it will bring
down all the IP interfaces on the corresponding ill (IP Lower
level structure), and bring up those interfaces over again;
this will in turn cause the ARP to "replumb" the interface.
During the replumb process, both IP and ARP will send
downstream the necessary DL_DISABMULTI_REQ and DL_UNBIND_REQ
messages and cleanup the old state of the underlying softMAC,
following with the necessary DL_BIND_REQ and DL_ENABMULTI_REQ
messages to setup the new state. Between the cleanup and re-setup
process, IP/ARP will also send down a DL_NOTE_REPLUMB_DONE
DL_NOTIFY_CONF messages to the softMAC to indicate the
*switching point*;
3) When softmac receives the DL_NOTE_REPLUMB_DONE message, it
either creates or destroys the dedicated lower-stream (depending
on which data-path mode the softMAC switches to), and change
the softmac data-path mode. From then on, softmac will process
all the succeeding control messages (including the DL_BIND_REQ and
DL_ENABMULTI_REQ messages) and data messages based on new
data-path mode.
Interfaces
==========
- mac_fastpath_disable()/mac_fastpath_enable() (Consolidation Private)
Two MAC client interfaces will be added:
int mac_fastpath_disable(mac_handle_t);
This function will be called when a mac client requires to disable
the softmac data fast-path over a specific mac. It will return errno
in case that the fast-path is failed to be disabled. If the given mac
is a mac over a GLDv3 device, 0 (success) will be directly returned.
void mac_fastpath_enable(mac_handle_t);
The mac_fastpath_enable() function will be called when a mac client
cancels its softmac fast-path disabling requests over a specific mac.
When the last fast-path disabling request is canceled, the underlying
softmac will try to reenable the data fast-path. It is not considered
as a failure if fast-path fails to be reenabled, as that will only
have performance impact.
- dld_str_open()/dld_str_close()/dld_str_private() (Project Private)
Three new dls interfaces will be added:
int dld_str_open(queue_t *, dev_t *, void *opaque);
This function will be called by softmac to setup the dld stream
related structures, which will be used when softmac fast-path is
not used. An opaque pointer will be passed into this function
to keep softmac specific state associated with the given stream.
int dld_str_close(queue_t *);
This function will be called by softmac to finish up the dld stream
operation and destroy dld related structures.
void *dld_str_private(queue_t *);
This function will be called to query the opaque pointer associated
with the given stream.
- the mac_capab_legacy_t structure (Project Private)
The mac_capab_legacy_t structure was introduced by PSARC/2008/002,
it is the data pointer of the MAC_CAPAB_LEGACY MAC capability.
This case proposes to add the following fields to the data structure:
/*
* Info and callbacks of legacy devices.
*/
typedef struct mac_capab_legacy_s {
...
int (*ml_fastpath_disable)(void *);
void (*ml_fastpath_enable)(void *);
boolean_t (*ml_active_set)(void *);
void (*ml_active_clear)(void *);
} mac_capab_legacy_t;
The ml_fastpath_enable() and ml_fastpath_disable() callbacks will be
called as the result of first mac_fastpath_disable() and last
mac_fastpath_enable() request on a particular softMAC.
Further, to assure there is at most one exclusive active MAC consumer
at a time (see [3]), we must prevent other exclusive active MAC
consumers when there is already IP/ARP stream on a softMAC, and prevent
IP/ARP streams if there is already exclusive active MAC consumers.
This is done by the mac module calling the ml_active_set() and
ml_active_clear() callbacks to inform softmac the existence and
disappearance of an exclusive active MAC consumer.
- the DL_NOTIFY_CONF primitive (M_PROTO type) (Project Private)
The DL_NOTIFY_CONF primitive is similar to DL_NOTIFY_IND, but it is
notification sent from upstream (from a DLPI consumer to a DLPI
provider).
DL_NOTIFY_CONF is a M_PROTO message with the following payload:
typedef struct {
t_uscalar_t dl_primitive; /* set to DL_NOTIFY_CONF */
uint32_t dl_notification; /* Which notification? */
} dl_notify_conf_t;
The only dl_notification type we will support is DL_NOTE_REPLUMB_DONE.
- the DL_NOTE_REPLUMB DL_NOTIFY_IND message (Project Private)
As described above, the DL_NOTE_REPLUMB message is sent by softmac
to inform the IP/ARP upstream to "replumb" the stream.
References
==========
[1] PSARC 2006/499 Clearview Nemo unification and vanity naming -
http://sac.sfbay/PSARC/2006/499/
[2] PSARC 2008/002 Clearview UV Updates - http://sac.sfbay/PSARC/2008/002/
[3] PSARC 2005/207 Clearview for Nemo - http://sac.sfbay/PSARC/2005/207/
