Hi Venu, venugopal iyer wrote: > Following is a short description on assigning priorities to a link (via > dladm). Flows will have a similar mechanism (via flowadm). > > thanks, > > -venu > > ---------------------------------------------------------------------------- > > The components associated with a link or flow, such as Soft Ring Sets, > Soft Rings and squeues, have kernel threads processing inbound/outbound > packets. The scheduler processes threads based on a thread's priority > level. > will link/flow priority be static or dynamic (can re-adjust as per flows/links get added)? > The priority level of a thread determine how it is scheduled w.r.t other > kernel threads. Thus by assigning a priority to a link/flow, we can control > how the threads associated with that link/flow is processed w.r.t other > link/flows. When the system is busy, threads with higher priority will have > a greater slice of CPU time, than the ones with lower priority. > > The kernel provides priority range between MINCLSYSPRI (60) and MAXCLSYSPRI > (100) for threads in the system class. > > We define three levels of priority - low, medium and high within the range > MINCLSYSPRI and MAXCLSYSPRI. The priority level can be assigned when creating > a virtual link (say VNIC) or can be modified for an existing link using > dladm(1m). The priority level is a property of the link. > > Setting a priority essentially means setting priorities for the following > elements associated with a MAC client[1], specifically: > > Soft Ring Set(s), SRS, (Rx and Tx): > (srs_pri) > o Worker thread > o Poll Thread (Rx) > o Associated Soft Ring(s) > -> Worker Thread > -> Poll Thread (Rx) > -> squeue > (sq_priority) > worker thread > poll thread (Rx) > > > Setting a Default Priority > -------------------------- > There are two ways a MAC client can be opened, either implicitly when plumbing > a physical link, e.g. bge0 (by DLS) or explicitly when creating a virtual > link, e.g VNIC. In the case of the latter, it is quite possible that the > properties of the link, including priority, is present when the MAC client > is created. > in case of link priorities, would it make sense to take into account the bandwidth associated with link? (cause, the thread servicing link with higher bandwidth cap, would make sense to have higher priority). > When a MAC client is created, we always set a default priority. The default > priority is set to be medium. Subsequently, the priority can be modified, > if specified (e.g. in the case of a VNIC), as part of creating the virtual > link itself. > > We add the following members to mac_client_impl_t > > pri_t mci_pri; > pri_t mci_max_pri; > pri_t mci_min_pri; > > why min_pri and max_pri range is different per mac_client_imp based? are there any min/max bandwidth ranges as well and there is any association with above priority ranges?
> where mci_pri is the priority assigned with the MAC client. mci_min_pri and > mci_max_pri is the available range corresponding to the priority for > the MAC client. The range will be used for any flows that are created on > top of the MAC client. > > are the min, max ranges dynamic/static? what could be the deciding factor to calculate min/max ranges for given mac_impl? can you have this min,max ranges static based on type whether NIC, VNIC or FLOW? > Given a range min and max, we assign low, medium and high value as follows: > > priority = ((min) + ((((max) - (min)) / MAC_PRI_LEVELS) * ((pri) + > 1))) > > where MAC_PRI_LEVELS is 3 and pri is 0 (low), 1 (medium) or 2 (high). > Additionally, we obtain the range for the given priority as: > > min_pri = ((min) + ((((max) - (min)) / MAC_PRI_LEVELS) * (pri))) > > max_pri = ((min_pri) + (((max) - (min)) / MAC_PRI_LEVELS)) > > for e.g.: when we create a MAC client given the range of MINCLSYSPRI and > MAXCLSYSPRI, we set a default priority of medium, which translates to the > following for the MAC client: > > mci_pri = 86 > mci_min_pri = 73 > mci_max_pri = 86 > > All the threads associated with the MAC client elements (listed above) will > have a thread priority of 86. you mean vnics and physical NICs? > Any flow created on the MAC client will now > derive its priority value using the specified priority (low, medium and high) > for that flow within the range 73 and 86. > so flows created on NICs or VNICs will have priority upto NICs or vnics existing priority. so threads processing packets on flows will have priorities lower than thread processing packets deposited on vnics? > If we create a virtual link, say a VNIC, with a specific priority, say high, > mac_client_open() will initialize the values for the VNIC's MAC client with > the default priority. I think we would dis-honor user's request of a high priority in that case? I may be wrong. if it is not in the range, do we error out? what should be the default priority in this case? > Subsequently, mac_resource_ctl_set() at what events, it would get called? > will result in > modifying the priority values for the MAC client to the value corresponding > to high. > > Modifying Priority > ------------------ > > A set-linkprop after the MAC client has been opened can be used to modify > the priority of a link. The result is a new priority value and range for > the MAC client based on the new priority. Additionally, any flows on top > of the MAC client will also need to be reprioritized as the range for the > MAC client will change. > what events could trigger re-adjusting the thread priorities? I mean, can user change it through flowadm or dladm? can addition/removal of more vnics or flows on given NICs could cause this change? or change of bandwidth on given link or flow cause this? if, there are multiple threads with equal priorities on the system, does solaris scheduler is designed to resolve this contention, does it fall back on say round-robin etc? (I understand, scope of your changes for crossbow is limited to assigning and maintaining right priorities, but just curious. :-)) > Setting priority before opening a MAC client > --------------------------------------------- > > A set-linkprop before a MAC client has been created will require the priority > values (in fact, any property) to be cached in the mac_impl_t. When the > primary MAC client is opened, the associated properties can be set on the > newly created MAC client post mac_client_open(). > would above initialize the priority to be used, or max priority limit? -Deepti > _______________________________________________________________________________ > References: > > [1]http://opensolaris.org/os/project/crossbow/Docs/crossbow-virt.pdf > > _______________________________________________ > crossbow-discuss mailing list > crossbow-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >
