Garrett D'Amore wrote: > Sunay Tripathi wrote: >> Garrett D'Amore wrote: >> >>> Sunay Tripathi wrote: >>> >>>> Garrett D'Amore wrote: >>>> >>>>> Am I reading this correctly, in that polling for receive packets is >>>>> only used if the NIC has more than a single rx ring? >>>>> >>>> Yes. The issue is that we can't let one consumer (which can be a >>>> service or a VNIC tied to a virtual machine) own the entire NIC. >>>> So if the NIC has only one Rx ring, we never put that in poll >>>> mode and let the packets always fly till the S/W classifer and >>>> soft ring set (SRS) (which mimic Rx rings in pseudo H/W layer). At that >>>> point, the SRS can still be put in a polling mode since that is >>>> unique to the VNIC or the service. >>>> >>>> Now having said that, this is probably suboptimal to the small packet >>>> forwarding performance. We can still put the entire NIC in the >>>> poll mode but what would be a good way to figure this out (don't >>>> necessarily like the idea of adding a tunable). Can we do that >>>> by adding a property to the NIC? Is there a more intutive way? >>>> >>> When inbound packets arrive faster than the upper layers can process >>> them, this would be the point to do that. So when the squeues or soft >>> rings (or whatever the crossbow analog is) fill up, then may as well put >>> the NIC in polling mode. Take it back out of polling mode when you >>> catch back up (i.e. you are able to verify that that ring is no longer >>> full and the poll returns no packet.) >>> >> Its this determination thats hard. The inbound packet goes through the >> system through multiple paths and it has different processing overheads. >> The fact that one path can't keep up doesn't mean that entire NIC >> should stop because packets of different types can still be easily >> dealt by via a different path. In mixed traffic environments, its not >> easy to determine that we are struggling. Sometimes the user specified >> policies also play a role in making this hard. Assume the traffic >> for a Zone/Xen (determined via its mac address) is not getting processed >> because that particular Zone/Xen has very little CPU assigned to it. >> In such case, even though we can figure out that system can't keep >> up, we can't put the NIC in the polling mode. The NIC in such as case >> is a shared resource. >> >> >>> If there is no soft ring in the middle, then simply let the NIC tell >>> you that its hardware ring is full... perhaps the NIC driver can *ask* >>> to be put into polling mode in this situation? >>> >> You assumption is based on that all packets are meant for one consumer >> which is not necessarily true. >> > > > Is the problem that there is logical consumer to do the polling?
Correct. > What about in the software classification layer? > > If you assume that the problem is that the *hardware* ring is full, then > it doesn't matter *who* the traffic is for... the packets are arriving > in the hardware ring too fast, and the host cpu can't keep up with > them... even if all it has to do is shuffle them to another rx queue. I haven't noticed that to the issue. The bottle necks seems to be above that. If what you say is true, then we can do somethings between the S/W classification layers and the NIC. Worth investigating. In the small packet forwarding case, can you do some experiments to verify this? Sunay -- Sunay Tripathi Distinguished Engineer Solaris Core Operating System Sun MicroSystems Inc. Solaris Networking: http://www.opensolaris.org/os/community/networking Project Crossbow: http://www.opensolaris.org/os/project/crossbow
