On Mon, Sep 10, 2012 at 09:16:29AM +0300, Michael S. Tsirkin wrote: > On Mon, Sep 10, 2012 at 11:42:25AM +0930, Rusty Russell wrote: > > OK, I read the spec (pasted below for easy of reading), but I'm still > > confused over how this will work. > > > > I thought normal net drivers have the hardware provide an rxhash for > > each packet, and we map that to CPU to queue the packet on[1]. We hope > > that the receiving process migrates to that CPU, so xmit queue > > matches. > > This ony works sometimes. For example it's common to pin netperf to a > cpu to get consistent performance. Proper hardware must obey what > applications want it to do, not the other way around. > > > For virtio this would mean a new per-packet rxhash value, right? > > > > Why are we doing something different? What am I missing? > > > > Thanks, > > Rusty. > > [1] Everything I Know About Networking I Learned From LWN: > > https://lwn.net/Articles/362339/ > > I think you missed this: > > Some network interfaces can help with the distribution of incoming > packets; they have multiple receive queues and multiple interrupt lines. > Others, though, are equipped with a single queue, meaning that the > driver for that hardware must deal with all incoming packets in a > single, serialized stream. Parallelizing such a stream requires some > intelligence on the part of the host operating system. > > In other words RPS is a hack to speed up networking on cheapo > hardware, this is one of the reasons it is off by default. > Good hardware has multiple receive queues. > We can implement a good one so we do not need RPS. > > Also not all guest OS-es support RPS. > > Does this clarify?
I would like to add that on many processors, sending IPCs between guest CPUs requires exits on sending *and* receiving path, making it very expensive. > > --- > > Transmit Packet Steering > > > > When VIRTIO_NET_F_MULTIQUEUE feature bit is negotiated, guest can use any > > of multiple configured transmit queues to transmit a given packet. To avoid > > packet reordering by device (which generally leads to performance > > degradation) driver should attempt to utilize the same transmit virtqueue > > for all packets of a given transmit flow. For bi-directional protocols (in > > practice, TCP), a given network connection can utilize both transmit and > > receive queues. For best performance, packets from a single connection > > should utilize the paired transmit and receive queues from the same > > virtqueue pair; for example both transmitqN and receiveqN. This rule makes > > it possible to optimize processing on the device side, but this is not a > > hard requirement: devices should function correctly even when this rule is > > not followed. > > > > Driver selects an active steering rule using VIRTIO_NET_CTRL_STEERING > > command (this controls both which virtqueue is selected for a given packet > > for receive and notifies the device which virtqueues are about to be used > > for transmit). > > > > This command accepts a single out argument in the following format: > > > > #define VIRTIO_NET_CTRL_STEERING 4 > > > > The field rule specifies the function used to select transmit virtqueue for > > a given packet; the field param makes it possible to pass an extra > > parameter if appropriate. When rule is set to > > VIRTIO_NET_CTRL_STEERING_SINGLE (this is the default) all packets are > > steered to the default virtqueue transmitq (1); param is unused; this is > > the default. With any other rule, When rule is set to > > VIRTIO_NET_CTRL_STEERING_RX_FOLLOWS_TX packets are steered by driver to the > > first N=(param+1) multiqueue virtqueues transmitq1...transmitqN; the > > default transmitq is unused. Driver must have configured all these > > (param+1) virtqueues beforehand. > > > > Supported steering rules can be added and removed in the future. Driver > > should check that the request to change the steering rule was successful by > > checking ack values of the command. As selecting a specific steering is an > > optimization feature, drivers should avoid hard failure and fall back on > > using a supported steering rule if this command fails. The default steering > > rule is VIRTIO_NET_CTRL_STEERING_SINGLE. It will not be removed. > > > > When the steering rule is modified, some packets can still be outstanding > > in one or more of the transmit virtqueues. Since drivers might choose to > > modify the current steering rule at a high rate (e.g. adaptively in > > response to changes in the workload) to avoid reordering packets, device is > > recommended to complete processing of the transmit queue(s) utilized by the > > original steering before processing any packets delivered by the modified > > steering rule. > > > > For debugging, the current steering rule can also be read from the > > configuration space. > > > > Receive Packet Steering > > > > When VIRTIO_NET_F_MULTIQUEUE feature bit is negotiated, device can use any > > of multiple configured receive queues to pass a given packet to driver. > > Driver controls which virtqueue is selected in practice by configuring > > packet steering rule using VIRTIO_NET_CTRL_STEERING command, as described > > above[sub:Transmit-Packet-Steering]. > > > > The field rule specifies the function used to select receive virtqueue for > > a given packet; the field param makes it possible to pass an extra > > parameter if appropriate. When rule is set to > > VIRTIO_NET_CTRL_STEERING_SINGLE all packets are steered to the default > > virtqueue receiveq (0); param is unused; this is the default. When rule is > > set to VIRTIO_NET_CTRL_STEERING_RX_FOLLOWS_TX packets are steered by host > > to the first N=(param+1) multiqueue virtqueues receiveq1...receiveqN; the > > default receiveq is unused. Driver must have configured all these (param+1) > > virtqueues beforehand. For best performance for bi-directional flows (such > > as TCP) device should detect the flow to virtqueue pair mapping on transmit > > and select the receive virtqueue from the same virtqueue pair. For > > uni-directional flows, or when this mapping information is missing, a > > device-specific steering function is used. > > > > Supported steering rules can be added and removed in the future. Driver > > should probe for supported rules by checking ack values of the command. > > > > When the steering rule is modified, some packets can still be outstanding > > in one or more of the virtqueues. Device is not required to wait for these > > packets to be consumed before delivering packets using the new streering > > rule. Drivers modifying the steering rule at a high rate (e.g. adaptively > > in response to changes in the workload) are recommended to complete > > processing of the receive queue(s) utilized by the original steering before > > processing any packets delivered by the modified steering rule. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
