Hello Jan, > On 3 Mar 2017, at 09:58, Jan Scheurich <[email protected] > <mailto:[email protected]>> wrote: > > Hi Rafael, > > Some answers to your questions: > > 1. You are probably right that OpenStack Mitaka in principle supports > assigning one vhost queue per vCPU of an instance, but since this > requires support in VNFs we cannot utilize this in general.
Okay, thanks for letting me know. > Some VNFs we need to support with our NFV Infrastructure are not able to > deal with multiple vhost queues or, if they can, may not distribute > traffic evenly over multiple TX queues. That's is why vhost-multique is > not a general solution to the problem we see with short virtio queues. I see. Just separating the problems and each "problem" competency. For the throughtput (not the packet loss) It is not improbable that upstream would tell you to use multiqueue for that problem - changing VNFs - instead of changing vring buffer size. I totally understand the situation about the multiqueue. I'm working (reading/researching) in this part right now to be honest (since I'm waiting the testing environment). > 2. The run-time behavior of VNFs on Qemu 2.5 has degraded compared to Qemu 2.2 and 2.3. The increased burstiness of TX traffic is much more likely to overrun the short 256 slot virtio queues, which leads to the increase of packet drops at lower traffic rates. Yes, what led us to think about the queue flush scheduling issue because of some change. > But even with Qemu 2.0 certain VNFs drop TX packets at traffic rates > well below the maximum vSwitch throughput because of too short TX > queues. We have seen a 30% increase in throughput at same packet drop > level between Qemu 2.5 with 1024 queue slots and Qemu 2.2 with the > original 256 queue slots, which indicates that the original queue size > is underdimensioned. This is the tricky part. You can always maintain your own QEMU package - by patching it with your own changes every release - if upstream doesn't accept this change (because they would enter the TX vring buffer size discussion for virtio). Upstream would likely give you some ways to "solve" this problem and you wouldn't be able to comply because you might not be able to change your VNC to take advantage of those "new features" (from underlaying virtio hypervisor/driver code). I still have to study more this topic, I'm reading the code. Will likely need some more days here. > 3. We agree that your bisection approach is the only way to find the commit between Qemu 2.3 and 2.5 that is responsible for the increased burstiness. Then we can assess if this is bug, or an avoidable consequence of some new feature implemented in Qemu 2.4 or 2.5 and decide on the right upstreaming strategy for this. With the 1K queue length option in place fixing this is clearly no longer as critical. That is why I was insisting so much on the bisection. If i just provide you a fix for the throughput - that couldn't even be accepted as a SRU for our package - the original issue - packet drops - would loose importance. I'm glad we are differentiating both and following different path. The throughput is not guaranteed BUT it might be enlightened after he packet drops resolution. > It is not guaranteed that all of the 12 intermediate commits picked by > the bisect procedure will be fully working in end-to-end NFV context, > though. We might need to try out more commits than 12 to hunt down the > guilty one. When we have the test channel ready for this procedure, we > will find out. If this happens I can always do a bisect skip and we check other commits. Thank you for the effort on the test channel. > 4. For the above reasons we really want to make the virtio queue length configurable in rx and tx direction in upstream Qemu up to the limit of 1024 as earlier proposed by Patrick Hermansson (https://patchwork.ozlabs.org/patch/549544/ <https://patchwork.ozlabs.org/patch/549544/>). This can be done per port or by a global default configuration option. Yes, this is part of the "upstream" issue - which I'm reading/preparing myself better. If we could raise vring after device was online we could create a QOM command to raise the buffer. Problem is that this is done during device probe time (virtio_realize phase). With a QOM command, even with openstack/libvirt not supporting it, we could do it for oneline instances on the hypervisor. I'm leaving this for the end in my priority list. I'm glad we are synchronised. Thanks Jan for the comments!! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1668829 Title: Performance regression from qemu 2.3 to 2.5 for vhost-user with ovs + dpdk To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1668829/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
