Re: Regression in throughput between kvm guests over virtual bridge

On 09/13/2017 04:13 AM, Jason Wang wrote: > > > On 2017年09月13日 09:16, Jason Wang wrote: >> >> >> On 2017年09月13日 01:56, Matthew Rosato wrote: >>> We are seeing a regression for a subset of workloads across KVM guests >>> over a virtual bridge

Re: Regression in throughput between kvm guests over virtual bridge

> It looks like vhost is slowed down for some reason which leads to more > idle time on 4.13+VHOST_RX_BATCH=1. Appreciated if you can collect the > perf.diff on host, one for rx and one for tx. > perf data below for the associated vhost threads, baseline=4.12, delta1=4.13,

Re: Regression in throughput between kvm guests over virtual bridge

> Is the issue gone if you reduce VHOST_RX_BATCH to 1? And it would be > also helpful to collect perf diff to see if anything interesting. > (Consider 4.4 shows more obvious regression, please use 4.4). > Issue still exists when I force VHOST_RX_BATCH = 1 Collected perf data, with 4.12 as the

Re: Regression in throughput between kvm guests over virtual bridge

On 09/18/2017 03:36 AM, Jason Wang wrote: > > > On 2017年09月18日 11:13, Jason Wang wrote: >> >> >> On 2017年09月16日 03:19, Matthew Rosato wrote: >>>> It looks like vhost is slowed down for some reason which leads to more >>>> idle time on 4.

Re: Regression in throughput between kvm guests over virtual bridge

> Seems to make some progress on wakeup mitigation. Previous patch tries > to reduce the unnecessary traversal of waitqueue during rx. Attached > patch goes even further which disables rx polling during processing tx. > Please try it to see if it has any difference. Unfortunately, this patch

Regression in throughput between kvm guests over virtual bridge

We are seeing a regression for a subset of workloads across KVM guests over a virtual bridge between host kernel 4.12 and 4.13. Bisecting points to c67df11f "vhost_net: try batch dequing from skb array" In the regressed environment, we are running 4 kvm guests, 2 running as uperf servers and 2

Re: Regression in throughput between kvm guests over virtual bridge

On 09/22/2017 12:03 AM, Jason Wang wrote: > > > On 2017年09月21日 03:38, Matthew Rosato wrote: >>> Seems to make some progress on wakeup mitigation. Previous patch tries >>> to reduce the unnecessary traversal of waitqueue during rx. Attached >>> patch goes ev

Re: Regression in throughput between kvm guests over virtual bridge

On 09/25/2017 04:18 PM, Matthew Rosato wrote: > On 09/22/2017 12:03 AM, Jason Wang wrote: >> >> >> On 2017年09月21日 03:38, Matthew Rosato wrote: >>>> Seems to make some progress on wakeup mitigation. Previous patch tries >>>> to reduce the unnecessa

Re: Regression in throughput between kvm guests over virtual bridge

On 10/12/2017 02:31 PM, Wei Xu wrote: > On Thu, Oct 05, 2017 at 04:07:45PM -0400, Matthew Rosato wrote: >> >> Ping... Jason, any other ideas or suggestions? > > Hi Matthew, > Recently I am doing similar test on x86 for this patch, here are some, > differences b

Re: Regression in throughput between kvm guests over virtual bridge

On 11/12/2017 01:34 PM, Wei Xu wrote: > On Sat, Nov 11, 2017 at 03:59:54PM -0500, Matthew Rosato wrote: >>>> This case should be quite similar with pkgten, if you got improvement with >>>> pktgen, usually it was also the same for UDP, could you please try to >>&

Re: Regression in throughput between kvm guests over virtual bridge

On 11/27/2017 08:36 PM, Jason Wang wrote: > > > On 2017年11月28日 00:21, Wei Xu wrote: >> On Mon, Nov 20, 2017 at 02:25:17PM -0500, Matthew Rosato wrote: >>> On 11/14/2017 03:11 PM, Matthew Rosato wrote: >>>> On 11/12/2017 01:34 PM, Wei Xu wrote: >>&g

Re: Regression in throughput between kvm guests over virtual bridge

>> This case should be quite similar with pkgten, if you got improvement with >> pktgen, usually it was also the same for UDP, could you please try to disable >> tso, gso, gro, ufo on all host tap devices and guest virtio-net devices? >> Currently >> the most significant tests would be like this

Re: Regression in throughput between kvm guests over virtual bridge

On 11/14/2017 03:11 PM, Matthew Rosato wrote: > On 11/12/2017 01:34 PM, Wei Xu wrote: >> On Sat, Nov 11, 2017 at 03:59:54PM -0500, Matthew Rosato wrote: >>>>> This case should be quite similar with pkgten, if you got improvement with >>>>> pktgen, usually

Re: Regression in throughput between kvm guests over virtual bridge

On 11/04/2017 07:35 PM, Wei Xu wrote: > On Fri, Nov 03, 2017 at 12:30:12AM -0400, Matthew Rosato wrote: >> On 10/31/2017 03:07 AM, Wei Xu wrote: >>> On Thu, Oct 26, 2017 at 01:53:12PM -0400, Matthew Rosato wrote: >>>> >>>>> >>>>> Are y

Re: Regression in throughput between kvm guests over virtual bridge

On 10/31/2017 03:07 AM, Wei Xu wrote: > On Thu, Oct 26, 2017 at 01:53:12PM -0400, Matthew Rosato wrote: >> >>> >>> Are you using the same binding as mentioned in previous mail sent by you? it >>> might be caused by cpu convention between pktgen and vhost, co

Re: [PATCH net,stable v4 0/3] vhost: fix a few skb leaks

On 12/01/2017 09:47 AM, Michael S. Tsirkin wrote: > On Fri, Dec 01, 2017 at 05:10:35AM -0500, w...@redhat.com wrote: >> From: Wei Xu >> >> Matthew found a roughly 40% tcp throughput regression with commit >> c67df11f(vhost_net: try batch dequing from skb array) as discussed >> in

Re: Regression in throughput between kvm guests over virtual bridge

On 10/23/2017 09:57 AM, Wei Xu wrote: > On Wed, Oct 18, 2017 at 04:17:51PM -0400, Matthew Rosato wrote: >> On 10/12/2017 02:31 PM, Wei Xu wrote: >>> On Thu, Oct 05, 2017 at 04:07:45PM -0400, Matthew Rosato wrote: >>>> >>>> Ping... Jason, any othe

Re: Regression in throughput between kvm guests over virtual bridge

On 10/22/2017 10:06 PM, Jason Wang wrote: > > > On 2017年10月19日 04:17, Matthew Rosato wrote: >>> 2. It might be useful to short the traffic path as a reference, What >>> I am running >>> is briefly like: >>> pktgen(host kernel) -> tap(x)

Re: Regression in throughput between kvm guests over virtual bridge

> > Are you using the same binding as mentioned in previous mail sent by you? it > might be caused by cpu convention between pktgen and vhost, could you please > try to run pktgen from another idle cpu by adjusting the binding? I don't think that's the case -- I can cause pktgen to hang in the