Holy cow Simon!!! That did the trick!!! Thank you so much! I feel like I should send you coffee or beer or something :)
Steve On Fri, Jul 8, 2022 at 11:48 AM Simon Weller <[email protected]> wrote: > I recall an issue a while back where the Windows Virtio drivers didn't > support generic support offload (GRO) and it created some performance > problems with Cisco NICs. > This was particular to Windows and didn't occur for Linux VMs. > > Can you try disabling GRO on your NICs and retesting? > ethtool -K <interface> gro off > > -Si > ________________________________ > From: S.Fuller <[email protected]> > Sent: Thursday, July 7, 2022 9:00 AM > To: [email protected] <[email protected]> > Subject: Re: Asymmetric traffic issues? > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when > clicking links, opening attachments, or complying with requests. Click the > "Phish Alert Report" button above the email, or contact MIS, regarding any > suspicious message. > > > > The servers in question are in a Cisco UCS blade chassis. The physical NIC > card is a Cisco UCS VIC 1240, it's a multiport 10gb card that presents > itself multiple virtual NICs to the server. Those virtual NICs use the > Cisco enic driver. > > Steve > > On Fri, Jul 1, 2022 at 4:05 PM Simon Weller <[email protected]> > wrote: > > > ok, I managed to access your xml file via the mailing list archives. > > > > <interface type="bridge"> > > <mac address="02:00:69:fe:00:17"/> > > <source bridge="breth2-315"/> > > <target dev="vnet4"/> > > <model type="virtio"/> > > <link state="up"/> > > <alias name="net2"/> > > <address type="pci" domain="0x0000" bus="0x00" slot="0x03" > function="0x0"/> > > </interface> > > > > So, it does look correct in terms of virtio-net. One thing of note is > that > > kvm guests can't transmit or recieve packets in parallel, since > virtio-net > > only has one TX and RX queue. This tends to reduce performance on Windows > > guests. > > This can be addressed by enabling multi-queue (you assign a queue for > > vCPU). There was talk of this back in 2021 on the list, but I'm not sure > > whether it was implemented or not. > > > > What brand and chipset are your NICs on the servers? > > > > > > > > > > > > > > ________________________________ > > From: Simon Weller <[email protected]> > > Sent: Friday, July 1, 2022 2:34 PM > > To: [email protected] <[email protected]> > > Subject: Re: Asymmetric traffic issues? > > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when > > clicking links, opening attachments, or complying with requests. Click > the > > "Phish Alert Report" button above the email, or contact MIS, regarding > any > > suspicious message. > > > > > > > > I'm assuming the para-virtualized network driver is probably not being > > used either based on the template. > > If you take a close look at the xml for the interface, you should be able > > to confirm that or not. > > > > The templates tend to package all the para-virtualized drivers together, > > so they're normally in use, or they're not. > > > > > > ________________________________ > > From: S.Fuller <[email protected]> > > Sent: Friday, July 1, 2022 1:52 PM > > To: [email protected] <[email protected]> > > Subject: Re: Asymmetric traffic issues? > > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when > > clicking links, opening attachments, or complying with requests. Click > the > > "Phish Alert Report" button above the email, or contact MIS, regarding > any > > suspicious message. > > > > > > > > Will give that a try. I'm curious why you think changing the disk drivers > > over to virtio will have an effect on the network throughput. > > > > - Steve > > > > On Fri, Jul 1, 2022 at 10:55 AM Simon Weller <[email protected]> > > wrote: > > > > > Unfortunately, I can't open the xml via the mailing list, but based on > > the > > > template you're using, it's not setup for Virtio-blk or virtio-scsi. > > Change > > > the template to Virtio-SCSI 64bit and the correct interface drive type > > will > > > be specified in the XML. > > > > > > ________________________________ > > > From: S.Fuller <[email protected]> > > > Sent: Thursday, June 30, 2022 12:43 PM > > > To: [email protected] <[email protected]> > > > Subject: Re: Asymmetric traffic issues? > > > > > > > > > I'm using the Windows Server 2016 profile for the template. After the > VM > > > came up, I installed the virtio drivers from the latest ISO, and then > > added > > > a new NIC so that it used the virtio network drivers vs the default > E1000 > > > drivers. > > > > > > XML dump is attached > > > > > > On Thu, Jun 30, 2022 at 12:03 PM Simon Weller <[email protected] > > > > > wrote: > > > Can you provide a redacted dump of your libvirt xml file for the VM? > > > Also, which OS profile are you using in Cloudstack for this? The > Windows > > > specific ones don't use virtio if I remember correctly, so you will > need > > to > > > select Virtio-SCSI 64bit in order for the xml to be built correctly. > > > > > > -Si > > > ________________________________ > > > From: S.Fuller <[email protected]<mailto:[email protected]>> > > > Sent: Thursday, June 30, 2022 11:38 AM > > > To: [email protected]<mailto:[email protected]> < > > > [email protected]<mailto:[email protected]>> > > > Subject: Re: Asymmetric traffic issues? > > > > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution > when > > > clicking links, opening attachments, or complying with requests. Click > > the > > > "Phish Alert Report" button above the email, or contact MIS, regarding > > any > > > suspicious message. > > > > > > > > > > > > Neither the block nor the scsi drivers appear to be running as far as I > > can > > > tell. > > > > > > - Steve > > > > > > On Thu, Jun 30, 2022 at 11:07 AM Simon Weller <[email protected] > > > > > wrote: > > > > > > > Steve, > > > > > > > > Are you running the virtio-block or virtio-scsi drivers? > > > > > > > > -Si > > > > > > > > > > > > ________________________________ > > > > From: S.Fuller <[email protected]<mailto:[email protected]>> > > > > Sent: Thursday, June 30, 2022 8:52 AM > > > > To: [email protected]<mailto:[email protected]> > < > > > [email protected]<mailto:[email protected]>> > > > > Subject: Re: Asymmetric traffic issues? > > > > > > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution > > when > > > > clicking links, opening attachments, or complying with requests. > Click > > > the > > > > "Phish Alert Report" button above the email, or contact MIS, > regarding > > > any > > > > suspicious message. > > > > > > > > > > > > > > > > Well, I checked everything that I could for any QOS settings and > there > > is > > > > nothing configured there. What is curious is that Idon't see this > > > behavior > > > > when the transmitting host is running Linux, or if I'm using the > E1000 > > > > drivers (although with the E1000 driver, the overall throughput is > > > lower). > > > > It really feels like I'mrunning into some weird issue with the virtio > > > > drivers on Windows. My Windows hosts are (to my knowledge) using the > > > latest > > > > version of the virtio drivers - 100.90.104.21700 dated 2/23/2022. > > > > > > > > Steve Fuller > > > > [email protected]<mailto:[email protected]> > > > > > > > > On Wed, Jun 29, 2022 at 3:04 PM S.Fuller <[email protected] > > <mailto: > > > [email protected]>> wrote: > > > > > > > > > Vivek, > > > > > > > > > > Thanks for the reply. I am using the KVM hypervisor. I'll will > review > > > the > > > > > QoS on the hypervisor for both of the nodes. > > > > > > > > > > I'm checking throughput between two different VMs that running on > two > > > > > different hosts within the same cluster. As of right now, I'm > > receiving > > > > > similar results using both iperf3 and nuttcp as the testing tools. > We > > > are > > > > > only seeing this issue when the VM is not on the same host as the > > > vrouter > > > > > for its isolated network. > > > > > > > > > > On Tue, Jun 28, 2022 at 6:40 AM Vivek Kumar > > > > > <[email protected]<mailto:[email protected]>.invalid> > > > wrote: > > > > > > > > > >> Hey Fuller, > > > > >> > > > > >> What hypervisor are you using ? I know you have checked all > > bandwidth > > > > >> limit on templates and global settings, but it’s worth to check > the > > > QoS > > > > on > > > > >> the hypervisor level, because at the end it’s the hypervisor which > > > > manages > > > > >> all. And from where are you trying to check the network > throughout, > > > > >> between client and server ? > > > > >> > > > > >> > > > > >> Vivek Kumar > > > > >> Sr. Manager - Cloud & DevOps > > > > >> TechOps | Indiqus Technologies > > > > >> > > > > >> + 91 7503460090 <tel:++91+7503460090> > > > > >> [email protected]<mailto:[email protected]> > > > <mailto:[email protected]<mailto:[email protected]>> > > > > >> www.indiqus.com<http://www.indiqus.com>< > > > http://www.indiqus.com> < > > > > https://www.indiqus.com/> > > > > >> > > > > >> > > > > >> > > > > >> > On 28-Jun-2022, at 1:58 AM, S.Fuller <[email protected] > > <mailto: > > > [email protected]>> wrote: > > > > >> > > > > > >> > Environment: > > > > >> > > > > > >> > Two physical hosts > > > > >> > - Cloudstack 4.11.3 > > > > >> > - Verified that there are no bandwidth limits in place on any of > > the > > > > >> > templates or in global settings. > > > > >> > > > > > >> > Two isolated networks ("Client" and "Server") > > > > >> > - Each has a vrouter with a public and private address > > > > >> > - One Windows 2016 VM on each network (running the latest virtio > > > > >> drivers) > > > > >> > - each node running latest version of Iperf3 to test throughput > > > > >> > > > > > >> > Testing/Observation: > > > > >> > > > > > >> > If the Client VM and the vrouter for the isolated Client network > > are > > > > on > > > > >> the > > > > >> > same physical host, we see symmetrical throughput in the 2 Gbps > > > range, > > > > >> > whether we run iperf in regular mode or in reverse mode (iperf > > -R). > > > > >> > > > > > >> > If the Client VM and the vrouter for the isolated Client network > > are > > > > on > > > > >> > different physical hosts, we are seeing 25% of the throughput > > > running > > > > >> iperf > > > > >> > in regular mode vs running it in reverse mode. > > > > >> > > > > > >> > Has anyone encountered this issue before? If we change the > Client > > VM > > > > to > > > > >> > Linux (either CentOS 7 or Ubuntu) OR we use the E1000 driver, we > > see > > > > >> > symmetrical throughput in our tests, no matter where the vrouter > > is > > > in > > > > >> > relation to the Client VM. > > > > >> > > > > > >> > -- > > > > >> > Steve Fuller > > > > >> > [email protected]<mailto:[email protected]> > > > > >> > > > > >> > > > > >> -- > > > > >> This message is intended only for the use of the individual or > > entity > > > to > > > > >> which it is addressed and may contain confidential and/or > privileged > > > > >> information. If you are not the intended recipient, please delete > > the > > > > >> original message and any copy of it from your computer system. You > > are > > > > >> hereby notified that any dissemination, distribution or copying of > > > this > > > > >> communication is strictly prohibited unless proper authorization > has > > > > been > > > > >> obtained for such action. If you have received this communication > in > > > > >> error, > > > > >> please notify the sender immediately. Although IndiQus attempts to > > > sweep > > > > >> e-mail and attachments for viruses, it does not guarantee that > both > > > are > > > > >> virus-free and accepts no liability for any damage sustained as a > > > result > > > > >> of > > > > >> viruses. > > > > >> > > > > > > > > > > > > > > > -- > > > > > Steve Fuller > > > > > [email protected]<mailto:[email protected]> > > > > > > > > > > > > > > > > > -- > > > > Steve Fuller > > > > [email protected]<mailto:[email protected]> > > > > > > > > > > > > > -- > > > Steve Fuller > > > [email protected]<mailto:[email protected]> > > > > > > > > > -- > > > Steve Fuller > > > [email protected]<mailto:[email protected]> > > > > > > > > > -- > > Steve Fuller > > [email protected] > > > > > -- > Steve Fuller > [email protected] > -- Steve Fuller [email protected]
