The servers in question are in a Cisco UCS blade chassis. The physical NIC card is a Cisco UCS VIC 1240, it's a multiport 10gb card that presents itself multiple virtual NICs to the server. Those virtual NICs use the Cisco enic driver.
Steve On Fri, Jul 1, 2022 at 4:05 PM Simon Weller <[email protected]> wrote: > ok, I managed to access your xml file via the mailing list archives. > > <interface type="bridge"> > <mac address="02:00:69:fe:00:17"/> > <source bridge="breth2-315"/> > <target dev="vnet4"/> > <model type="virtio"/> > <link state="up"/> > <alias name="net2"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/> > </interface> > > So, it does look correct in terms of virtio-net. One thing of note is that > kvm guests can't transmit or recieve packets in parallel, since virtio-net > only has one TX and RX queue. This tends to reduce performance on Windows > guests. > This can be addressed by enabling multi-queue (you assign a queue for > vCPU). There was talk of this back in 2021 on the list, but I'm not sure > whether it was implemented or not. > > What brand and chipset are your NICs on the servers? > > > > > > > ________________________________ > From: Simon Weller <[email protected]> > Sent: Friday, July 1, 2022 2:34 PM > To: [email protected] <[email protected]> > Subject: Re: Asymmetric traffic issues? > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when > clicking links, opening attachments, or complying with requests. Click the > "Phish Alert Report" button above the email, or contact MIS, regarding any > suspicious message. > > > > I'm assuming the para-virtualized network driver is probably not being > used either based on the template. > If you take a close look at the xml for the interface, you should be able > to confirm that or not. > > The templates tend to package all the para-virtualized drivers together, > so they're normally in use, or they're not. > > > ________________________________ > From: S.Fuller <[email protected]> > Sent: Friday, July 1, 2022 1:52 PM > To: [email protected] <[email protected]> > Subject: Re: Asymmetric traffic issues? > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when > clicking links, opening attachments, or complying with requests. Click the > "Phish Alert Report" button above the email, or contact MIS, regarding any > suspicious message. > > > > Will give that a try. I'm curious why you think changing the disk drivers > over to virtio will have an effect on the network throughput. > > - Steve > > On Fri, Jul 1, 2022 at 10:55 AM Simon Weller <[email protected]> > wrote: > > > Unfortunately, I can't open the xml via the mailing list, but based on > the > > template you're using, it's not setup for Virtio-blk or virtio-scsi. > Change > > the template to Virtio-SCSI 64bit and the correct interface drive type > will > > be specified in the XML. > > > > ________________________________ > > From: S.Fuller <[email protected]> > > Sent: Thursday, June 30, 2022 12:43 PM > > To: [email protected] <[email protected]> > > Subject: Re: Asymmetric traffic issues? > > > > > > I'm using the Windows Server 2016 profile for the template. After the VM > > came up, I installed the virtio drivers from the latest ISO, and then > added > > a new NIC so that it used the virtio network drivers vs the default E1000 > > drivers. > > > > XML dump is attached > > > > On Thu, Jun 30, 2022 at 12:03 PM Simon Weller <[email protected]> > > wrote: > > Can you provide a redacted dump of your libvirt xml file for the VM? > > Also, which OS profile are you using in Cloudstack for this? The Windows > > specific ones don't use virtio if I remember correctly, so you will need > to > > select Virtio-SCSI 64bit in order for the xml to be built correctly. > > > > -Si > > ________________________________ > > From: S.Fuller <[email protected]<mailto:[email protected]>> > > Sent: Thursday, June 30, 2022 11:38 AM > > To: [email protected]<mailto:[email protected]> < > > [email protected]<mailto:[email protected]>> > > Subject: Re: Asymmetric traffic issues? > > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when > > clicking links, opening attachments, or complying with requests. Click > the > > "Phish Alert Report" button above the email, or contact MIS, regarding > any > > suspicious message. > > > > > > > > Neither the block nor the scsi drivers appear to be running as far as I > can > > tell. > > > > - Steve > > > > On Thu, Jun 30, 2022 at 11:07 AM Simon Weller <[email protected]> > > wrote: > > > > > Steve, > > > > > > Are you running the virtio-block or virtio-scsi drivers? > > > > > > -Si > > > > > > > > > ________________________________ > > > From: S.Fuller <[email protected]<mailto:[email protected]>> > > > Sent: Thursday, June 30, 2022 8:52 AM > > > To: [email protected]<mailto:[email protected]> < > > [email protected]<mailto:[email protected]>> > > > Subject: Re: Asymmetric traffic issues? > > > > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution > when > > > clicking links, opening attachments, or complying with requests. Click > > the > > > "Phish Alert Report" button above the email, or contact MIS, regarding > > any > > > suspicious message. > > > > > > > > > > > > Well, I checked everything that I could for any QOS settings and there > is > > > nothing configured there. What is curious is that Idon't see this > > behavior > > > when the transmitting host is running Linux, or if I'm using the E1000 > > > drivers (although with the E1000 driver, the overall throughput is > > lower). > > > It really feels like I'mrunning into some weird issue with the virtio > > > drivers on Windows. My Windows hosts are (to my knowledge) using the > > latest > > > version of the virtio drivers - 100.90.104.21700 dated 2/23/2022. > > > > > > Steve Fuller > > > [email protected]<mailto:[email protected]> > > > > > > On Wed, Jun 29, 2022 at 3:04 PM S.Fuller <[email protected] > <mailto: > > [email protected]>> wrote: > > > > > > > Vivek, > > > > > > > > Thanks for the reply. I am using the KVM hypervisor. I'll will review > > the > > > > QoS on the hypervisor for both of the nodes. > > > > > > > > I'm checking throughput between two different VMs that running on two > > > > different hosts within the same cluster. As of right now, I'm > receiving > > > > similar results using both iperf3 and nuttcp as the testing tools. We > > are > > > > only seeing this issue when the VM is not on the same host as the > > vrouter > > > > for its isolated network. > > > > > > > > On Tue, Jun 28, 2022 at 6:40 AM Vivek Kumar > > > > <[email protected]<mailto:[email protected]>.invalid> > > wrote: > > > > > > > >> Hey Fuller, > > > >> > > > >> What hypervisor are you using ? I know you have checked all > bandwidth > > > >> limit on templates and global settings, but it’s worth to check the > > QoS > > > on > > > >> the hypervisor level, because at the end it’s the hypervisor which > > > manages > > > >> all. And from where are you trying to check the network throughout, > > > >> between client and server ? > > > >> > > > >> > > > >> Vivek Kumar > > > >> Sr. Manager - Cloud & DevOps > > > >> TechOps | Indiqus Technologies > > > >> > > > >> + 91 7503460090 <tel:++91+7503460090> > > > >> [email protected]<mailto:[email protected]> > > <mailto:[email protected]<mailto:[email protected]>> > > > >> www.indiqus.com<http://www.indiqus.com>< > > http://www.indiqus.com> < > > > https://www.indiqus.com/> > > > >> > > > >> > > > >> > > > >> > On 28-Jun-2022, at 1:58 AM, S.Fuller <[email protected] > <mailto: > > [email protected]>> wrote: > > > >> > > > > >> > Environment: > > > >> > > > > >> > Two physical hosts > > > >> > - Cloudstack 4.11.3 > > > >> > - Verified that there are no bandwidth limits in place on any of > the > > > >> > templates or in global settings. > > > >> > > > > >> > Two isolated networks ("Client" and "Server") > > > >> > - Each has a vrouter with a public and private address > > > >> > - One Windows 2016 VM on each network (running the latest virtio > > > >> drivers) > > > >> > - each node running latest version of Iperf3 to test throughput > > > >> > > > > >> > Testing/Observation: > > > >> > > > > >> > If the Client VM and the vrouter for the isolated Client network > are > > > on > > > >> the > > > >> > same physical host, we see symmetrical throughput in the 2 Gbps > > range, > > > >> > whether we run iperf in regular mode or in reverse mode (iperf > -R). > > > >> > > > > >> > If the Client VM and the vrouter for the isolated Client network > are > > > on > > > >> > different physical hosts, we are seeing 25% of the throughput > > running > > > >> iperf > > > >> > in regular mode vs running it in reverse mode. > > > >> > > > > >> > Has anyone encountered this issue before? If we change the Client > VM > > > to > > > >> > Linux (either CentOS 7 or Ubuntu) OR we use the E1000 driver, we > see > > > >> > symmetrical throughput in our tests, no matter where the vrouter > is > > in > > > >> > relation to the Client VM. > > > >> > > > > >> > -- > > > >> > Steve Fuller > > > >> > [email protected]<mailto:[email protected]> > > > >> > > > >> > > > >> -- > > > >> This message is intended only for the use of the individual or > entity > > to > > > >> which it is addressed and may contain confidential and/or privileged > > > >> information. If you are not the intended recipient, please delete > the > > > >> original message and any copy of it from your computer system. You > are > > > >> hereby notified that any dissemination, distribution or copying of > > this > > > >> communication is strictly prohibited unless proper authorization has > > > been > > > >> obtained for such action. If you have received this communication in > > > >> error, > > > >> please notify the sender immediately. Although IndiQus attempts to > > sweep > > > >> e-mail and attachments for viruses, it does not guarantee that both > > are > > > >> virus-free and accepts no liability for any damage sustained as a > > result > > > >> of > > > >> viruses. > > > >> > > > > > > > > > > > > -- > > > > Steve Fuller > > > > [email protected]<mailto:[email protected]> > > > > > > > > > > > > > -- > > > Steve Fuller > > > [email protected]<mailto:[email protected]> > > > > > > > > > -- > > Steve Fuller > > [email protected]<mailto:[email protected]> > > > > > > -- > > Steve Fuller > > [email protected]<mailto:[email protected]> > > > > > -- > Steve Fuller > [email protected] > -- Steve Fuller [email protected]
