That's great news! Glad it all worked out. -Si ________________________________ From: S.Fuller <steveful...@gmail.com> Sent: Friday, July 8, 2022 12:49 PM To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: Asymmetric traffic issues?
EXTERNAL EMAIL: This message originated outside of ENA. Use caution when clicking links, opening attachments, or complying with requests. Click the "Phish Alert Report" button above the email, or contact MIS, regarding any suspicious message. Holy cow Simon!!! That did the trick!!! Thank you so much! I feel like I should send you coffee or beer or something :) Steve On Fri, Jul 8, 2022 at 11:48 AM Simon Weller <swel...@ena.com.invalid> wrote: > I recall an issue a while back where the Windows Virtio drivers didn't > support generic support offload (GRO) and it created some performance > problems with Cisco NICs. > This was particular to Windows and didn't occur for Linux VMs. > > Can you try disabling GRO on your NICs and retesting? > ethtool -K <interface> gro off > > -Si > ________________________________ > From: S.Fuller <steveful...@gmail.com> > Sent: Thursday, July 7, 2022 9:00 AM > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > Subject: Re: Asymmetric traffic issues? > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when > clicking links, opening attachments, or complying with requests. Click the > "Phish Alert Report" button above the email, or contact MIS, regarding any > suspicious message. > > > > The servers in question are in a Cisco UCS blade chassis. The physical NIC > card is a Cisco UCS VIC 1240, it's a multiport 10gb card that presents > itself multiple virtual NICs to the server. Those virtual NICs use the > Cisco enic driver. > > Steve > > On Fri, Jul 1, 2022 at 4:05 PM Simon Weller <swel...@ena.com.invalid> > wrote: > > > ok, I managed to access your xml file via the mailing list archives. > > > > <interface type="bridge"> > > <mac address="02:00:69:fe:00:17"/> > > <source bridge="breth2-315"/> > > <target dev="vnet4"/> > > <model type="virtio"/> > > <link state="up"/> > > <alias name="net2"/> > > <address type="pci" domain="0x0000" bus="0x00" slot="0x03" > function="0x0"/> > > </interface> > > > > So, it does look correct in terms of virtio-net. One thing of note is > that > > kvm guests can't transmit or recieve packets in parallel, since > virtio-net > > only has one TX and RX queue. This tends to reduce performance on Windows > > guests. > > This can be addressed by enabling multi-queue (you assign a queue for > > vCPU). There was talk of this back in 2021 on the list, but I'm not sure > > whether it was implemented or not. > > > > What brand and chipset are your NICs on the servers? > > > > > > > > > > > > > > ________________________________ > > From: Simon Weller <swel...@ena.com.INVALID> > > Sent: Friday, July 1, 2022 2:34 PM > > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > > Subject: Re: Asymmetric traffic issues? > > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when > > clicking links, opening attachments, or complying with requests. Click > the > > "Phish Alert Report" button above the email, or contact MIS, regarding > any > > suspicious message. > > > > > > > > I'm assuming the para-virtualized network driver is probably not being > > used either based on the template. > > If you take a close look at the xml for the interface, you should be able > > to confirm that or not. > > > > The templates tend to package all the para-virtualized drivers together, > > so they're normally in use, or they're not. > > > > > > ________________________________ > > From: S.Fuller <steveful...@gmail.com> > > Sent: Friday, July 1, 2022 1:52 PM > > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > > Subject: Re: Asymmetric traffic issues? > > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when > > clicking links, opening attachments, or complying with requests. Click > the > > "Phish Alert Report" button above the email, or contact MIS, regarding > any > > suspicious message. > > > > > > > > Will give that a try. I'm curious why you think changing the disk drivers > > over to virtio will have an effect on the network throughput. > > > > - Steve > > > > On Fri, Jul 1, 2022 at 10:55 AM Simon Weller <swel...@ena.com.invalid> > > wrote: > > > > > Unfortunately, I can't open the xml via the mailing list, but based on > > the > > > template you're using, it's not setup for Virtio-blk or virtio-scsi. > > Change > > > the template to Virtio-SCSI 64bit and the correct interface drive type > > will > > > be specified in the XML. > > > > > > ________________________________ > > > From: S.Fuller <steveful...@gmail.com> > > > Sent: Thursday, June 30, 2022 12:43 PM > > > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > > > Subject: Re: Asymmetric traffic issues? > > > > > > > > > I'm using the Windows Server 2016 profile for the template. After the > VM > > > came up, I installed the virtio drivers from the latest ISO, and then > > added > > > a new NIC so that it used the virtio network drivers vs the default > E1000 > > > drivers. > > > > > > XML dump is attached > > > > > > On Thu, Jun 30, 2022 at 12:03 PM Simon Weller <swel...@ena.com.invalid > > > > > wrote: > > > Can you provide a redacted dump of your libvirt xml file for the VM? > > > Also, which OS profile are you using in Cloudstack for this? The > Windows > > > specific ones don't use virtio if I remember correctly, so you will > need > > to > > > select Virtio-SCSI 64bit in order for the xml to be built correctly. > > > > > > -Si > > > ________________________________ > > > From: S.Fuller <steveful...@gmail.com<mailto:steveful...@gmail.com>> > > > Sent: Thursday, June 30, 2022 11:38 AM > > > To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> < > > > users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>> > > > Subject: Re: Asymmetric traffic issues? > > > > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution > when > > > clicking links, opening attachments, or complying with requests. Click > > the > > > "Phish Alert Report" button above the email, or contact MIS, regarding > > any > > > suspicious message. > > > > > > > > > > > > Neither the block nor the scsi drivers appear to be running as far as I > > can > > > tell. > > > > > > - Steve > > > > > > On Thu, Jun 30, 2022 at 11:07 AM Simon Weller <swel...@ena.com.invalid > > > > > wrote: > > > > > > > Steve, > > > > > > > > Are you running the virtio-block or virtio-scsi drivers? > > > > > > > > -Si > > > > > > > > > > > > ________________________________ > > > > From: S.Fuller <steveful...@gmail.com<mailto:steveful...@gmail.com>> > > > > Sent: Thursday, June 30, 2022 8:52 AM > > > > To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> > < > > > users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>> > > > > Subject: Re: Asymmetric traffic issues? > > > > > > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution > > when > > > > clicking links, opening attachments, or complying with requests. > Click > > > the > > > > "Phish Alert Report" button above the email, or contact MIS, > regarding > > > any > > > > suspicious message. > > > > > > > > > > > > > > > > Well, I checked everything that I could for any QOS settings and > there > > is > > > > nothing configured there. What is curious is that Idon't see this > > > behavior > > > > when the transmitting host is running Linux, or if I'm using the > E1000 > > > > drivers (although with the E1000 driver, the overall throughput is > > > lower). > > > > It really feels like I'mrunning into some weird issue with the virtio > > > > drivers on Windows. My Windows hosts are (to my knowledge) using the > > > latest > > > > version of the virtio drivers - 100.90.104.21700 dated 2/23/2022. > > > > > > > > Steve Fuller > > > > steveful...@gmail.com<mailto:steveful...@gmail.com> > > > > > > > > On Wed, Jun 29, 2022 at 3:04 PM S.Fuller <steveful...@gmail.com > > <mailto: > > > steveful...@gmail.com>> wrote: > > > > > > > > > Vivek, > > > > > > > > > > Thanks for the reply. I am using the KVM hypervisor. I'll will > review > > > the > > > > > QoS on the hypervisor for both of the nodes. > > > > > > > > > > I'm checking throughput between two different VMs that running on > two > > > > > different hosts within the same cluster. As of right now, I'm > > receiving > > > > > similar results using both iperf3 and nuttcp as the testing tools. > We > > > are > > > > > only seeing this issue when the VM is not on the same host as the > > > vrouter > > > > > for its isolated network. > > > > > > > > > > On Tue, Jun 28, 2022 at 6:40 AM Vivek Kumar > > > > > <vivek.ku...@indiqus.com<mailto:vivek.ku...@indiqus.com>.invalid> > > > wrote: > > > > > > > > > >> Hey Fuller, > > > > >> > > > > >> What hypervisor are you using ? I know you have checked all > > bandwidth > > > > >> limit on templates and global settings, but it’s worth to check > the > > > QoS > > > > on > > > > >> the hypervisor level, because at the end it’s the hypervisor which > > > > manages > > > > >> all. And from where are you trying to check the network > throughout, > > > > >> between client and server ? > > > > >> > > > > >> > > > > >> Vivek Kumar > > > > >> Sr. Manager - Cloud & DevOps > > > > >> TechOps | Indiqus Technologies > > > > >> > > > > >> + 91 7503460090 <tel:++91+7503460090> > > > > >> vivek.ku...@indiqus.com<mailto:vivek.ku...@indiqus.com> > > > <mailto:vivek.ku...@indiqus.com<mailto:vivek.ku...@indiqus.com>> > > > > >> www.indiqus.com<http://www.indiqus.com>< > > > http://www.indiqus.com> < > > > > https://www.indiqus.com/> > > > > >> > > > > >> > > > > >> > > > > >> > On 28-Jun-2022, at 1:58 AM, S.Fuller <steveful...@gmail.com > > <mailto: > > > steveful...@gmail.com>> wrote: > > > > >> > > > > > >> > Environment: > > > > >> > > > > > >> > Two physical hosts > > > > >> > - Cloudstack 4.11.3 > > > > >> > - Verified that there are no bandwidth limits in place on any of > > the > > > > >> > templates or in global settings. > > > > >> > > > > > >> > Two isolated networks ("Client" and "Server") > > > > >> > - Each has a vrouter with a public and private address > > > > >> > - One Windows 2016 VM on each network (running the latest virtio > > > > >> drivers) > > > > >> > - each node running latest version of Iperf3 to test throughput > > > > >> > > > > > >> > Testing/Observation: > > > > >> > > > > > >> > If the Client VM and the vrouter for the isolated Client network > > are > > > > on > > > > >> the > > > > >> > same physical host, we see symmetrical throughput in the 2 Gbps > > > range, > > > > >> > whether we run iperf in regular mode or in reverse mode (iperf > > -R). > > > > >> > > > > > >> > If the Client VM and the vrouter for the isolated Client network > > are > > > > on > > > > >> > different physical hosts, we are seeing 25% of the throughput > > > running > > > > >> iperf > > > > >> > in regular mode vs running it in reverse mode. > > > > >> > > > > > >> > Has anyone encountered this issue before? If we change the > Client > > VM > > > > to > > > > >> > Linux (either CentOS 7 or Ubuntu) OR we use the E1000 driver, we > > see > > > > >> > symmetrical throughput in our tests, no matter where the vrouter > > is > > > in > > > > >> > relation to the Client VM. > > > > >> > > > > > >> > -- > > > > >> > Steve Fuller > > > > >> > steveful...@gmail.com<mailto:steveful...@gmail.com> > > > > >> > > > > >> > > > > >> -- > > > > >> This message is intended only for the use of the individual or > > entity > > > to > > > > >> which it is addressed and may contain confidential and/or > privileged > > > > >> information. If you are not the intended recipient, please delete > > the > > > > >> original message and any copy of it from your computer system. You > > are > > > > >> hereby notified that any dissemination, distribution or copying of > > > this > > > > >> communication is strictly prohibited unless proper authorization > has > > > > been > > > > >> obtained for such action. If you have received this communication > in > > > > >> error, > > > > >> please notify the sender immediately. Although IndiQus attempts to > > > sweep > > > > >> e-mail and attachments for viruses, it does not guarantee that > both > > > are > > > > >> virus-free and accepts no liability for any damage sustained as a > > > result > > > > >> of > > > > >> viruses. > > > > >> > > > > > > > > > > > > > > > -- > > > > > Steve Fuller > > > > > steveful...@gmail.com<mailto:steveful...@gmail.com> > > > > > > > > > > > > > > > > > -- > > > > Steve Fuller > > > > steveful...@gmail.com<mailto:steveful...@gmail.com> > > > > > > > > > > > > > -- > > > Steve Fuller > > > steveful...@gmail.com<mailto:steveful...@gmail.com> > > > > > > > > > -- > > > Steve Fuller > > > steveful...@gmail.com<mailto:steveful...@gmail.com> > > > > > > > > > -- > > Steve Fuller > > steveful...@gmail.com > > > > > -- > Steve Fuller > steveful...@gmail.com > -- Steve Fuller steveful...@gmail.com
