Good to hear. Thanks for providing a follow up Steve. -Si ________________________________ From: S.Fuller <steveful...@gmail.com> Sent: Friday, July 15, 2022 1:08 PM To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: Asymmetric traffic issues?
EXTERNAL EMAIL: This message originated outside of ENA. Use caution when clicking links, opening attachments, or complying with requests. Click the "Phish Alert Report" button above the email, or contact MIS, regarding any suspicious message. Update on this, we were able to enable the same performance changes by using ethtool to disable gro on the private and public interfaces on the virtual routers. Steve On Fri, Jul 8, 2022 at 2:14 PM Simon Weller <swel...@ena.com.invalid> wrote: > That's great news! Glad it all worked out. > > -Si > ________________________________ > From: S.Fuller <steveful...@gmail.com> > Sent: Friday, July 8, 2022 12:49 PM > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > Subject: Re: Asymmetric traffic issues? > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when > clicking links, opening attachments, or complying with requests. Click the > "Phish Alert Report" button above the email, or contact MIS, regarding any > suspicious message. > > > > Holy cow Simon!!! That did the trick!!! > > Thank you so much! I feel like I should send you coffee or beer or > something :) > > Steve > > On Fri, Jul 8, 2022 at 11:48 AM Simon Weller <swel...@ena.com.invalid> > wrote: > > > I recall an issue a while back where the Windows Virtio drivers didn't > > support generic support offload (GRO) and it created some performance > > problems with Cisco NICs. > > This was particular to Windows and didn't occur for Linux VMs. > > > > Can you try disabling GRO on your NICs and retesting? > > ethtool -K <interface> gro off > > > > -Si > > ________________________________ > > From: S.Fuller <steveful...@gmail.com> > > Sent: Thursday, July 7, 2022 9:00 AM > > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > > Subject: Re: Asymmetric traffic issues? > > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when > > clicking links, opening attachments, or complying with requests. Click > the > > "Phish Alert Report" button above the email, or contact MIS, regarding > any > > suspicious message. > > > > > > > > The servers in question are in a Cisco UCS blade chassis. The physical > NIC > > card is a Cisco UCS VIC 1240, it's a multiport 10gb card that presents > > itself multiple virtual NICs to the server. Those virtual NICs use the > > Cisco enic driver. > > > > Steve > > > > On Fri, Jul 1, 2022 at 4:05 PM Simon Weller <swel...@ena.com.invalid> > > wrote: > > > > > ok, I managed to access your xml file via the mailing list archives. > > > > > > <interface type="bridge"> > > > <mac address="02:00:69:fe:00:17"/> > > > <source bridge="breth2-315"/> > > > <target dev="vnet4"/> > > > <model type="virtio"/> > > > <link state="up"/> > > > <alias name="net2"/> > > > <address type="pci" domain="0x0000" bus="0x00" slot="0x03" > > function="0x0"/> > > > </interface> > > > > > > So, it does look correct in terms of virtio-net. One thing of note is > > that > > > kvm guests can't transmit or recieve packets in parallel, since > > virtio-net > > > only has one TX and RX queue. This tends to reduce performance on > Windows > > > guests. > > > This can be addressed by enabling multi-queue (you assign a queue for > > > vCPU). There was talk of this back in 2021 on the list, but I'm not > sure > > > whether it was implemented or not. > > > > > > What brand and chipset are your NICs on the servers? > > > > > > > > > > > > > > > > > > > > > ________________________________ > > > From: Simon Weller <swel...@ena.com.INVALID> > > > Sent: Friday, July 1, 2022 2:34 PM > > > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > > > Subject: Re: Asymmetric traffic issues? > > > > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution > when > > > clicking links, opening attachments, or complying with requests. Click > > the > > > "Phish Alert Report" button above the email, or contact MIS, regarding > > any > > > suspicious message. > > > > > > > > > > > > I'm assuming the para-virtualized network driver is probably not being > > > used either based on the template. > > > If you take a close look at the xml for the interface, you should be > able > > > to confirm that or not. > > > > > > The templates tend to package all the para-virtualized drivers > together, > > > so they're normally in use, or they're not. > > > > > > > > > ________________________________ > > > From: S.Fuller <steveful...@gmail.com> > > > Sent: Friday, July 1, 2022 1:52 PM > > > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > > > Subject: Re: Asymmetric traffic issues? > > > > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution > when > > > clicking links, opening attachments, or complying with requests. Click > > the > > > "Phish Alert Report" button above the email, or contact MIS, regarding > > any > > > suspicious message. > > > > > > > > > > > > Will give that a try. I'm curious why you think changing the disk > drivers > > > over to virtio will have an effect on the network throughput. > > > > > > - Steve > > > > > > On Fri, Jul 1, 2022 at 10:55 AM Simon Weller <swel...@ena.com.invalid> > > > wrote: > > > > > > > Unfortunately, I can't open the xml via the mailing list, but based > on > > > the > > > > template you're using, it's not setup for Virtio-blk or virtio-scsi. > > > Change > > > > the template to Virtio-SCSI 64bit and the correct interface drive > type > > > will > > > > be specified in the XML. > > > > > > > > ________________________________ > > > > From: S.Fuller <steveful...@gmail.com> > > > > Sent: Thursday, June 30, 2022 12:43 PM > > > > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > > > > Subject: Re: Asymmetric traffic issues? > > > > > > > > > > > > I'm using the Windows Server 2016 profile for the template. After the > > VM > > > > came up, I installed the virtio drivers from the latest ISO, and then > > > added > > > > a new NIC so that it used the virtio network drivers vs the default > > E1000 > > > > drivers. > > > > > > > > XML dump is attached > > > > > > > > On Thu, Jun 30, 2022 at 12:03 PM Simon Weller > <swel...@ena.com.invalid > > > > > > > wrote: > > > > Can you provide a redacted dump of your libvirt xml file for the VM? > > > > Also, which OS profile are you using in Cloudstack for this? The > > Windows > > > > specific ones don't use virtio if I remember correctly, so you will > > need > > > to > > > > select Virtio-SCSI 64bit in order for the xml to be built correctly. > > > > > > > > -Si > > > > ________________________________ > > > > From: S.Fuller <steveful...@gmail.com<mailto:steveful...@gmail.com>> > > > > Sent: Thursday, June 30, 2022 11:38 AM > > > > To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> > < > > > > users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>> > > > > Subject: Re: Asymmetric traffic issues? > > > > > > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution > > when > > > > clicking links, opening attachments, or complying with requests. > Click > > > the > > > > "Phish Alert Report" button above the email, or contact MIS, > regarding > > > any > > > > suspicious message. > > > > > > > > > > > > > > > > Neither the block nor the scsi drivers appear to be running as far > as I > > > can > > > > tell. > > > > > > > > - Steve > > > > > > > > On Thu, Jun 30, 2022 at 11:07 AM Simon Weller > <swel...@ena.com.invalid > > > > > > > wrote: > > > > > > > > > Steve, > > > > > > > > > > Are you running the virtio-block or virtio-scsi drivers? > > > > > > > > > > -Si > > > > > > > > > > > > > > > ________________________________ > > > > > From: S.Fuller <steveful...@gmail.com<mailto:steveful...@gmail.com > >> > > > > > Sent: Thursday, June 30, 2022 8:52 AM > > > > > To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org > > > > < > > > > users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>> > > > > > Subject: Re: Asymmetric traffic issues? > > > > > > > > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution > > > when > > > > > clicking links, opening attachments, or complying with requests. > > Click > > > > the > > > > > "Phish Alert Report" button above the email, or contact MIS, > > regarding > > > > any > > > > > suspicious message. > > > > > > > > > > > > > > > > > > > > Well, I checked everything that I could for any QOS settings and > > there > > > is > > > > > nothing configured there. What is curious is that Idon't see this > > > > behavior > > > > > when the transmitting host is running Linux, or if I'm using the > > E1000 > > > > > drivers (although with the E1000 driver, the overall throughput is > > > > lower). > > > > > It really feels like I'mrunning into some weird issue with the > virtio > > > > > drivers on Windows. My Windows hosts are (to my knowledge) using > the > > > > latest > > > > > version of the virtio drivers - 100.90.104.21700 dated 2/23/2022. > > > > > > > > > > Steve Fuller > > > > > steveful...@gmail.com<mailto:steveful...@gmail.com> > > > > > > > > > > On Wed, Jun 29, 2022 at 3:04 PM S.Fuller <steveful...@gmail.com > > > <mailto: > > > > steveful...@gmail.com>> wrote: > > > > > > > > > > > Vivek, > > > > > > > > > > > > Thanks for the reply. I am using the KVM hypervisor. I'll will > > review > > > > the > > > > > > QoS on the hypervisor for both of the nodes. > > > > > > > > > > > > I'm checking throughput between two different VMs that running on > > two > > > > > > different hosts within the same cluster. As of right now, I'm > > > receiving > > > > > > similar results using both iperf3 and nuttcp as the testing > tools. > > We > > > > are > > > > > > only seeing this issue when the VM is not on the same host as the > > > > vrouter > > > > > > for its isolated network. > > > > > > > > > > > > On Tue, Jun 28, 2022 at 6:40 AM Vivek Kumar > > > > > > <vivek.ku...@indiqus.com<mailto:vivek.ku...@indiqus.com > >.invalid> > > > > wrote: > > > > > > > > > > > >> Hey Fuller, > > > > > >> > > > > > >> What hypervisor are you using ? I know you have checked all > > > bandwidth > > > > > >> limit on templates and global settings, but it’s worth to check > > the > > > > QoS > > > > > on > > > > > >> the hypervisor level, because at the end it’s the hypervisor > which > > > > > manages > > > > > >> all. And from where are you trying to check the network > > throughout, > > > > > >> between client and server ? > > > > > >> > > > > > >> > > > > > >> Vivek Kumar > > > > > >> Sr. Manager - Cloud & DevOps > > > > > >> TechOps | Indiqus Technologies > > > > > >> > > > > > >> + 91 7503460090 <tel:++91+7503460090> > > > > > >> vivek.ku...@indiqus.com<mailto:vivek.ku...@indiqus.com> > > > > <mailto:vivek.ku...@indiqus.com<mailto:vivek.ku...@indiqus.com>> > > > > > >> www.indiqus.com<http://www.indiqus.com>< > > > > http://www.indiqus.com> < > > > > > https://www.indiqus.com/> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > On 28-Jun-2022, at 1:58 AM, S.Fuller <steveful...@gmail.com > > > <mailto: > > > > steveful...@gmail.com>> wrote: > > > > > >> > > > > > > >> > Environment: > > > > > >> > > > > > > >> > Two physical hosts > > > > > >> > - Cloudstack 4.11.3 > > > > > >> > - Verified that there are no bandwidth limits in place on any > of > > > the > > > > > >> > templates or in global settings. > > > > > >> > > > > > > >> > Two isolated networks ("Client" and "Server") > > > > > >> > - Each has a vrouter with a public and private address > > > > > >> > - One Windows 2016 VM on each network (running the latest > virtio > > > > > >> drivers) > > > > > >> > - each node running latest version of Iperf3 to test > throughput > > > > > >> > > > > > > >> > Testing/Observation: > > > > > >> > > > > > > >> > If the Client VM and the vrouter for the isolated Client > network > > > are > > > > > on > > > > > >> the > > > > > >> > same physical host, we see symmetrical throughput in the 2 > Gbps > > > > range, > > > > > >> > whether we run iperf in regular mode or in reverse mode (iperf > > > -R). > > > > > >> > > > > > > >> > If the Client VM and the vrouter for the isolated Client > network > > > are > > > > > on > > > > > >> > different physical hosts, we are seeing 25% of the throughput > > > > running > > > > > >> iperf > > > > > >> > in regular mode vs running it in reverse mode. > > > > > >> > > > > > > >> > Has anyone encountered this issue before? If we change the > > Client > > > VM > > > > > to > > > > > >> > Linux (either CentOS 7 or Ubuntu) OR we use the E1000 driver, > we > > > see > > > > > >> > symmetrical throughput in our tests, no matter where the > vrouter > > > is > > > > in > > > > > >> > relation to the Client VM. > > > > > >> > > > > > > >> > -- > > > > > >> > Steve Fuller > > > > > >> > steveful...@gmail.com<mailto:steveful...@gmail.com> > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> This message is intended only for the use of the individual or > > > entity > > > > to > > > > > >> which it is addressed and may contain confidential and/or > > privileged > > > > > >> information. If you are not the intended recipient, please > delete > > > the > > > > > >> original message and any copy of it from your computer system. > You > > > are > > > > > >> hereby notified that any dissemination, distribution or copying > of > > > > this > > > > > >> communication is strictly prohibited unless proper authorization > > has > > > > > been > > > > > >> obtained for such action. If you have received this > communication > > in > > > > > >> error, > > > > > >> please notify the sender immediately. Although IndiQus attempts > to > > > > sweep > > > > > >> e-mail and attachments for viruses, it does not guarantee that > > both > > > > are > > > > > >> virus-free and accepts no liability for any damage sustained as > a > > > > result > > > > > >> of > > > > > >> viruses. > > > > > >> > > > > > > > > > > > > > > > > > > -- > > > > > > Steve Fuller > > > > > > steveful...@gmail.com<mailto:steveful...@gmail.com> > > > > > > > > > > > > > > > > > > > > > -- > > > > > Steve Fuller > > > > > steveful...@gmail.com<mailto:steveful...@gmail.com> > > > > > > > > > > > > > > > > > -- > > > > Steve Fuller > > > > steveful...@gmail.com<mailto:steveful...@gmail.com> > > > > > > > > > > > > -- > > > > Steve Fuller > > > > steveful...@gmail.com<mailto:steveful...@gmail.com> > > > > > > > > > > > > > -- > > > Steve Fuller > > > steveful...@gmail.com > > > > > > > > > -- > > Steve Fuller > > steveful...@gmail.com > > > > > -- > Steve Fuller > steveful...@gmail.com > -- Steve Fuller steveful...@gmail.com
