I'm using the Windows Server 2016 profile for the template. After the VM
came up, I installed the virtio drivers from the latest ISO, and then added
a new NIC so that it used the virtio network drivers vs the default E1000
drivers.
XML dump is attached
On Thu, Jun 30, 2022 at 12:03 PM Simon Weller <[email protected]>
wrote:
> Can you provide a redacted dump of your libvirt xml file for the VM?
> Also, which OS profile are you using in Cloudstack for this? The Windows
> specific ones don't use virtio if I remember correctly, so you will need to
> select Virtio-SCSI 64bit in order for the xml to be built correctly.
>
> -Si
> ________________________________
> From: S.Fuller <[email protected]>
> Sent: Thursday, June 30, 2022 11:38 AM
> To: [email protected] <[email protected]>
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> Neither the block nor the scsi drivers appear to be running as far as I can
> tell.
>
> - Steve
>
> On Thu, Jun 30, 2022 at 11:07 AM Simon Weller <[email protected]>
> wrote:
>
> > Steve,
> >
> > Are you running the virtio-block or virtio-scsi drivers?
> >
> > -Si
> >
> >
> > ________________________________
> > From: S.Fuller <[email protected]>
> > Sent: Thursday, June 30, 2022 8:52 AM
> > To: [email protected] <[email protected]>
> > Subject: Re: Asymmetric traffic issues?
> >
> > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> > clicking links, opening attachments, or complying with requests. Click
> the
> > "Phish Alert Report" button above the email, or contact MIS, regarding
> any
> > suspicious message.
> >
> >
> >
> > Well, I checked everything that I could for any QOS settings and there is
> > nothing configured there. What is curious is that Idon't see this
> behavior
> > when the transmitting host is running Linux, or if I'm using the E1000
> > drivers (although with the E1000 driver, the overall throughput is
> lower).
> > It really feels like I'mrunning into some weird issue with the virtio
> > drivers on Windows. My Windows hosts are (to my knowledge) using the
> latest
> > version of the virtio drivers - 100.90.104.21700 dated 2/23/2022.
> >
> > Steve Fuller
> > [email protected]
> >
> > On Wed, Jun 29, 2022 at 3:04 PM S.Fuller <[email protected]> wrote:
> >
> > > Vivek,
> > >
> > > Thanks for the reply. I am using the KVM hypervisor. I'll will review
> the
> > > QoS on the hypervisor for both of the nodes.
> > >
> > > I'm checking throughput between two different VMs that running on two
> > > different hosts within the same cluster. As of right now, I'm receiving
> > > similar results using both iperf3 and nuttcp as the testing tools. We
> are
> > > only seeing this issue when the VM is not on the same host as the
> vrouter
> > > for its isolated network.
> > >
> > > On Tue, Jun 28, 2022 at 6:40 AM Vivek Kumar
> > > <[email protected]> wrote:
> > >
> > >> Hey Fuller,
> > >>
> > >> What hypervisor are you using ? I know you have checked all bandwidth
> > >> limit on templates and global settings, but it’s worth to check the
> QoS
> > on
> > >> the hypervisor level, because at the end it’s the hypervisor which
> > manages
> > >> all. And from where are you trying to check the network throughout,
> > >> between client and server ?
> > >>
> > >>
> > >> Vivek Kumar
> > >> Sr. Manager - Cloud & DevOps
> > >> TechOps | Indiqus Technologies
> > >>
> > >> + 91 7503460090 <tel:++91+7503460090>
> > >> [email protected] <mailto:[email protected]>
> > >> www.indiqus.com<http://www.indiqus.com> <
> > https://www.indiqus.com/>
> > >>
> > >>
> > >>
> > >> > On 28-Jun-2022, at 1:58 AM, S.Fuller <[email protected]> wrote:
> > >> >
> > >> > Environment:
> > >> >
> > >> > Two physical hosts
> > >> > - Cloudstack 4.11.3
> > >> > - Verified that there are no bandwidth limits in place on any of the
> > >> > templates or in global settings.
> > >> >
> > >> > Two isolated networks ("Client" and "Server")
> > >> > - Each has a vrouter with a public and private address
> > >> > - One Windows 2016 VM on each network (running the latest virtio
> > >> drivers)
> > >> > - each node running latest version of Iperf3 to test throughput
> > >> >
> > >> > Testing/Observation:
> > >> >
> > >> > If the Client VM and the vrouter for the isolated Client network are
> > on
> > >> the
> > >> > same physical host, we see symmetrical throughput in the 2 Gbps
> range,
> > >> > whether we run iperf in regular mode or in reverse mode (iperf -R).
> > >> >
> > >> > If the Client VM and the vrouter for the isolated Client network are
> > on
> > >> > different physical hosts, we are seeing 25% of the throughput
> running
> > >> iperf
> > >> > in regular mode vs running it in reverse mode.
> > >> >
> > >> > Has anyone encountered this issue before? If we change the Client VM
> > to
> > >> > Linux (either CentOS 7 or Ubuntu) OR we use the E1000 driver, we see
> > >> > symmetrical throughput in our tests, no matter where the vrouter is
> in
> > >> > relation to the Client VM.
> > >> >
> > >> > --
> > >> > Steve Fuller
> > >> > [email protected]
> > >>
> > >>
> > >> --
> > >> This message is intended only for the use of the individual or entity
> to
> > >> which it is addressed and may contain confidential and/or privileged
> > >> information. If you are not the intended recipient, please delete the
> > >> original message and any copy of it from your computer system. You are
> > >> hereby notified that any dissemination, distribution or copying of
> this
> > >> communication is strictly prohibited unless proper authorization has
> > been
> > >> obtained for such action. If you have received this communication in
> > >> error,
> > >> please notify the sender immediately. Although IndiQus attempts to
> sweep
> > >> e-mail and attachments for viruses, it does not guarantee that both
> are
> > >> virus-free and accepts no liability for any damage sustained as a
> result
> > >> of
> > >> viruses.
> > >>
> > >
> > >
> > > --
> > > Steve Fuller
> > > [email protected]
> > >
> >
> >
> > --
> > Steve Fuller
> > [email protected]
> >
>
>
> --
> Steve Fuller
> [email protected]
>
--
Steve Fuller
[email protected]
<domain type='kvm' id='9'>
<name>i-2-355-VM</name>
<uuid>xxxxx</uuid>
<description>Windows Server 2016 (64-bit)</description>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static'>1</vcpu>
<cputune>
<shares>2</shares>
</cputune>
<resource>
<partition>/machine</partition>
</resource>
<sysinfo type='smbios'>
<system>
<entry name='manufacturer'>Apache Software Foundation</entry>
<entry name='product'>CloudStack KVM Hypervisor</entry>
<entry name='uuid'>xxxxx</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type>
<boot dev='cdrom'/>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<cpu mode='custom' match='exact' check='full'>
<model fallback='forbid'>IvyBridge</model>
<feature policy='require' name='hypervisor'/>
<feature policy='require' name='arat'/>
<feature policy='require' name='xsaveopt'/>
</cpu>
<clock offset='localtime'>
<timer name='rtc' tickpolicy='catchup'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/mnt/122bc56f-356a-3b2f-b6f2-c4c11f519284/ff703849-3dd2-4867-8a54-efd3979d6927'/>
<backingStore type='file' index='1'>
<format type='raw'/>
<source file='/mnt/122bc56f-356a-3b2f-b6f2-c4c11f519284/d6164e07-da08-4e3e-8272-ac1177cd99ba'/>
<backingStore/>
</backingStore>
<target dev='hda' bus='ide'/>
<serial>ff7038493dd248678a54</serial>
<alias name='ide0-0-0'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/mnt/08f84e27-b7f9-316f-8883-06b9cceca392/283-2-5d3e89b1-8f32-3d6d-a946-f5f11d9077c6.iso'/>
<backingStore/>
<target dev='hdc' bus='ide'/>
<readonly/>
<alias name='ide0-1-0'/>
<address type='drive' controller='0' bus='1' target='0' unit='0'/>
</disk>
<controller type='usb' index='0' model='piix3-uhci'>
<alias name='usb'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<controller type='ide' index='0'>
<alias name='ide'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='virtio-serial' index='0'>
<alias name='virtio-serial0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='02:00:69:fe:00:17'/>
<source bridge='breth2-315'/>
<target dev='vnet4'/>
<model type='virtio'/>
<link state='up'/>
<alias name='net2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/2'/>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/2'>
<source path='/dev/pts/2'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/i-2-355-VM.org.qemu.guest_agent.0'/>
<target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<input type='tablet' bus='usb'>
<alias name='input0'/>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'>
<alias name='input1'/>
</input>
<input type='keyboard' bus='ps2'>
<alias name='input2'/>
</input>
<graphics type='vnc' port='5902' autoport='yes' listen='xx.xx.xx.xxx'>
<listen type='address' address='xx.xx.xx.xxx'/>
</graphics>
<video>
<model type='cirrus' vram='16384' heads='1' primary='yes'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<watchdog model='i6300esb' action='none'>
<alias name='watchdog0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</watchdog>
<memballoon model='none'/>
</devices>
<seclabel type='dynamic' model='dac' relabel='yes'>
<label>+0:+0</label>
<imagelabel>+0:+0</imagelabel>
</seclabel>
</domain>
