Re: [vfio-users] threadripper slowness
You need to tweak NUMA because your previous CPU was not NUMA, and your new one is. Having the VM running on some CPU cores on one physical package, but accessing ram that's attached to other CPU cores on a different package can cause a lot of overhead. You want to avoid that. On Wed, Aug 14, 2019 at 7:31 PM Roger Lawhorn wrote: > Just a note: > So far to resolve this I have had to use the kernel boot line options > "rcu_nocbs=0-23 processor.max_cstate=5". > This fixes a known hardware bug in ryzen cpus that leads to constant > crashing in linux. > I also compiled qemu 4.1.0-rc4. > Though it seems better I still suffer severe lag during heavy loading of > textures and 3d objects. > The test games are fallout 4 and fallout 76. > The virtual machine is stored on a 1tb intel ssd. > On a 4 core i7 4940mx it was perfect. > On my new 12 core threadripper it lags out. > As long as the game isnt loading anything new it runs fine. > I never tweaked 'numa' before and I don't see why i need to now. > > Any ideas? > > > I am on this till it works or I am out $3000. > > > On 8/12/19 6:56 AM, Roger Lawhorn wrote: > > > > On 8/11/19 9:59 PM, gamana...@gmail.com wrote: > > You could probably get rid of all the lag you experience. Possibly qemu 4.0 > provides better topology to the guest. Generally Threadripper CPUs have 2 > NUMA nodes, and you would want to pin the vcpus of the guest to one NUMA > node, and also allocate RAM from the same NUMA node. You would also have to > make sure the passthrough devices sit on corresponding PCIe lanes which that > NUMA node controls. There is lots of good info in reddit.com/r/vfio. Please > also provide your xml file (or script), and what you use as a virtualized > hard drive. Is it a passthrough NVMe, or virtio-scsi? In the latter case you > should also pin iothreads on the same NUMA node as everything else. > You can see the topology and NUMA nodes of your cpu by running lstopo. > > I am eager to see a configuration that uses both NUMA nodes of a > Threadripper CPU in one guest with minimal latency. > > > -Original Message- > From: vfio-users-boun...@redhat.com > > On Behalf Of Roger Lawhorn > Sent: Sunday, August 11, 2019 9:31 PM > To: vfio-users@redhat.com > Subject: Re: [vfio-users] threadripper slowness > > Hello, > > Just wanted to mention that I got rid of a lot of the lag by compiling > > qemu 4.0. > > I dod not know if its a threadripper issue or not. > The lag directly coincides with hard drive access. > No hard drive access, no game lag. > I have redhat virtio drivers installed for the hard drive and the nic. > Are these drivers intel only? > They are left only from my i7-4940mx processor install. > > > On 8/9/19 11:51 AM, Roger Lawhorn wrote: > > Hello. > I am new to the list. > I have been doing gpu passthrough for almost 5 years now with a i7 > 4940mx cpu on a msi gt70 laptop. > > My new PC build is: > MSI x399 carbon pro motherboard : bios xx.1c0 : svm=enabled: > iommu=enabled > Threadripper 2920 12 core > Radeon pro duo R9 Fury X > Nvidia 980 TI OC > > I was unable to passthrough the nvidia card due to it not getting past > the "Running option rom..." message when debugging. > > The Radeon passed through with flying colors. > > The issue: > I used the radeon to run triple A games in linux using steam for linux > and proton. No problems. > > In Windows 7 in qemu I have passed all 24 cpus to it. > They max out to 80% while playing the same triple A games and the > games lag. > I have never had this issue before. > > Admittedly I am using the same script I used on the i7 to get going. > I had to remove threads from the -smp switch and use cores only. > > I also removed the kvm=off which is used for nvidia cards. > > I am looking for links to articles or direct info on what to do to > optimize the cpus. > When not lagging the radeon performs extremely well. > > I am concerned that perhaps kvm is not being used. > > > Thanks > > ___ > vfio-users mailing > listvfio-users@redhat.comhttps://www.redhat.com/mailman/listinfo/vfio-users > > _______ > vfio-users mailing > listvfio-users@redhat.comhttps://www.redhat.com/mailman/listinfo/vfio-users > > > > ___ > vfio-users mailing > listvfio-users@redhat.comhttps://www.redhat.com/mailman/listinfo/vfio-users > > > ___ > vfio-users mailing list > vfio-users@redhat.com > https://www.redhat.com/mailman/listinfo/vfio-users > ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Z170X IOMMU Groups
Well, I'd been considering getting a Z170 setup, if I do, I now know what vendor. Kudos indeed. On Thu, Sep 22, 2016 at 2:55 PM, Alex Williamson < alex.l.william...@gmail.com> wrote: > On Thu, Sep 22, 2016 at 12:47 PM, Nick Sarnie <commendsar...@gmail.com> > wrote: > >> Hi again, >> >> Very much to my surprise, Gigabyte replied and sent me a fixed BIOS. The >> new IOMMU groups (with ACS override patch kernel commandline removed for >> this boot), as well as my lspci information are below. I see four messages >> the following messages in dmesg now: >> >> [0.523892] pci :00:1c.0: Intel SPT PCH root port ACS workaround >> enabled >> [0.524031] pci :00:1c.4: Intel SPT PCH root port ACS workaround >> enabled >> [0.524159] pci :00:1c.5: Intel SPT PCH root port ACS workaround >> enabled >> [0.524292] pci :00:1d.0: Intel SPT PCH root port ACS workaround >> enabled >> >> >> IOMMU Groups: http://pastebin.com/raw/0dcHk8Xk >> lspci: http://pastebin.com/raw/1zAZuPBM >> >> Alex, please let me know if they missed anything else, so I can report it >> to them. >> > > :-O > > /me picks jaw off the ground > > Kudos to Gigabyte! > > So we now have: > > 100: 01 00 01 14 > > 0x14010001 = ID 0x0001 (AER), version 0x1, next 0x140 > > 140: 0d 00 01 22 > > 0x2201000d = ID 0x000d (ACS), version 0x1, next 0x220 > > 220: 19 00 01 00 > > 0x00010019 = ID 0x0019 (Secondary PCIe), version 0x1, next 0x0 > > So they've just added ACS into the chain, perfect. Thanks, > > Alex > > ___ > vfio-users mailing list > vfio-users@redhat.com > https://www.redhat.com/mailman/listinfo/vfio-users > > ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] USB >>port<< passthrough
Just to chime in, yes and no. You can't pass a port, you can pass a single device, but it's best to pass a whole USB controller. I also picked up a USB PCIe card for vfio use, it has two USB 3.0 ports on the PCI bracket and a header for two front mounted ports, and I have labels from my Brother labeller to mark those as being the VM ports. I considered using an A/B switch like Jayme described, but ended up using a piece of software called Synergy (popular and mentioned elsewhere on the list before) to switch targets. I turn off scroll lock, I can mouse from host to guest and back (and even copy/paste), and when I hit scroll lock again, the mouse and keyboard are locked to the instance where they are currently. It works very nicely once you're used to it, it cuts down on cabling, and you can plug in a spare keyboard/mouse to connect directly to the guest if needed. On Tue, Aug 23, 2016 at 2:09 PM, Jayme Howard <g.pr...@gmail.com> wrote: > You can't pass the port itself. You can pass individual devices attached > to a port, but the latency can get weird sometimes. I had pretty > significant problems when I did it that way. I opted to use Francisco's > solution above, with a slight adjustment. I have a dedicated card that > gets allocated to the VM. Connected to that, and to one of the host's USB > ports, I have a USB A/B switch. Connected to the switch is a 4 port hub. > Connected to the hub is my keyboard and mouse. All I have to do to switch > the inputs from the host to the guest and back is to choose A or B on the > switch, and a few seconds later all input is directed to the correct > location. > > On Tue, Aug 23, 2016 at 5:28 AM, Quentin Deldycke < > quentindeldy...@gmail.com> wrote: > >> Yes, it is best way. >> >> My motherboard have 2 controllers. One is all time on Linux, the other >> one move between both. >> >> Keyboard mouse on the moving one, printer and other device all time on >> Linux. >> >> On 23 Aug 2016 12:16 p.m., "Francisco Menendez" <aterfe...@gmail.com> >> wrote: >> >>> If you don't mind spending a few dollars, what I did was basically buy >>> a PCIe USB port card, then give it exclusive control of the device to >>> the guest. I can't use it in the host, but I have other USB ports for >>> the host, so no biggie. >>> >>> On Tue, Aug 23, 2016 at 4:07 PM, Rokas Kupstys <rok...@zoho.com> wrote: >>> > Is it possible to pass-through USB port? I know we can pass-through >>> > specific usb devices or entire usb controllers however it is not ideal >>> > in all cases. For instance in my case single pci device drives all usb >>> > ports on the back panel. Naturally i cant pass-through that pci device >>> > because it would leave me without access to the host. Passing through >>> > USB devices is of little use if one wishes to use kvm switch - >>> > mouse/keyboard switched to another port would still have same >>> > vendor/device ids. Is there any solution to this? >>> > >>> > -- >>> > Rokas Kupstys >>> > >>> > >>> > ___ >>> > vfio-users mailing list >>> > vfio-users@redhat.com >>> > https://www.redhat.com/mailman/listinfo/vfio-users >>> >>> ___ >>> vfio-users mailing list >>> vfio-users@redhat.com >>> https://www.redhat.com/mailman/listinfo/vfio-users >>> >> >> ___ >> vfio-users mailing list >> vfio-users@redhat.com >> https://www.redhat.com/mailman/listinfo/vfio-users >> >> > > ___ > vfio-users mailing list > vfio-users@redhat.com > https://www.redhat.com/mailman/listinfo/vfio-users > > ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Host hard lockups
On 08/16/2016 12:51 PM, vfio wrote: > I noticed that the symptoms of the freeze are very similar to the > freezes I sometimes get when shutting down the guest. After searching > the archives, I decided to try enabling MSI for the Titan X in the Win10 > guest. This did indeed stop the freezes at shutdown time. I've been > using the guest for 10 days since my original post. During this time I > experienced only one freeze, but I was not nearby at the time so it's > hard to say if the guest caused it. It took longer than expected, but a definite crash happened yesterday. Sadly, it seems that MSI was not a fix for the in-use crashes. At this point I'm worried that it's some sort of weird hardware-specific interaction that is unlikely to be fixed. If anybody experiences similar symptoms or can suggest any debugging techniques, I'd greatly appreciate any suggestions. Thanks! ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Host hard lockups
Thanks for the suggestions. I'm definitely not running out of RAM; the host has 32GB and I've lowered the guest assignment to 8GB. I also have a constant RAM use monitor that never shows anything near capacity (even in the cases where the guest has frozen but the adjacent CPU monitor on the host has not yet frozen). I have a minor update on the issue. I noticed that the symptoms of the freeze are very similar to the freezes I sometimes get when shutting down the guest. After searching the archives, I decided to try enabling MSI for the Titan X in the Win10 guest. This did indeed stop the freezes at shutdown time. I've been using the guest for 10 days since my original post. During this time I experienced only one freeze, but I was not nearby at the time so it's hard to say if the guest caused it. I still sometimes get similar freezes when starting up the machine after it has already been shutdown once during the host session. In these cases, I've noticed that lspci shows that my USB3 PCIe card is missing all of its details and displays an error. I have not yet saved this error message since it happens so infrequently in my use that I don't care too much about this particular freeze. Random question: does it matter that my host OS boots using compatibility mode (i.e., not UEFI)? The guest machine is using OVMF. One of the problems with debugging this is that it's hard to tell if the problem was fixed due to the odd random distribution of the freezes. I will provide more updates if the situation changes. Thanks. ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] Boot using second GPU?
I'm using an Asus Z97 board and a Gigabyte Z97 board currently, and both allow for switching the boot GPU between PCIE1, PCIE[2-5] or the iGPU. All slots are one ACS group, though. On Sun, Aug 7, 2016 at 10:14 AM, Rokas Kupstys <rok...@zoho.com> wrote: > Thanks for reply. Since i am making a new build i am looking for proper > motherboard. One i sided with is from asus, but it seems asus > motherboards do not support switching primary GPU. I might go with > gigabyte if there is no way to solve this. I am not sure yet though. > Switching primary gpu in bios is ultimate solution. Directions i > provided in earlier mail are workaround for people who do not have such > capability. It is a tricky choice.. Go with beloved asus and suffer > minor inconvenience or go with gigabyte.. Did you have any problems with > gigabyte motherboard(s)? > > > Rokas Kupstys > > On 2016.08.05 14:44, Hristo Iliev wrote: > > Am 05.08.2016 10:22, schrieb Rokas Kupstys: > > > >> Okay this is unexpected luck. After more tinkering i got it to work! > >> Here is my setup: > >> > >> * AMD FX-8350 CPU + Sabertooth 990FX R2 motherboard > >> * :01:00.0 - gpu in first slot > >> * :06:00.0 - gpu in third slot > >> * UEFI on host and guest. > >> * Archlinux > >> > >> In order to make host use non-boot GPU: > >> > >> 1. Add Kernel boot parameter "video=efifb:off". This makes kernel not > >> use first gpu and boot messages appear on second gpu. > >> > >> 2. Bind first gpu (:01:00.0) to vfio-pci driver. I did this by > >> adding line > >> > >>> options vfio-pci ids=1002:677B,1002:AA98 > >> to /etc/modprobe.d/kvm.conf. They are obtained from "lspci -n" which > >> in my case show: > >> > >>> 01:00.0 0300: 1002:677B > >>> 01:00.1 0403: 1002:AA98 > >> 3. Configure xorg to use second gpu (:06:00.0). I added file > >> /etc/X11/xorg.conf.d/secondary-gpu.conf with contents: > >> > >>> Section "Device" > >>> Identifier "Device0" > >>> Driver "radeon" > >>> VendorName "AMD Corporation" > >>> BoardName "AMD Secondary" > >>> BusID "PCI:6:0:0" > >>> EndSection > >> And thats it! Now when machine boots it shows POST messages and > >> bootloader on first gpu, but as soon as boot option is selected > >> display goes blank and kernel boot messages show on second gpu. After > >> boot you can assign first gpu to VM as usual and it works. > >> HELP REQUEST: could someone with intel hardware (ideally x99 chipset) > >> test this method? I am planning a build and if this works i could > >> settle with 28 lane cpu and save couple hundred dollars. Intel's 40 > >> lane cpus are way overpriced.. And with 28 lane cpus only first slot > >> can run at x16 speed while other slots downgrade to x8 or less. > >> Anyhow i would love to hear if this works on intel hardware. > >> > > > > Hi, > > > > I have a Gigabyte GA-X99-UD4 motherboard and i7-5820K. There are two GPUs > > in it - a GTX 970 for pass-through on 03:00.0 and a GT 730 as primary GPU > > on 06:00.0. The PCIE slot of the GT is selected as primary video output > > in the UEFI settings. All text and graphics output goes to it - the > > output > > of the GTX card remains off the entire time until the VM is booted. The X > > server does see both cards but since the nvidia module is prevented from > > binding to the GTX, X cannot use it and starts on the GT. No fiddling > > with > > the console driver parameters necessary. > > > > Distribution: > >Arch Linux, 4.6.4-1-ARCH > > > > Kernel parameters: > >... pci-stub.ids=10de:13c2,10de:0fbb,8086:8d20 nvidia-drm.modeset=1 > > ... > > > > /etc/modprobe.d/vfio.conf: > >options vfio-pci ids=10de:13c2,10de:0fbb,8086:8d20 > > > > /etc/mkinitcpio.conf: > >... > >MODULES="vfio vfio_iommu_type1 vfio_pci vfio_virqfd vfat aes_x86_64 > > crc32c_intel nvidia nvidia_modeset nvidia_uvm nvidia_drm" > >... > > > > /etc/X11/xorg.conf.d/20-nvidia.conf: > >Section "Device" > > Identifier"Device0" > > Driver"nvidia" > > VendorName"NVIDIA Corporation" > > Option "ConnectToAcpid" "0" > >
Re: [vfio-users] Stability Testing/Benchmarking software for Windows VMs
It's been a long time since I did it, but I remember 3Dmark (commercial and free versions) having a looping mode, and I know Furmark was a real GPU killer last time I looked. If you have an eVGA graphics card, their Precision software includes a multiple-test-type version of Furmark built in. On Mon, Jul 11, 2016 at 9:19 PM, Brian Yglesias < br...@atlanticdigitalsolutions.com> wrote: > I'm not much of a gamer, so I'm looking for software to use for > troubleshooting stability issues. > > E.g. the overclockers all use prime95, or at least they used to. Is there > some analog de facto standard for GPU testing? > > The software I've found so far doesn't have a continuous testing mode. > Anything with diagnostic info would be a plus. > > _______ > vfio-users mailing list > vfio-users@redhat.com > https://www.redhat.com/mailman/listinfo/vfio-users > > ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] GTX 580 Passthrough - Successes and Failures
Ah, interesting. I started by trying virtio 0.1.102 (stable), but this does not appear to support Windows 10 yet. When using virtio 0.1.112 (latest), I experienced the freezing issues. After that, I simply stopped using virtio and this allowed Windows to install. I did not benchmark performance, but I assume it is poor. Windows update froze at 24% even without using virtio, but it may be the case that I was simply too impatient. Back when I had two 580 cards, it worked fine as long as the SLI cable was removed and I only passed through the card that was in the physically "lower" slot on the board. I was never able to successfully pass through the card in the slot physically closest to the CPU. In other words, a card in slot 1 will never work, and a card in slot 2 will only work as long as there is a graphics card in slot 1. Another way to look at it is that the card attached to the monitor showing the host UEFI screen will never work with passthrough. That being said, it is possible that I am wrong about this: it could be the case that updates in the past year have resolved the issue, or that my problems back then were also caused by the PCI lane DIP switches. ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
Re: [vfio-users] GTX 580 Passthrough - Successes and Failures
I have been experiencing the precise symptoms that you describe in this thread and your previous thread. I have resolved the issue of the card not resetting after running the VM once, but I have not investigated the I/O issues yet. Have you since identified the issue with I/O? I thought I should reply since I have very similar hardware to you, but with some differences. I've discovered that some of the GPU issues are caused by a feature of our motherboards. My current setup: Processor: i7-4960X Graphics: GeForce GTX 580 (EVGA GeForce GTX 580 FTW Hydro Copper 2) GeForce 210 (ASUS GeForce 210 SILENT) GeForce GTX TITAN X (EVGA GeForce GTX TITAN X HYDRO COPPER) RAM: 4x Corsair Vengeance Black, total 32 GB, set to 1333 MHz Monitors: Monitor [1] hooked up to 580 DVI 1, and TITAN DVI Monitor [2] hooked up to 580 DVI 2 Monitor [3] hooked up to 210 DVI Motherboard: ASUS Rampage IV Extreme Host OS: Debian (sid) Guest OS: Windows 10 Education 64 Bit Kernel: 4.4.0-1-amd64 Host Graphics Driver: nouveau 1.0.12 Many of my problems with VGA passthrough have been caused by the ASUS Rampage IV Extreme motherboard. The board has the following PCI slots: PCIE_X16_1 PCIE_X8_2A PCIE_X8_2B PCIE_X16/8_3 PCIEX1_1 PCIE_X8_4 One year ago, I had two GTX 580 cards---one in PCIE_X16_1 and another in PXIE_X16/8_3. I put the 210 in PCIE_X8_4 and set it as the primary adapter, with the intention of passing both 580s in SLi to the VM. Unfortunately, this board CANNOT perform VGA passthrough with any card placed in PCIE_X16_1 (we established this as part of the discussion in the Arch forum). Since the motherboard does not have integrated graphics, it always performs some initialization on the card in the first PCIE slot; even pci-stub will not prevent the first card from being "claimed". To address this problem, I bought the TITAN X and discarded one of the 580s (having all three in at once was not physically possible). In my current setup, I have the 580 in PCIE_X16_1, the 210 in PCIE_X8_2A, the TITAN in PCIE_X16/8_3, and a Blackmagic Intensity Pro (capture card) in PCIE_X8_4. PCIE_X8_2B and PCIEX1_1 are empty. I mention all of these details because they are relevant to my problem with the TITAN not being reset after launching the VM. The Rampage IV Extreme includes a very handy feature: DIP switches that allow the PCI lanes to be turned on and off. I was using these switches to quickly "unplug" graphics cards while I was testing various configurations; since my system uses open-loop liquid cooling, it is impractical to frequently move the cards between slots. It turns out that VGA passthrough does not work well with the Rampage IV Extreme's PCI lane switches. When I left all of the lanes turned on except for PCIE_X8_4 (I haven't used the Blackmagic card for some time, so I just leave it off), I experienced similar issues: "Invalid ROM contents" messages, errors about not bringing the card out of the D3 power state, and a VM that only worked a fraction of the time. Additionally, my TITAN card would appear to switch between bus 03:00 and bus 04:00, even within the same session if I "removed" the device and then ran /sys/bus/pci/rescan. These issues stopped as soon as I turned all of the PCI lane DIP switches to the "on" position---both with the Blackmagic card plugged in and with it physically removed. The TITAN stopped switching between buses, and the VGA passthrough started working 100% of the time, even without reboots. "systemctl suspend" never resolved the issue for me. Hopefully this helps other people who may have had strange problems with this motherboard. I am planning to reinstall the guest shortly, but I HAVE experienced the same I/O issues as you have (i.e., the Windows installer freezing on multiple "expanding files" percentages, and freezes when performing Windows update). However, I have always been assigning a physical HDD to the guest using "-drive file=/dev/sdf,format=raw". I would be interested to hear if you identify the cause of this problem. ___ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
