Re: [vfio-users] intel_iommu=on breaks audio on haswell cpu

[Le anonymous faker]

 "intel_iommu=on,igfx_off" means that HDMI audio output will work but
PCI passthrough won't.

I already tried this, with lots of outputs:
https://bbs.archlinux.org/viewtopic.php?id=204460

Thanks !

2016-05-21 19:09 GMT+02:00 Colin Godsey :

> intel_iommu=on,igfx_off should work
>
> On Sat, May 21, 2016 at 11:01 AM Le anonymous faker 
> wrote:
>
>> Lė Patrick 🚗👔☀ Sneida:
>> Hey there,
>>
>> just recently I discovered that the kernel command line "intel_iommu=on"
>> breaks audio output (glitching/freezing output via laptop speakers, no
>> output at all via HDMI) for owners of haswell cpu's.
>> Changing it to "intel_iommu=igfx_off" resolved the issue.
>> How ever, in order to use PCI passthrough of my NVIDIA Card to
>> VirtualBox, I need the parameter "intel_immou=on" ?
>>
>> Is there a way to fix the audio glitching with intel_iommu=on ?
>> Is there a way to get nvidia pci passthrough to working without
>> intel_iommu=on (e.g. intel_iommu=igfx_off) ?
>>
>> Lots of outputs and other users reporting:
>> https://bbs.archlinux.org/viewtopic.php?id=204460
>> https://bugzilla.kernel.org/show_bug.cgi?id=60769
>>
>> Kind regards !___
>> vfio-users mailing list
>> vfio-users@redhat.com
>> https://www.redhat.com/mailman/listinfo/vfio-users
>>
>
___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Recommendation for LGA2011 board

[Bronek Kozicki]

On 23/05/2016 19:20, Brian Yglesias wrote:


I have another thread whereby I try and fail to get vfio/VT-d working 
on two separate lga1366 boards.  (Asus Rampage II Gene and 
GA-EX58-UD5; both due to broken BIOS).


I was considering a rampage III, also 1366/X58, which is reported to 
work, or an OEM system, bug given some of the issues with X58, I'm 
starting to feel like I should just pony up for a new platform.


However, since the GB board was also reported to work with Linux 
(albeit different kernel version from ), I'm worried about pulling the 
trigger and getting stuck with a 3rd incompatible board.


As such, I was hoping someone here could recommend something that is 
working for them right now in the 2011 size.


I'd like 3 x16 ports and dual lan, but those aren't deal breakers.  
I'd rather stay away from brand new flagship stuff, for reasons of 
cost.  that said, my main concern is that it can be made to work, 
without crafting glue and rubber bands. ;)




When buying a motherboard I recommend reading the manual, CPU and memory 
compatibility list, rather than relying on someone's opinion. Checking 
the FAQ for mentions of Linux (and in the context of this thread, VT-d) 
is also helpful. Having said that, I have SuperMicro X9DA7 which I can 
recommend, but this is dual-socket and EATX (i.e. rather large), also it 
does not support the current generation of Intel Xeons. You may want to 
look at any of those instead 
https://www.supermicro.nl/products/motherboard/Xeon3000/#2011 . Other 
trusted brands are http://www.asrockrack.com/general/products.asp#Server 
, http://b2b.gigabyte.com/products/list.aspx?cg=11&p=189&v=35&ck=101 . I 
also have good experience with Tyan, but they do not seem to have a 
motherboard which would suit you.


Asus is typical consumer brand, they do not pay attention to enterprise 
features or even compatibility.



B.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Recommendation for LGA2011 board

[Brandon Ganem]

Haven't had issues with: http://www.asrock.com/mb/Intel/X99%20Extreme4/

Using ECC memory with a XEON right now. No issues

On Mon, May 23, 2016 at 1:17 PM, A de Beus  wrote:

> The Rampage IV Extreme is what I use, there are multiple success stories
> with it.
>
> On May 23, 2016, at 11:20 AM, Brian Yglesias <
> br...@atlanticdigitalsolutions.com> wrote:
>
> I have another thread whereby I try and fail to get vfio/VT-d working on
> two separate lga1366 boards.  (Asus Rampage II Gene and GA-EX58-UD5; both
> due to broken BIOS).
>
> I was considering a rampage III, also 1366/X58, which is reported to work,
> or an OEM system, bug given some of the issues with X58, I'm starting to
> feel like I should just pony up for a new platform.
>
> However, since the GB board was also reported to work with Linux (albeit
> different kernel version from ), I'm worried about pulling the trigger and
> getting stuck with a 3rd incompatible board.
>
> As such, I was hoping someone here could recommend something that is
> working for them right now in the 2011 size.
>
> I'd like 3 x16 ports and dual lan, but those aren't deal breakers.  I'd
> rather stay away from brand new flagship stuff, for reasons of cost.  that
> said, my main concern is that it can be made to work, without crafting glue
> and rubber bands.  ;)
>
> Thanks in advance.
>
> ___
> vfio-users mailing list
> vfio-users@redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
>
>
> ___
> vfio-users mailing list
> vfio-users@redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
>
>
___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Recommendation for LGA2011 board

[A de Beus]

The Rampage IV Extreme is what I use, there are multiple success stories with 
it.

> On May 23, 2016, at 11:20 AM, Brian Yglesias 
>  wrote:
> 
> I have another thread whereby I try and fail to get vfio/VT-d working on two 
> separate lga1366 boards.  (Asus Rampage II Gene and GA-EX58-UD5; both due to 
> broken BIOS).
> 
> I was considering a rampage III, also 1366/X58, which is reported to work, or 
> an OEM system, bug given some of the issues with X58, I'm starting to feel 
> like I should just pony up for a new platform.
> 
> However, since the GB board was also reported to work with Linux (albeit 
> different kernel version from ), I'm worried about pulling the trigger and 
> getting stuck with a 3rd incompatible board.
> 
> As such, I was hoping someone here could recommend something that is working 
> for them right now in the 2011 size.
> 
> I'd like 3 x16 ports and dual lan, but those aren't deal breakers.  I'd 
> rather stay away from brand new flagship stuff, for reasons of cost.  that 
> said, my main concern is that it can be made to work, without crafting glue 
> and rubber bands.  ;)
> 
> Thanks in advance.
> 
> ___
> vfio-users mailing list
> vfio-users@redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

[vfio-users] Recommendation for LGA2011 board

[Brian Yglesias]

I have another thread whereby I try and fail to get vfio/VT-d working on two separate lga1366 boards.  (Asus Rampage II Gene and GA-EX58-UD5; both due to broken BIOS).
I was considering a rampage III, also 1366/X58, which is reported to work, or an OEM system, bug given some of the issues with X58, I'm starting to feel like I should just pony up for a new platform.
However, since the GB board was also reported to work with Linux (albeit different kernel version from ), I'm worried about pulling the trigger and getting stuck with a 3rd incompatible board.
As such, I was hoping someone here could recommend something that is working for them right now in the 2011 size.
I'd like 3 x16 ports and dual lan, but those aren't deal breakers.  I'd rather stay away from brand new flagship stuff, for reasons of cost.  that said, my main concern is that it can be made to work, without crafting glue and rubber bands.  ;)
Thanks in advance.
___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Passing intel_iommu=on in grub causes boot to immediately lock on two different X58 LGA1366 MB.

[Alex Williamson]

On Mon, May 23, 2016 at 3:20 AM, Brian Yglesias <
br...@atlanticdigitalsolutions.com> wrote:

> Sorry for the schizophrenic posts.
>
> Updating the BIOS reset it to default, which returned VT-d to the default
> setting of 'disabled', and that is the only reason I was able to boot.
>
> I tried the new card in both motherboards, and the result is the same.
>
> So, just to update my previous response:  VT-d remains broken on both the
> Rampage II Gene and GA-EX58-UD5 (probably only for Linux, not sure).
>
> Idk if it would be better if a moderator just deleted this and the last
> post, just to avoid any confusion.
>
> At this point I am back at square one.  I don't want to rely on the list I
> found over at the Arch Linux forum, as that got me the Gigabyte board.  The
> info on the Xen wiki just says VT-d is supported on X58, but my experience
> with two higher end boards strong suggests that's an overstatement, if not
> just mostly wrong.
>
> If anyone knows of any half decent LGA 1366 board that supports VT-d
> correctly in Linux, then I would really appreciate the pointer.  It would
> save me a a lot of time and money, both of which I've been been spending
> thus far with no result.
>
> I'll settle for /anything/ with two x16 slots that works.
>

I have a Lenovo Thinkstation S20 (4157CTO) that works reasonably well for
X58-based VT-d work.  Complete systems on ebay are cheap and the power
supply and maybe even motherboard appear to be standard form factors (have
never tried swapping them).  A couple caveats on X58 systems, 1) interrupt
remapping is generally broken on these systems due to processor hardware
errata, it will be disabled and you'll need to
use allow_unsafe_interrupts=1 on the vfio_iommu_type1 module (not a big
deal if you minimally trust your guests), 2) ICH10 does not have ACS
support and will never have quirks, due to being too old for Intel to care
to investigate enabling isolation.  Processor root ports, such as the x16
slots, do have ACS.  Caveat specific to the S20: the onboard NIC is
attached to the ICH10 where surprise hotplug is enabled on the root port
(yes, even though it's soldered onto the motherboard), the NIC (tg3) has no
reset mechanism, so we do a PCI bus reset, which can trigger said surprise
hotplug.  This seems like something I'm constantly fixing upstream, but
it's attached to the ICH10 so these are best left for the host anyway.
___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] VFIO and random host crashes

[Colin Godsey]

I’m now 98% certain I was seeing the ‘skylake freeze’, or one of the other
random bugs in the skylake microcode. Updated to the newest BIOS and
everything is rock solid (my old one was only 5 months old).

There were two big issues regarding c-states and ‘complex workloads’ that
would result in a CPU fault. I’ve now also learned that this type of hard
freeze (where the reset button doesnt work and 0 system logs) is almost
always some kind of CPU fault. The chip basically stops doing anything
(including handling interrupts) until power cycled.

On Sat, May 21, 2016 at 9:55 AM Colin Godsey  wrote:

> Hmm, I’m still on 4.4 but I did notice I didn’t have THP disabled- going
> to try that.
>
> (also just for a small note- I’ve tried various combinations of edge/msi
> IRQs for the devices, no difference really)
>
> I replaced the PSU on my system yesterday, and it improved the stability a
> bunch! But not totally. I got about 4 hours of burin-in followed by 4 hours
> of gaming- still crashed at the end. I didn’t get any artifacts at all, and
> I swear my GPUs were running a bit faster (I finally saw my cards hit the
> ‘pwr’ limit, instead of just ‘voltage rel’). But, it still crashed in the
> end. Same hang as before- 0 logs, system wont restart, etc.
>
> One of these cards is a horrible 3-yr old monster that by-far produces the
> most heat in the system than anything else. Probably more than the other
> card, PSU and CPU combined. When this card was running solo, I did notice
> what I think were device ‘resets’. Screen would blank out and come back.
>
> I’m wondering if maybe these hard device-side resets could cause this- GPU
> resets can be amplified by heat, usage, age/faults etc.
>
> I remember really weird behavior trying to detach/attach devices that were
> already bound with VFIO- perhaps theres something similar here when the
> device disconnects itself, possibly bricking the DMAR or interrupt
> remapping.
>
> I remember reading that the reset switch functions over some kind of ACPI
> based interrupt- bricking the interrupt handler could result in the system
> being unable to log or respond to… anything. Does anybody know how linux
> might respond to a missing EOI or something like that with VFIO? I
> understand VFIO is supposed to give us great device emulation, but with
> interrupt remapping… an interrupt is an interrupt. I’m assuming any
> critical failure in IRQ handling for VFIO devices would be just as bad as a
> host-level device.
>
> Has any one ever like…. yanked a passthrough’d PCI card out while a guest
> was running? Might try that today… just dont want to damage anything
> further.
>
> On Thu, May 19, 2016 at 10:00 AM Alex Williamson <
> alex.l.william...@gmail.com> wrote:
>
>> I'm not convinced the ones I've seen are power related, my system has
>> been running videos in a loop for days, completed a few folding @home jobs,
>> done some compiling, and I even added another (idle, low power) video card
>> since the last hang.  Storage doesn't jive with my observations either.
>> I'm still leaning towards some isolcpus/nohz_full interaction, but I
>> haven't yet started to add those options back.  If you're running a v4.5
>> kernel, please be sure to rule out a transparent hugepage issue with
>> transparent_hugepage=never on the kernel command line or run the latest
>> stable v4.5.5 release (now fixed).
>>
>
___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Passing intel_iommu=on in grub causes boot to immediately lock on two different X58 LGA1366 MB.

[Brian Yglesias]

Sorry for the schizophrenic posts.
Updating the BIOS reset it to default, which returned VT-d to the default setting of 'disabled', and that is the only reason I was able to boot.
I tried the new card in both motherboards, and the result is the same.
So, just to update my previous response:  VT-d remains broken on both the Rampage II Gene and GA-EX58-UD5 (probably only for Linux, not sure).
Idk if it would be better if a moderator just deleted this and the last post, just to avoid any confusion.
At this point I am back at square one.  I don't want to rely on the list I found over at the Arch Linux forum, as that got me the Gigabyte board.  The info on the Xen wiki just says VT-d is supported on X58, but my experience with two higher end boards strong suggests that's an overstatement, if not just mostly wrong.
If anyone knows of any half decent LGA 1366 board that supports VT-d correctly in Linux, then I would really appreciate the pointer.  It would save me a a lot of time and money, both of which I've been been spending thus far with no result.
I'll settle for /anything/ with two x16 slots that works.
Thanks again, and thanks ADB and Alex for the help so far.
On May 18, 2016 6:26 PM, A de Beus  wrote:Make sure you have the latest BIOS for that board. Some of the ASUS boards needed an update for IOMMU.On May 18, 2016, at 2:28 PM, Alex Williamson  wrote:[re-adding vfio-users]On Wed, May 18, 2016 at 1:48 PM, Brian Yglesias  wrote:I've attached with quiet removed (should have done that already).
Nothing jumps out at me, hopefully someone else will see something.
With iommu=pt in lieu of intel_iommu=on the system boots, but no iommu support.  If I append iommu=pt it continues to hard lock.
I've included two images.
It does seem to be somehow iommu related.
I found this post on centos forums:
https://www.centos.org/forums/viewtopic.php?t=46809
That seems to be the same problem with a non-X58 board.  Unfortunately, I don't have the BIOS setting he appears to have used to fix it.
Thanks for all the help thus far.
I'm going to post on Gigabyte forums.  I've seen on both the archlinux forum and the former that ppl have gotten iommu working with this board, which is why I bought it.  Not sure what my problem could be yet.I don't think that centos issue is related, that's a DMA alias issue with broken I/O hardware.  What you're seeing is a hang in qi_submit_sync(), which is an impossible hang according the hardware specs, but comes up a fair bit more often than never.  AIUI, it means that magic, undocumented bits in hardware aren't set the way they should be and you're going to need to hope for a BIOS update, which is all but impossible on a system as old as X58.  Good luck :-\Alex
___vfio-users mailing listvfio-users@redhat.comhttps://www.redhat.com/mailman/listinfo/vfio-users___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Passing intel_iommu=on in grub causes boot to immediately lock on two different X58 LGA1366 MB.

[Brian Yglesias]

In a spectacular oversight on my part, I did not update the Asus Rampage II Gene's BIOS to the most recent, as I thought I had.
In addition, I had purchased a cheap GPU (EVGA GT 730), and when I put it in the board, it booted up normally, and dmesg shows IOMMU and DMAR working.
There is a lot of negative information about Asus and Linux VT-d out there, which for all I know is true, but in terms of this thread I wanted to correct the record:
Enabling IOMMU in grub.conf does not hard lock the Rampage II with kernel 4.4.
On May 18, 2016 6:26 PM, A de Beus  wrote:Make sure you have the latest BIOS for that board. Some of the ASUS boards needed an update for IOMMU.On May 18, 2016, at 2:28 PM, Alex Williamson  wrote:[re-adding vfio-users]On Wed, May 18, 2016 at 1:48 PM, Brian Yglesias  wrote:I've attached with quiet removed (should have done that already).
Nothing jumps out at me, hopefully someone else will see something.
With iommu=pt in lieu of intel_iommu=on the system boots, but no iommu support.  If I append iommu=pt it continues to hard lock.
I've included two images.
It does seem to be somehow iommu related.
I found this post on centos forums:
https://www.centos.org/forums/viewtopic.php?t=46809
That seems to be the same problem with a non-X58 board.  Unfortunately, I don't have the BIOS setting he appears to have used to fix it.
Thanks for all the help thus far.
I'm going to post on Gigabyte forums.  I've seen on both the archlinux forum and the former that ppl have gotten iommu working with this board, which is why I bought it.  Not sure what my problem could be yet.I don't think that centos issue is related, that's a DMA alias issue with broken I/O hardware.  What you're seeing is a hang in qi_submit_sync(), which is an impossible hang according the hardware specs, but comes up a fair bit more often than never.  AIUI, it means that magic, undocumented bits in hardware aren't set the way they should be and you're going to need to hope for a BIOS update, which is all but impossible on a system as old as X58.  Good luck :-\Alex
___vfio-users mailing listvfio-users@redhat.comhttps://www.redhat.com/mailman/listinfo/vfio-users___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users