Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Alan Grimes]

Here's the grub command line I was using on my Phenom II on a 990FX
board, I had only commented it out after upgrading to Ryzen

#GRUB_CMDLINE_LINUX="check_enable_amd_mmconf amd_iommu=fullflush iommu=soft"


-- 
Strange Game.
The only winning move is not to play. 

Powers are not rights.

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Corbin Bird]

On 05/21/2017 06:12 PM, Adam Carter wrote:
> 
> 
> > [0.991863] iommu: Adding device :06:00.0 to group 12
> > [0.991982] iommu: Adding device :07:04.0 to group 12
> > [1.063849] AMD-Vi: Found IOMMU at :00:00.2 cap 0x40
> > [1.063962] AMD-Vi: Interrupt remapping enabled
> > [1.064145] AMD-Vi: Lazy IO/TLB flushing enabled
> > [1.065331] perf: AMD NB counters detected
> 
> 
> I'm similar, but have a couple of extra entries. I've read a little bit
> about them, but so far am unable to determine if their existence
> indicates a better or worse kernel config.
> 
> [1.036309] AMD-Vi: Lazy IO/TLB flushing enabled
> [1.036419] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> [1.036529] software IO TLB [mem 0xba61a000-0xbe61a000] (64MB) mapped
> at [a3b87a61a000-a3b87e619fff]
> [1.036744] perf: AMD NB counters detected
> 
> And the Linux AGP Driver ( in-kernel ) is working now.
> 
> Now this is showing properly with lspci :
> 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD890S/RD990 I/O
> Memory Management Unit (IOMMU)
> 
> 
> Same.

The (SWIOTLB) should not be the default.

What kernel parameters for IOMMUs are you using now?

The listed result I posted is with nothing related to IOMMUs in the
kernel parameters, and NO GART IOMMU support compiled.


Corbin

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Adam Carter]

On Mon, May 22, 2017 at 7:15 PM, taii...@gmx.com  wrote:

> Worse, ideally you wouldn't be using SWIOTLB but I don't know how to
> disable this without re-compiling the kernel.
>

Can you point to some definitive documentation on this?

When i read https://wiki.gentoo.org/wiki/IOMMU_SWIOTLB i dont come to that
conclusion.

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[taii...@gmx.com]

Worse, ideally you wouldn't be using SWIOTLB but I don't know how to 
disable this without re-compiling the kernel.



On 05/21/2017 07:12 PM, Adam Carter wrote:




[0.991863] iommu: Adding device :06:00.0 to group 12
[0.991982] iommu: Adding device :07:04.0 to group 12
[1.063849] AMD-Vi: Found IOMMU at :00:00.2 cap 0x40
[1.063962] AMD-Vi: Interrupt remapping enabled
[1.064145] AMD-Vi: Lazy IO/TLB flushing enabled
[1.065331] perf: AMD NB counters detected

q
I'm similar, but have a couple of extra entries. I've read a little bit
about them, but so far am unable to determine if their existence indicates
a better or worse kernel config.

[1.036309] AMD-Vi: Lazy IO/TLB flushing enabled
[1.036419] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[1.036529] software IO TLB [mem 0xba61a000-0xbe61a000] (64MB) mapped at
[a3b87a61a000-a3b87e619fff]
[1.036744] perf: AMD NB counters detected

And the Linux AGP Driver ( in-kernel ) is working now.

Now this is showing properly with lspci :
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD890S/RD990 I/O
Memory Management Unit (IOMMU)


Same.

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Adam Carter]

> > [0.991863] iommu: Adding device :06:00.0 to group 12
> > [0.991982] iommu: Adding device :07:04.0 to group 12
> > [1.063849] AMD-Vi: Found IOMMU at :00:00.2 cap 0x40
> > [1.063962] AMD-Vi: Interrupt remapping enabled
> > [1.064145] AMD-Vi: Lazy IO/TLB flushing enabled
> > [1.065331] perf: AMD NB counters detected
>

I'm similar, but have a couple of extra entries. I've read a little bit
about them, but so far am unable to determine if their existence indicates
a better or worse kernel config.

[1.036309] AMD-Vi: Lazy IO/TLB flushing enabled
[1.036419] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[1.036529] software IO TLB [mem 0xba61a000-0xbe61a000] (64MB) mapped at
[a3b87a61a000-a3b87e619fff]
[1.036744] perf: AMD NB counters detected

And the Linux AGP Driver ( in-kernel ) is working now.
>
> Now this is showing properly with lspci :
> 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD890S/RD990 I/O
> Memory Management Unit (IOMMU)
>

Same.

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Corbin Bird]

On 05/19/2017 01:39 AM, Adam Carter wrote:
> On Wed, May 17, 2017 at 12:25 AM, Corbin Bird  > wrote:
> 
> On 05/16/2017 01:34 AM, Adam Carter wrote:
> > Answer #1 :
> >
> >
> > Asrock, Extreme6, 990FX, UEFI, AMD FX-9590
> > Gigabyte, GA-MA790FX-UD5P, 790FX, BIOS, AMD PhenomII x4 980
> >
> >
> > Hi Corbin,
> >
> > I noticed i didnt have x2apic enabled on my 990FX / FX-8350 system, so
> > i've rebuilt/rebooted etc, but;
> >
> > # zgrep X2APIC /proc/config.gz
> > CONFIG_X86_X2APIC=y
> > # dmesg | grep -i x2apic
> > #
> >
> > On my skylake box
> > # dmesg | grep -i x2apic
> > [0.044148] DMAR-IR: Queued invalidation will be enabled to support
> > x2apic and Intr-remapping.
> > [0.045802] DMAR-IR: Enabled IRQ remapping in x2apic mode
> > [0.045908] x2apic enabled
> > [0.046011] Switched APIC routing to cluster x2apic.
> >
> > What x2apic messages do you get on your 990FX / FX-9590 system?
> 
> 
> Interesting kernel parameter for you to try :
> ( from '/usr/src/linux/Documentation/kernel-parameters.txt' )
> 
> x2apic_phys [X86-64,APIC] Use x2apic physical mode instead of
> default x2apic cluster mode on platforms
> supporting x2apic.
> 
> With that switch I have not been getting any output in 'dmesg' about
> this at all.
> 
> 
> I tried adding x2apic_phys, but dmesg output did not change.
> 
> From wikip;
> "In 2012, AMD announced their /Advanced Virtual Interrupt Controller/
> (/AVIC/) targeting interrupt overhead reduction in virtualization
> environments.^[33]
>  This
> technology has materialize in hardware and (as announced) does not
> support x2APIC .^[34]
>  In 2016,
> AVIC is available on the AMD family 15h models 6Xh (Carrizo) processors
> and newer.^[35]"
> 
>  
> So;
> FX-8350 processors (bulldozer gen2) dont have AVIC
> Carrizo processors (bulldozer gen4), have AVIC, but AVIC doesnt support
> x2apic
> seems reasonable to conclude that there's no x2apic on bulldozer gen2,
> and that's why there's nothing in dmesg about it.
> 
> 
> Question :
> Do you have this enabled in your kernel ...
> >   [ ]   Support for Intel IOMMU using DMA Remapping Devices
> 
> 
> On the Skylake box, yes, FX box, no.

Thanks for the info. Will kill the kernel parameter on my box.

---

Tried an experiment, recompiled the kernel, no GART IOMMU support.
This was not enabled either.
 < > AMD IOMMU Version 2 driver

CPU FX-9590, 990FX Chipset

Now getting this instead :

> [0.988027] PCI: CLS 64 bytes, default 64
> [0.988362] iommu: Adding device :00:00.0 to group 0
> [0.988530] iommu: Adding device :00:02.0 to group 1
> [0.988689] iommu: Adding device :00:0a.0 to group 2
> [0.988872] iommu: Adding device :00:0d.0 to group 3
> [0.989029] iommu: Adding device :00:11.0 to group 4
> [0.989194] iommu: Adding device :00:12.0 to group 5
> [0.989318] iommu: Adding device :00:12.2 to group 5
> [0.989484] iommu: Adding device :00:13.0 to group 6
> [0.989615] iommu: Adding device :00:13.2 to group 6
> [0.989780] iommu: Adding device :00:14.0 to group 7
> [0.989943] iommu: Adding device :00:14.1 to group 8
> [0.990128] iommu: Adding device :00:14.3 to group 9
> [0.990292] iommu: Adding device :00:14.4 to group 10
> [0.990458] iommu: Adding device :00:14.5 to group 11
> [0.990637] iommu: Adding device :00:15.0 to group 12
> [0.990763] iommu: Adding device :00:15.1 to group 12
> [0.990935] iommu: Adding device :00:16.0 to group 13
> [0.991058] iommu: Adding device :00:16.2 to group 13
> [0.991252] iommu: Adding device :01:00.0 to group 14
> [0.991387] iommu: Adding device :01:00.1 to group 14
> [0.991560] iommu: Adding device :02:00.0 to group 15
> [0.991742] iommu: Adding device :03:00.0 to group 16
> [0.991863] iommu: Adding device :06:00.0 to group 12
> [0.991982] iommu: Adding device :07:04.0 to group 12
> [1.063849] AMD-Vi: Found IOMMU at :00:00.2 cap 0x40
> [1.063962] AMD-Vi: Interrupt remapping enabled
> [1.064145] AMD-Vi: Lazy IO/TLB flushing enabled
> [1.065331] perf: AMD NB counters detected
> [1.065622] LVT offset 0 assigned for vector 0x400
> [1.065836] perf: AMD IBS detected (0x00ff)

And the Linux AGP Driver ( in-kernel ) is working now.

Now this is showing properly with lspci :
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD890S/RD990 I/O
Memory Management Unit (IOMMU)

And the GART IOMMU is 

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Adam Carter]

On Wed, May 17, 2017 at 12:25 AM, Corbin Bird 
wrote:

> On 05/16/2017 01:34 AM, Adam Carter wrote:
> > Answer #1 :
> >
> >
> > Asrock, Extreme6, 990FX, UEFI, AMD FX-9590
> > Gigabyte, GA-MA790FX-UD5P, 790FX, BIOS, AMD PhenomII x4 980
> >
> >
> > Hi Corbin,
> >
> > I noticed i didnt have x2apic enabled on my 990FX / FX-8350 system, so
> > i've rebuilt/rebooted etc, but;
> >
> > # zgrep X2APIC /proc/config.gz
> > CONFIG_X86_X2APIC=y
> > # dmesg | grep -i x2apic
> > #
> >
> > On my skylake box
> > # dmesg | grep -i x2apic
> > [0.044148] DMAR-IR: Queued invalidation will be enabled to support
> > x2apic and Intr-remapping.
> > [0.045802] DMAR-IR: Enabled IRQ remapping in x2apic mode
> > [0.045908] x2apic enabled
> > [0.046011] Switched APIC routing to cluster x2apic.
> >
> > What x2apic messages do you get on your 990FX / FX-9590 system?
>
>
> Interesting kernel parameter for you to try :
> ( from '/usr/src/linux/Documentation/kernel-parameters.txt' )
>
> x2apic_phys [X86-64,APIC] Use x2apic physical mode instead of
> default x2apic cluster mode on platforms
> supporting x2apic.
>
> With that switch I have not been getting any output in 'dmesg' about
> this at all.
>

I tried adding x2apic_phys, but dmesg output did not change.

>From wikip;
"In 2012, AMD announced their *Advanced Virtual Interrupt Controller* (
*AVIC*) targeting interrupt overhead reduction in virtualization
environments.[33]
 This
technology has materialize in hardware and (as announced) does not support
x2APIC .[34]
 In 2016,
AVIC is available on the AMD family 15h models 6Xh (Carrizo) processors and
newer.[35]" 

So;
FX-8350 processors (bulldozer gen2) dont have AVIC
Carrizo processors (bulldozer gen4), have AVIC, but AVIC doesnt support
x2apic
seems reasonable to conclude that there's no x2apic on bulldozer gen2, and
that's why there's nothing in dmesg about it.


> Question :
> Do you have this enabled in your kernel ...
> >   [ ]   Support for Intel IOMMU using DMA Remapping Devices
>

On the Skylake box, yes, FX box, no.

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[taii...@gmx.com]

FYI:
IOMMU=pt means pass-through, as in no DMA protection.

AMD_IOMMU is for the bulldozer and piledriver based systems, v2 is for 
the newer excavator and beyond stuff that has vAPIC




On 05/16/2017 10:33 AM, Corbin Bird wrote:

On 05/15/2017 09:59 PM, taii...@gmx.com wrote:

On 05/15/2017 12:58 AM, Ian Zimmerman wrote:


The 990FX / 790FX Chipset doesn't have a GART / AGP Aperature or IOMMU
in it.  The CPU contains the original K8 IOMMU ( v1 ) engineered /
converted from a GART.


The 8 and 9 series (not 7) does have an IOMMU, AMD-Vi v1.26.

I have two 890 series (but the server flavor - SR5690 chipset) boards
with an IOMMU and it works great, I play games in a VM with an attached
graphics card on my libre firmware KGPE-D16 and devices are DMA restricted.

Most consumer boards don't properly implement this feature, in fact I
have never seen one that did which is why I bought my coreboot (libre
init variant) D16.
It wouldn't be that difficult to port coreboot to your board if you want
this to work FYI.


Thank you for that info.

Corrections based on fact are appreciated.

http://support.amd.com/TechDocs/43869.pdf
Hell yeah dude, this documentation backs up the fact that the IOMMU is 
on the northbridge - not the CPU.


---

How does one 'port' coreboot?

Last time I went to the coreboot site, I didn't see anything really
helpful to me.
The documentation really sucks, its DIY or die - but I have never 
programmed anything before in my life and I figured out how to port boards.

Doesn't the 'CPU voltage table firmware blob' require signing NDA's?
Naah you don't need that on most *good* systems, and 
bulldozer/piledriver era AMD was cool with releasing documentation.



Corbin

https://www.coreboot.org/Developer_Manual
https://www.coreboot.org/Motherboard_Porting_Guide
You would start with the KCMA-D8, as it is the closest board - then 
change the superio, irq mappings, acpi etc. It takes some figuring out 
for yourself as there isn't really a detailed guide for it.

[gentoo-user] Re: Issues with AMD_IOMMU

[Ian Zimmerman]

On 2017-05-16 09:11, Corbin Bird wrote:

> > http://bogdan.org.ua/2009/09/30/iommu-this-costs-you-64-mb-of-ram.html
> 
> That link, read the rest of it.
> 
> It says to leave it alone, let the kernel use it as an IOMMU.
> 
> On AMD, with NO IOMMU kernel parameters, output in '/var/log/dmesg':
> NOTE : IOMMU is enabled in the UEFI firmware.
> 
> > [0.00] AGP: Checking aperture...
> > [0.00] AGP: No AGP bridge found
> > [0.00] AGP: Node 0: aperture [bus addr 0xb400-0xb5ff] (32MB)
> > [0.00] Aperture pointing to e820 RAM. Ignoring.
> > [0.00] AGP: Your BIOS doesn't leave an aperture memory hole
> > [0.00] AGP: Please enable the IOMMU option in the BIOS setup
> > [0.00] AGP: This costs you 64MB of RAM
> > [0.00] AGP: Mapping aperture over RAM [mem 0xb400-0xb7ff] 
> > (65536KB)
> 
> > [0.926772] PCI: CLS 64 bytes, default 64
> > [0.926983] PCI-DMA: Disabling AGP.
> > [0.927215] PCI-DMA: aperture base @ b400 size 65536 KB
> > [0.927336] PCI-DMA: using GART IOMMU.
> > [0.927448] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture

And that is exactly what I get.

It was you who suggested starting to add iommu=blah to the kernel args.

-- 
Please *no* private Cc: on mailing lists and newsgroups
Personal signed mail: please _encrypt_ and sign
Don't clear-text sign:
http://primate.net/~itz/blog/the-problem-with-gpg-signatures.html

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Corbin Bird]

On 05/15/2017 09:59 PM, taii...@gmx.com wrote:
> On 05/15/2017 12:58 AM, Ian Zimmerman wrote:
> 
>>> The 990FX / 790FX Chipset doesn't have a GART / AGP Aperature or IOMMU
>>> in it.  The CPU contains the original K8 IOMMU ( v1 ) engineered /
>>> converted from a GART.
>>>
> The 8 and 9 series (not 7) does have an IOMMU, AMD-Vi v1.26.
> 
> I have two 890 series (but the server flavor - SR5690 chipset) boards
> with an IOMMU and it works great, I play games in a VM with an attached
> graphics card on my libre firmware KGPE-D16 and devices are DMA restricted.
> 
> Most consumer boards don't properly implement this feature, in fact I
> have never seen one that did which is why I bought my coreboot (libre
> init variant) D16.
> It wouldn't be that difficult to port coreboot to your board if you want
> this to work FYI.
> 

Thank you for that info.

Corrections based on fact are appreciated.

---

How does one 'port' coreboot?

Last time I went to the coreboot site, I didn't see anything really
helpful to me.

Doesn't the 'CPU voltage table firmware blob' require signing NDA's?


Corbin

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Corbin Bird]

On 05/16/2017 01:34 AM, Adam Carter wrote:
> Answer #1 :
> 
> 
> Asrock, Extreme6, 990FX, UEFI, AMD FX-9590
> Gigabyte, GA-MA790FX-UD5P, 790FX, BIOS, AMD PhenomII x4 980
> 
> 
> Hi Corbin,
> 
> I noticed i didnt have x2apic enabled on my 990FX / FX-8350 system, so
> i've rebuilt/rebooted etc, but;
> 
> # zgrep X2APIC /proc/config.gz
> CONFIG_X86_X2APIC=y
> # dmesg | grep -i x2apic
> #
> 
> On my skylake box
> # dmesg | grep -i x2apic
> [0.044148] DMAR-IR: Queued invalidation will be enabled to support
> x2apic and Intr-remapping.
> [0.045802] DMAR-IR: Enabled IRQ remapping in x2apic mode
> [0.045908] x2apic enabled
> [0.046011] Switched APIC routing to cluster x2apic.
> 
> What x2apic messages do you get on your 990FX / FX-9590 system?


Interesting kernel parameter for you to try :
( from '/usr/src/linux/Documentation/kernel-parameters.txt' )

x2apic_phys [X86-64,APIC] Use x2apic physical mode instead of
default x2apic cluster mode on platforms
supporting x2apic.

With that switch I have not been getting any output in 'dmesg' about
this at all.

Question :
Do you have this enabled in your kernel ...
>   [ ]   Support for Intel IOMMU using DMA Remapping Devices  


Corbin

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Corbin Bird]

On 05/15/2017 09:33 PM, Ian Zimmerman wrote:
> On 2017-05-15 14:33, Corbin Bird wrote:
> 
>> Gigabyte has long been known to ship 'broken for Linux' e820 firmware.
>> 'e820' is basically the APG Aperature setup in the firmware.
>> Gigabyte kills OR overrides the IOMMU support in the firmware, to setup
>> the AGP Aperature.
>>
>> Closed source / binary video drivers WANT an AGP Aperature.
> 
> So is _that_ what it's all about?  Is the default assumption that I run
> non-free video drivers?  Why?  I _never_ use these, and if you bet on me
> using them in the future and win the bet, you can probably make a cool
> million or two.
> 
>> The kernel wants an IOMMU on x86_64, as an IOMMU.
> 
> But it has one!  The GART_IOMMU.  It was even added specifically for
> Linux, according to kernel.org discussions I remember seeing but can't
> locate now.
> 
> I also found [1] , which I think is the clearest and most human-oriented
> explanation of this issue, yet.  And ... the recommendation is pretty
> much the opposite of yours.  The ball is in your court, sir.
> 
>>> CONFIG_X86_X2APIC=y
> 
> I did this long ago, and it has no effect - still only one is found,
> according to dmesg.  As I wrote in the earlier thread.
> 
> Also, the Kconfig help for this option says it does something quite
> different.  
> 
>>> CONFIG_X86_MPPARSE=y
> 
> Ok, this one I have not set so far.  Will read up on it and try,
> 
> [1]
> http://bogdan.org.ua/2009/09/30/iommu-this-costs-you-64-mb-of-ram.html
> 

That link, read the rest of it.

It says to leave it alone, let the kernel use it as an IOMMU.



I once compiled and tried to run a 64bit kernel with NO IOMMU support,
NO virt/vm support, used kernel parameters to kill all forms of IOMMU
... would not run at all ( hard lock / kernel panic ).

The error output made it very clear that an IOMMU was REQUIRED for
x86_64 ( 64bit ).



On AMD, with NO IOMMU kernel parameters, output in '/var/log/dmesg':
NOTE : IOMMU is enabled in the UEFI firmware.

> [0.00] AGP: Checking aperture...
> [0.00] AGP: No AGP bridge found
> [0.00] AGP: Node 0: aperture [bus addr 0xb400-0xb5ff] (32MB)
> [0.00] Aperture pointing to e820 RAM. Ignoring.
> [0.00] AGP: Your BIOS doesn't leave an aperture memory hole
> [0.00] AGP: Please enable the IOMMU option in the BIOS setup
> [0.00] AGP: This costs you 64MB of RAM
> [0.00] AGP: Mapping aperture over RAM [mem 0xb400-0xb7ff] 
> (65536KB)

> [0.926772] PCI: CLS 64 bytes, default 64
> [0.926983] PCI-DMA: Disabling AGP.
> [0.927215] PCI-DMA: aperture base @ b400 size 65536 KB
> [0.927336] PCI-DMA: using GART IOMMU.
> [0.927448] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture

Corbin

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Adam Carter]

Answer #1 :

>
> Asrock, Extreme6, 990FX, UEFI, AMD FX-9590
> Gigabyte, GA-MA790FX-UD5P, 790FX, BIOS, AMD PhenomII x4 980
>
>
Hi Corbin,

I noticed i didnt have x2apic enabled on my 990FX / FX-8350 system, so i've
rebuilt/rebooted etc, but;

# zgrep X2APIC /proc/config.gz
CONFIG_X86_X2APIC=y
# dmesg | grep -i x2apic
#

On my skylake box
# dmesg | grep -i x2apic
[0.044148] DMAR-IR: Queued invalidation will be enabled to support
x2apic and Intr-remapping.
[0.045802] DMAR-IR: Enabled IRQ remapping in x2apic mode
[0.045908] x2apic enabled
[0.046011] Switched APIC routing to cluster x2apic.

What x2apic messages do you get on your 990FX / FX-9590 system?

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[taii...@gmx.com]

On 05/15/2017 10:59 PM, taii...@gmx.com wrote:


On 05/15/2017 12:58 AM, Ian Zimmerman wrote:


The 990FX / 790FX Chipset doesn't have a GART / AGP Aperature or IOMMU
in it.  The CPU contains the original K8 IOMMU ( v1 ) engineered /
converted from a GART.


The 8 and 9 series (not 7) does have an IOMMU, AMD-Vi v1.26.

I have two 890 series (but the server flavor - SR5690 chipset) boards 
with an IOMMU and it works great, I play games in a VM with an 
attached graphics card on my libre firmware KGPE-D16 and devices are 
DMA restricted.


Most consumer boards don't properly implement this feature, in fact I 
have never seen one that did which is why I bought my coreboot (libre 
init variant) D16.
It wouldn't be that difficult to port coreboot to your board if you 
want this to work FYI.



To be clear, it is present on the northbridge chipset not the CPU - AMD 
documentation backs this up.

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[taii...@gmx.com]

On 05/15/2017 12:58 AM, Ian Zimmerman wrote:


The 990FX / 790FX Chipset doesn't have a GART / AGP Aperature or IOMMU
in it.  The CPU contains the original K8 IOMMU ( v1 ) engineered /
converted from a GART.


The 8 and 9 series (not 7) does have an IOMMU, AMD-Vi v1.26.

I have two 890 series (but the server flavor - SR5690 chipset) boards 
with an IOMMU and it works great, I play games in a VM with an attached 
graphics card on my libre firmware KGPE-D16 and devices are DMA restricted.


Most consumer boards don't properly implement this feature, in fact I 
have never seen one that did which is why I bought my coreboot (libre 
init variant) D16.
It wouldn't be that difficult to port coreboot to your board if you want 
this to work FYI.

[gentoo-user] Re: Issues with AMD_IOMMU

[Ian Zimmerman]

On 2017-05-15 14:33, Corbin Bird wrote:

> Gigabyte has long been known to ship 'broken for Linux' e820 firmware.
> 'e820' is basically the APG Aperature setup in the firmware.
> Gigabyte kills OR overrides the IOMMU support in the firmware, to setup
> the AGP Aperature.
> 
> Closed source / binary video drivers WANT an AGP Aperature.

So is _that_ what it's all about?  Is the default assumption that I run
non-free video drivers?  Why?  I _never_ use these, and if you bet on me
using them in the future and win the bet, you can probably make a cool
million or two.

> The kernel wants an IOMMU on x86_64, as an IOMMU.

But it has one!  The GART_IOMMU.  It was even added specifically for
Linux, according to kernel.org discussions I remember seeing but can't
locate now.

I also found [1] , which I think is the clearest and most human-oriented
explanation of this issue, yet.  And ... the recommendation is pretty
much the opposite of yours.  The ball is in your court, sir.

> > CONFIG_X86_X2APIC=y

I did this long ago, and it has no effect - still only one is found,
according to dmesg.  As I wrote in the earlier thread.

Also, the Kconfig help for this option says it does something quite
different.  

> > CONFIG_X86_MPPARSE=y

Ok, this one I have not set so far.  Will read up on it and try,

[1]
http://bogdan.org.ua/2009/09/30/iommu-this-costs-you-64-mb-of-ram.html

-- 
Please *no* private Cc: on mailing lists and newsgroups
Personal signed mail: please _encrypt_ and sign
Don't clear-text sign:
http://primate.net/~itz/blog/the-problem-with-gpg-signatures.html

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Corbin Bird]

On 05/14/2017 11:58 PM, Ian Zimmerman wrote:
> On 2017-05-14 20:07, Corbin Bird wrote:
> 
>> Background / System Info : 2 systems.
>> Chipset 990FX, FX-9590 ( -march=bdver2 / Fam15h )
>> Chipset 790FX, PhenomII 980 ( -march=amdfam10 / Fam10h )
>> Gentoo x86_64, multilib, Kernel 4.9.x
>> IOMMU enabled in UEFI, BIOS.
> 
>> General Info : ( if you don't know already ... )
>>
>> The 990FX / 790FX Chipset doesn't have a GART / AGP Aperature or IOMMU
>> in it.  The CPU contains the original K8 IOMMU ( v1 ) engineered /
>> converted from a GART.
>>
>> Standard output in '/var/log/dmesg'.
>>
>>> [0.00] AGP: No AGP bridge found
>>> [0.926911] PCI-DMA: Disabling AGP.
>>> [0.927142] PCI-DMA: aperture base @ b400 size 65536 KB
>>> [0.927263] PCI-DMA: using GART IOMMU.
>>> [0.927376] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
> 
> I know we have been over this in an earlier thread "Another x11
> problem", but I am still very confused.
> 
> What BIOS do you have, on the computer that is BIOS based?  I assume you
> have one BIOS based and one EFI based, although you don't really say
> that.  I am asking because the setup screen of my BIOS (Award modular)
> doesn't mention any of these things at all.
> 
> Another source of my confusion is that my computer appears to work _now_
> without setting any additional kernel options.  The problem I had (X11
> turning the screen all black and never coming back) seems to have fixed
> itself with an upgrade of either kernel, X11, or radeon firmware, I
> don't remember which (and I probably never knew for sure which).
> 
> There are some 2 or 3 BIOS upgrades for my motherboard (GA-880GM-D2H)
> but they are all ancient anyway, from 2011 - the same year I built the
> computer, and the description Gigabyte gives sounds irrelevant to this
> stuff, so I never changed the BIOS.
> 
> I have looked at the kernel docs you recommend, but they leave a lot to
> be desired in the clarity department.  For example, the general
> kernel-parameters doc lists a iommu=pt option, without explanation but
> tagged with [X86][IA-64].  But the x86_64/boot-options doc doesn't
> mention this value at all.
> 
> I guess my high-level confusion is that I don't understand the
> relationships between all these bits of hardware, and I also don't
> understand why the automatic setup that Linux does wouldn't result in a
> normally working system.  So there's no aperture, so steal 64M of
> addresses from RAM.  Big deal, no?  What's the problem?
> 
> Finally, in your last article in that earlier thread you said that I
> should turn on multiple APICs in the kernel.  But I can find no kernel
> option for that.  What's the exact CONFIG_* symbol to tweak?
> 
> Thanks.
> 



Answer #1 :

Asrock, Extreme6, 990FX, UEFI, AMD FX-9590
Gigabyte, GA-MA790FX-UD5P, 790FX, BIOS, AMD PhenomII x4 980

Update the BIOS on that motherboard. Gigabyte doesn't state everything
done by their BIOS updates. The IOMMU enable option was added to the
BIOS at a later date on the GA-MA790FX-UD5P.

Gigabyte has long been known to ship 'broken for Linux' e820 firmware.
'e820' is basically the APG Aperature setup in the firmware.
Gigabyte kills OR overrides the IOMMU support in the firmware, to setup
the AGP Aperature.

Closed source / binary video drivers WANT an AGP Aperature.
The kernel wants an IOMMU on x86_64, as an IOMMU.
Squabbling over memory, and I/O, used for what, by whom, yes?

That is why I mentioned 'the monkey wrench'.



Answer #2 :

Here is what to look for, ( x2 APIC answer )

> 
> kernel 4.9.x, make menuconfig ... ( look just under "Symmetric 
> multi-processing support" )
> 
>  Processor type and features  ---> 
>   │ │  [*] Symmetric 
> multi-processing support  
>  │ │  
>   │ │  [*] Support x2apic 
>   
> │ │  
>   │ │  [*] Enable MPS 
> table 
> │ │  
>   │ │  [ ] Support for 
> extended (non-PC) x86 platforms   
>│ │  
>   │ │  [ ] Intel Low 
> Power Subsystem Support   
>  │ │  
>   │ │  [ ] AMD 
> ACPI2Platform devices support 
>│ │  
>   │ │  < > Intel SoC IOSF 
> Sideband 

[gentoo-user] Re: Issues with AMD_IOMMU

[Ian Zimmerman]

On 2017-05-14 20:07, Corbin Bird wrote:

> Background / System Info : 2 systems.
> Chipset 990FX, FX-9590 ( -march=bdver2 / Fam15h )
> Chipset 790FX, PhenomII 980 ( -march=amdfam10 / Fam10h )
> Gentoo x86_64, multilib, Kernel 4.9.x
> IOMMU enabled in UEFI, BIOS.

> General Info : ( if you don't know already ... )
> 
> The 990FX / 790FX Chipset doesn't have a GART / AGP Aperature or IOMMU
> in it.  The CPU contains the original K8 IOMMU ( v1 ) engineered /
> converted from a GART.
> 
> Standard output in '/var/log/dmesg'.
> 
> > [0.00] AGP: No AGP bridge found
> > [0.926911] PCI-DMA: Disabling AGP.
> > [0.927142] PCI-DMA: aperture base @ b400 size 65536 KB
> > [0.927263] PCI-DMA: using GART IOMMU.
> > [0.927376] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture

I know we have been over this in an earlier thread "Another x11
problem", but I am still very confused.

What BIOS do you have, on the computer that is BIOS based?  I assume you
have one BIOS based and one EFI based, although you don't really say
that.  I am asking because the setup screen of my BIOS (Award modular)
doesn't mention any of these things at all.

Another source of my confusion is that my computer appears to work _now_
without setting any additional kernel options.  The problem I had (X11
turning the screen all black and never coming back) seems to have fixed
itself with an upgrade of either kernel, X11, or radeon firmware, I
don't remember which (and I probably never knew for sure which).

There are some 2 or 3 BIOS upgrades for my motherboard (GA-880GM-D2H)
but they are all ancient anyway, from 2011 - the same year I built the
computer, and the description Gigabyte gives sounds irrelevant to this
stuff, so I never changed the BIOS.

I have looked at the kernel docs you recommend, but they leave a lot to
be desired in the clarity department.  For example, the general
kernel-parameters doc lists a iommu=pt option, without explanation but
tagged with [X86][IA-64].  But the x86_64/boot-options doc doesn't
mention this value at all.

I guess my high-level confusion is that I don't understand the
relationships between all these bits of hardware, and I also don't
understand why the automatic setup that Linux does wouldn't result in a
normally working system.  So there's no aperture, so steal 64M of
addresses from RAM.  Big deal, no?  What's the problem?

Finally, in your last article in that earlier thread you said that I
should turn on multiple APICs in the kernel.  But I can find no kernel
option for that.  What's the exact CONFIG_* symbol to tweak?

Thanks.

-- 
Please *no* private Cc: on mailing lists and newsgroups
Personal signed mail: please _encrypt_ and sign
Don't clear-text sign:
http://primate.net/~itz/blog/the-problem-with-gpg-signatures.html