Re: dmar pte read access not set error messages on hp dl388 gen8 systems

Thu, 05 Dec 2019 17:54:04 -0800 

Hi,

On 12/5/19 10:53 AM, Jerry Snitselaar wrote:

On Thu Dec 05 19, Lu Baolu wrote:

Hi,

On 12/5/19 10:25 AM, Jerry Snitselaar wrote:


It seems that iommu pci bus probe didn't enumerate device [01:00.2] and
[02:00.0], the corresponding context entries were not setup. Hence dma
fault generated when devices access the memory.

Do these two devices show in "lspci" output? How do these devices get
enumerated by the system?

Best regards,
baolu



They are there in the output, but it seems out of order:



[   23.446201] pci 0000:01:00.0: Adding to iommu group 25
[   23.448949] pci 0000:01:00.0: Using iommu dma mapping
[   23.450807] pci 0000:01:00.1: Adding to iommu group 25
[   23.452666] pci 0000:01:00.1: DMAR: Device uses a private identity domain. 
[   23.455063] pci 0000:01:00.2: Adding to iommu group 25
[   23.456881] pci 0000:01:00.4: Adding to iommu group 25
[   23.458693] pci 0000:01:00.4: DMAR: Device uses a private identity domain. 

Oh, yes!

So device 01:00.0 01:00.1 01:00.2 01:00.4 share a single group. The
default domain for this group has been set to DMA although iommu=pt has
been set. As the result, .0 .2 use DMA, but .1, .4 use IDENTITY. This is
not a valid configuration since all devices in a group should use a same
domain.

Do you mind posting the "lspci -vvv" output of these devices? I want to
figure out why these devices request different domain type.

Best regards,
baolu
01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 05) 
     Subsystem: Hewlett-Packard Company iLO4
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-     Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- 
     Latency: 0, Cache Line Size: 64 bytes
     Interrupt: pin A routed to IRQ 19
     NUMA node: 0
     Region 0: I/O ports at 3000 [size=256]
     Region 1: Memory at f7bf0000 (32-bit, non-prefetchable) [size=512]
     Region 2: I/O ports at 3400 [size=256]
     Capabilities: [78] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) 
         Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
     Capabilities: [b0] MSI: Enable- Count=1/1 Maskable- 64bit+
         Address: 0000000000000000  Data: 0000
     Capabilities: [c0] Express (v1) Legacy Endpoint, MSI 00
        DevCap:    MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited 
             ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
        DevCtl:    Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported- 
             RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
             MaxPayload 128 bytes, MaxReadReq 128 bytes
        DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-         LnkCap:    Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Exit Latency L0s <4us 
             ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
         LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk-
             ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- 
     Kernel driver in use: hpwdt
     Kernel modules: hpwdt
01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH (prog-if 00 [VGA controller]) 
     Subsystem: Hewlett-Packard Company iLO4
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx-     Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- 
     Latency: 0, Cache Line Size: 64 bytes
     Interrupt: pin B routed to IRQ 16
     NUMA node: 0
     Region 0: Memory at f5000000 (32-bit, prefetchable) [size=16M]
     Region 1: Memory at f7be0000 (32-bit, non-prefetchable) [size=16K]
     Region 2: Memory at f7000000 (32-bit, non-prefetchable) [size=8M]
     [virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
     Capabilities: [a8] Power Management version 3
        Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) 
         Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
     Capabilities: [b0] MSI: Enable- Count=1/1 Maskable- 64bit+
         Address: 0000000000000000  Data: 0000
     Capabilities: [c0] Express (v1) Legacy Endpoint, MSI 00
        DevCap:    MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited 
             ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
        DevCtl:    Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported- 
             RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
             MaxPayload 128 bytes, MaxReadReq 128 bytes
        DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-         LnkCap:    Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Exit Latency L0s <4us 
             ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
         LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk-
             ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- 
     Kernel driver in use: mgag200
     Kernel modules: mgag200
01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 05) 
     Subsystem: Hewlett-Packard Company iLO4
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-     Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- 
     Latency: 0, Cache Line Size: 64 bytes
     Interrupt: pin B routed to IRQ 16
     NUMA node: 0
     Region 0: I/O ports at 3800 [size=256]
     Region 1: Memory at f6ff0000 (32-bit, non-prefetchable) [size=256]
     Region 2: Memory at f6e00000 (32-bit, non-prefetchable) [size=1M]
     Region 3: Memory at f6d80000 (32-bit, non-prefetchable) [size=512K]
     Region 4: Memory at f6d70000 (32-bit, non-prefetchable) [size=32K]
     Region 5: Memory at f6d60000 (32-bit, non-prefetchable) [size=32K]
     [virtual] Expansion ROM at f6d00000 [disabled] [size=64K]
     Capabilities: [78] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) 
         Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
     Capabilities: [b0] MSI: Enable- Count=1/1 Maskable- 64bit+
         Address: 0000000000000000  Data: 0000
     Capabilities: [c0] Express (v1) Legacy Endpoint, MSI 00
        DevCap:    MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited 
             ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
        DevCtl:    Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported- 
             RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
             MaxPayload 128 bytes, MaxReadReq 128 bytes
        DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-         LnkCap:    Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Exit Latency L0s <4us 
             ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
         LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk-
             ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- 
     Kernel driver in use: hpilo
     Kernel modules: hpilo
01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02) (prog-if 00 [UHCI]) 
     Subsystem: Hewlett-Packard Company iLO4
    Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx-     Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- 
     Latency: 0, Cache Line Size: 64 bytes
     Interrupt: pin B routed to IRQ 16
     NUMA node: 0
     Region 4: I/O ports at 3c00 [size=32]
     Capabilities: [70] MSI: Enable- Count=1/1 Maskable- 64bit+
         Address: 0000000000000000  Data: 0000
     Capabilities: [80] Express (v1) Legacy Endpoint, MSI 00
        DevCap:    MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited 
             ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
        DevCtl:    Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported- 
             RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
             MaxPayload 128 bytes, MaxReadReq 128 bytes
        DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-         LnkCap:    Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Exit Latency L0s <4us 
             ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
         LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk-
             ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- 
     Capabilities: [f0] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) 
         Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
     Kernel driver in use: uhci_hcd
02:00.0 RAID bus controller: Hewlett-Packard Company Smart Array Gen8 Controllers (rev 01) 
     Subsystem: Hewlett-Packard Company P420i
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+     Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- 
     Latency: 0, Cache Line Size: 64 bytes
     Interrupt: pin A routed to IRQ 52
     NUMA node: 0
     Region 0: Memory at f7d00000 (64-bit, non-prefetchable) [size=1M]
     Region 2: Memory at f7cf0000 (64-bit, non-prefetchable) [size=1K]
     Region 4: I/O ports at 5000 [size=256]
     [virtual] Expansion ROM at f7c00000 [disabled] [size=512K]
     Capabilities: [80] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-) 
         Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
     Capabilities: [90] MSI: Enable- Count=1/32 Maskable- 64bit+
         Address: 0000000000000000  Data: 0000
     Capabilities: [b0] MSI-X: Enable+ Count=64 Masked-
         Vector table: BAR=0 offset=00002000
         PBA: BAR=0 offset=00003000
     Capabilities: [c0] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <1us             ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W         DevCtl:    Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported- 
             RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
             MaxPayload 256 bytes, MaxReadReq 4096 bytes
        DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- 
         LnkCap:    Port #0, Speed 8GT/s, Width x8, ASPM not supported
             ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
         LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk-
             ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 8GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-         DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR+, OBFF Via message 
              AtomicOpsCap: 32bit+ 64bit+ 128bitCAS+
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled 
              AtomicOpsCtl: ReqEn-
         LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- 
              Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+              EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- 
     Capabilities: [100 v2] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-         UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-         UESvrt:    DLP- SDES+ TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- 
         CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
         CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
        AERCap:    First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- 
             MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
         HeaderLog: 00000000 00000000 00000000 00000000
     Capabilities: [300 v1] #19
     Kernel driver in use: hpsa
     Kernel modules: hpsa



Can you please try below change? Let's check whether the afending
address has been mapped for device 01.00.2.

$ git diff
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index db7bfd4f2d20..d9daf66be849 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -663,6 +663,8 @@ static int iommu_group_create_direct_mappings(struct iommu_group *group, ret = iommu_map(domain, addr, addr, pg_size, entry->prot); 
                        if (ret)
                                goto out;
+
+ dev_info(dev, "Setting identity map [0x%Lx - 0x%Lx] for group %d\n", addr, addr + pg_size, group->id); 
                }

        }

I am doubting that device 01.00.2 is not in the device scope of

[    4.485108] DMAR: RMRR base: 0x000000bdf6f000 end: 0x000000bdf7efff

By the way, does device 01.00.2 works well after binding the driver?

Best regards,
baolu
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Reply via email to