Hi,
I am using VFIO-PCI to pass through a PCIe endpoint on an FPGA card to virtual
x86 and aarch64 QEMU instances. It works fine on x86 but I have problems with
aarch64. On aarch64, memory for both BARs on the device shows up as disabled,
and one BAR is ignored. From this
presentation<https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwi-vvyD1_TPAhVIy2MKHZmCCt4QFggcMAA&url=https%3A%2F%2Fwww.linux-kvm.org%2Fimages%2Fb%2Fb4%2F2012-forum-VFIO.pdf&usg=AFQjCNFpT9XRMMrhxR0U7_y1tIqXNp13ww&sig2=-4egMzF5EO-DRy5BJizgvg&bvm=bv.136593572,d.cGc>,
I'm under the impression that I can remap PCI devices without KVM to a virtual
aarch64 machine on an x86 host. Is this possible? Below are details for my
problem:
Host is running Ubuntu Xenial, 4.4 Kernel.
Both virtual machines are also Ubuntu Xenial, 4.4 Kernel, from Ubuntu Cloud
Images. Except for architecture, they are pretty much identical.
Qemu is 2.7.50 from git.
Command line to start qemu for x86 machine:
qemu-system-x86_64 -enable-kvm -net nic -net user -hda disk.img -hdb
my-seed.img -m 1024 -smp 2 -device
vfio-pci,host=01:00.0,addr=09.0,multifunction=on -redir tcp:2223::22
Lspci -vvv for remapped device in x86 machine. Note both regions are enabled.
00:09.0 Memory controller: Xilinx Corporation Device 7022
Subsystem: Xilinx Corporation Device 0007
Physical Slot: 9
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 10
Region 0: Memory at e0071000 (32-bit, non-prefetchable) [size=4K]
Region 1: Memory at c0000000 (32-bit, non-prefetchable)
[size=512M]
Capabilities: [80] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0
PME-
Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [c0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes,
PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd-
RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal+
Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc-
AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq
512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x8, ASPM
not supported, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep-
BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled-
CommClk+
ExtSynch- ClockPM- AutWidDis-
BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train-
SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B,
TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 65ms to 210ms,
TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-,
EqualizationPhase3-, LinkEqualizationRequest-
Command line to start qemu for aarch64 machine:
qemu-system-aarch64 -smp 2 -m 2048 -M virt -bios QEMU_EFI.fd -device
virtio-blk-device,drive=image -drive if=none,id=image,file=disk.img -device
virtio-blk-device,drive=cloud -drive if=none,id=cloud,file=cloud.img -netdev
user,id=user0 -device virtio-net-device,netdev=user0 -redir tcp:2222::22 -cpu
cortex-a57 -device vfio-pci,host=01:00.0,addr=09.0,multifunction=on
Lspci -vvv for remapped device in aarch64 machine. Note both regions are
disabled, one is ignored.
00:09.0 Memory controller: Xilinx Corporation Device 7022
Subsystem: Xilinx Corporation Device 0007
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 47
Region 0: Memory at 10000000 (32-bit, non-prefetchable)
[disabled] [size=4K]
Region 1: Memory at <ignored> (32-bit, non-prefetchable)
[disabled]
Capabilities: [80] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0
PME-
Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [c0] Express (v2) Root Complex Integrated
Endpoint, MSI 00
DevCap: MaxPayload 512 bytes,
PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal+
Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc-
AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq
512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr- TransPend-
DevCap2: Completion Timeout: Range B,
TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 65ms to 210ms,
TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+
UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover-
Timeout- NonFatalErr-
CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+
Timeout+ NonFatalErr+
AERCap: First Error Pointer: 00,
GenCap- CGenEn- ChkCap- ChkEn-
On the aarch64 machine, when I rescan the PCI bus, I see the following in dmesg:
[ 365.482929] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000
[ 365.521023] pci 0000:00:09.0: [10ee:7022] type 00 class 0x058000
[ 365.522971] pci 0000:00:09.0: reg 0x10: [mem 0x10000000-0x10000fff]
[ 365.523213] pci 0000:00:09.0: reg 0x14: [mem 0x80000000-0x9fffffff]
[ 365.539233] pci 0000:00:09.0: BAR 1: no space for [mem size 0x20000000]
[ 365.539406] pci 0000:00:09.0: BAR 1: failed to assign [mem size 0x20000000]
[ 365.539853] pci 0000:00:09.0: BAR 0: assigned [mem 0x10000000-0x10000fff]
Is 512MB too much for aarch64? What is the limit? I tried remapping another PCI
device with only a single 16kB BAR to the aarch64 machine and that showed as
disabled too. How do I enable the memory regions on aarch64? There are no
drivers for these devices loaded yet on either x86 or aarch64 yet the memory
shows as enable don x86.
Best Regards,
Steve Haynal
_______________________________________________
vfio-users mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/vfio-users
