Re: [vfio-users] Direct Assignment of Non-PCI devices to VM

[Auger Eric]

Hi Mica, Roja,

On 11/4/20 9:25 PM, Micah Morton wrote:
> On Wed, Nov 4, 2020 at 5:30 AM Roja Eswaran  wrote:
>>
>> I am using Jetson Xavier NX which integrates a Realtek RTL8211FDI Gigabit 
>> Ethernet controller.  The on-module Ethernet controller supports 10/100/1000 
>> Gigabit Ethernet IEEE 802.3u Media Access Controller (MAC).
>>  I am trying to assign Non-PCI NIC mentioned above directly to the Vanilla 
>> VM. As it's not a PCI device, I have no idea how I can achieve this.
>>
>> 1) Is it possible to do a direct assignment on Non-PCI devise to VM using 
>> VFIO drivers?
> 
> I don't see any mention of an IOMMU in any of these 3 DT files:
> 
> https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2972-.dts
> https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi
> https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194.dtsi

Jason AGX Xavier seems to have one but I don't know if it is a different
SOC.
https://forums.developer.nvidia.com/t/pcie-smmu-issue/67027
https://en.miivii.com/uploads/file/20200511/NV_Jetson_Xavier_NX_DataSheet_v0.1.pdf
> 
> Does the Jetson Xavier NX board have an iommu? If not, this is the
> most fundamental reason you wouldn't be able to use VFIO for
> assignment on that board -- especially if the ethernet controller is
> DMA-capable, since you won't be able to ensure the ethernet device's
> DMAs operate on the memory assigned to the guest.
> 
> You could check for this by doing "dmesg | grep -i -e iommu -e smmu"
> on the device and seeing if it prints anything, or looking at
> /sys/kernel/iommu_groups to see if that dir exists.
> 
> Beyond the IOMMU, vfio-platform/QEMU in their current state wouldn't
> be able to handle the embedded PHY device here:
> https://elixir.bootlin.com/linux/latest/source/arch/arm64/boot/dts/nvidia/tegra194-p2888.dtsi#L43
> , which would be another hurdle.
wondering if the phy-mode couldn' be changed or if some interaction with
the phy couldn't be managed in the vfio-platform reset driver.

As micah said SMMU is a must have for VFIO device assignment. A basic
reset module must be written on kernel side and in QEMU, if you use this
latter, you must generate a device tree node for the guest. To evaluate
the complexity of this work you should look at the dt on the host side
and look at the ethernet node.

Please look at /proc/device-tree or /sys/firmware/devicetree/ to find
out your device node.

Thanks

Eric
> 
>> 2) If not, are there any other drivers that would help me achieve the direct 
>> assignment?
>> The direct assignment is important because we are trying to reduce the CPU 
>> utilization even though virtio drivers give us a very small network latency 
>> overhead. Any comments or insights are really appreciated. Thank you so much 
>> for your time!
>>
>>
>> --
>> Thanks,
>> Roja Eswaran
>> Ph.D. Student, Operating Systems and Networks Laboratory
>> State University of New York, Binghamton
>> ___
>> vfio-users mailing list
>> vfio-users@redhat.com
>> https://www.redhat.com/mailman/listinfo/vfio-users
> 

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] location of regions VFIO_REGION_TYPE_MIGRATION/VFIO_REGION_SUBTYPE_MIGRATION

[Auger Eric]

Hi Thanos,

On 10/2/20 4:04 PM, Thanos Makatos wrote:
>> -Original Message-
>> From: Auger Eric 
>> Sent: 02 October 2020 14:56
>> To: Thanos Makatos ; vfio-
>> us...@redhat.com
>> Cc: John G Johnson 
>> Subject: Re: [vfio-users] location of regions
>> VFIO_REGION_TYPE_MIGRATION/VFIO_REGION_SUBTYPE_MIGRATION
>>
>> Hi Thanos,
>>
>> On 10/2/20 2:54 PM, Thanos Makatos wrote:
>>> According to linux/include/uapi/linux/vfio.h, for a device to support
>> migration
>>> it must provide a VFIO capability of type VFIO_REGION_INFO_CAP_TYPE
>> and set
>>> .type/.subtype to
>> VFIO_REGION_TYPE_MIGRATION/VFIO_REGION_TYPE_MIGRATION.
>>>
>>> What confuses me is the following:
>>>
>>> "The structure vfio_device_migration_info is placed at the 0th offset of
>>> the VFIO_REGION_SUBTYPE_MIGRATION region to get and set VFIO
>> device related
>>> migration information."
>>>
>>> Where do regions VFIO_REGION_TYPE_MIGRATION and
>> VFIO_REGION_SUBTYPE_MIGRATION
>>> live on the device's address space? Is just like any other region, except 
>>> that
>>> in the case of a PCI device it's index must be equal to, or larger than,
>>> VFIO_PCI_NUM_REGIONS, and set arbitrarily by the device
>> implementation? If
>>> so, then I suppose this index must appear in struct
>> vfio_device_info.num_regions?
>>
>> You need to call VFIO_DEVICE_GET_REGION_INFO with the region
>> type/subtype. The index of the region is stored in struct
>> vfio_region_info index.
> 
> Are you talking from perspective of the user or the driver? I want to know how
> to support migration in the driver. VFIO_DEVICE_GET_REGION_INFO doesn't take a
> type/subtype argument, it only takes the region index and the driver fills in
> the rest of the information.
> 
Yep sorry I was talking about the user perspective and rather looked at
the QEMU helpers. Iterate on all regions using
VFIO_DEVICE_GET_REGION_INFO. Test the retrieved info has the
VFIO_REGION_INFO_CAP_TYPE cap. If it has, compare
vfio_region_info_cap_type type/subtype against the one you look for. If
it has this is the one you look for and you have the index.

In the kernel such region is added using vfio_pci_register_dev_region()

Hope this clarifies

Eric


___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] location of regions VFIO_REGION_TYPE_MIGRATION/VFIO_REGION_SUBTYPE_MIGRATION

[Auger Eric]

Hi Thanos,

On 10/2/20 2:54 PM, Thanos Makatos wrote:
> According to linux/include/uapi/linux/vfio.h, for a device to support 
> migration
> it must provide a VFIO capability of type VFIO_REGION_INFO_CAP_TYPE and set
> .type/.subtype to VFIO_REGION_TYPE_MIGRATION/VFIO_REGION_TYPE_MIGRATION.
> 
> What confuses me is the following:
> 
> "The structure vfio_device_migration_info is placed at the 0th offset of
> the VFIO_REGION_SUBTYPE_MIGRATION region to get and set VFIO device related
> migration information."
> 
> Where do regions VFIO_REGION_TYPE_MIGRATION and VFIO_REGION_SUBTYPE_MIGRATION
> live on the device's address space? Is just like any other region, except that
> in the case of a PCI device it's index must be equal to, or larger than,
> VFIO_PCI_NUM_REGIONS, and set arbitrarily by the device implementation? If
> so, then I suppose this index must appear in struct 
> vfio_device_info.num_regions?

You need to call VFIO_DEVICE_GET_REGION_INFO with the region
type/subtype. The index of the region is stored in struct
vfio_region_info index.

Thanks

Eric
> 
> 
> ___
> vfio-users mailing list
> vfio-users@redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
> 

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] "Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff" when attempting to dump PCI ROM

[Auger Eric]

Hi Nicolas, Alex,

On 3/11/19 9:39 AM, Auger Eric wrote:
> Hi Nicolas,
> 
> On 3/10/19 11:06 PM, Nicolas Roy-Renaud wrote:
>> I've seen a lot of people before reccomand VFIO newcomers to flash their
>> GPU if they couldn't get their passthrough working right before, and
>> since I know how potentially risky and avoidable this sort of procedure
>> is (since QEMU lets you just pass your own ROM to the VM to be used
>> instead), and while attempting to go through the steps myself
>> <http://vfio.blogspot.com/2014/08/does-my-graphics-card-rom-support-efi.html>
>> (even though I've had a working VFIO setup for years), got something
>> unexpected.
>>
>> Attempting to dump the ROM from my guest card _/freshly /_/_after a
>> reboot_/ results in the following error message :
>>
>> cat: '/sys/bus/pci/devices/:07:00.0/rom': Input/output error
>>
>> Accompanied by the following like in dmesg :
>>
>> [ 1734.316429] vfio-pci :07:00.0: Invalid PCI ROM header signature: 
>> expecting 0xaa55, got 0x
> 
> Last time I got this error was when a readw was attempted in
> pci_map_rom/pci_get_rom_size while the Device Control memory space
> access bit was disabled in PCI cfg space. At the time you attempt to
> dump the rom, the guest may have disabled this bit?

Sorry I missed your intermediate replies. So it looks the same issue.

Thanks

Eric
> 
> Thanks
> 
> Eric
>>
>> This isn't a recent thing, I started investigating this several months
>> ago
>> <https://www.reddit.com/r/VFIO/comments/971umf/is_everyone_else_getting_invalid_pci_rom_header/>
>> and it turned out that I wasn't alone: Everyone on the VFIO discord chat
>> who actually tried this got the same error message.
>>
>> The interesting part is that launching and shutting down a VM using that
>> device clears the issue and it then becomes possible to dump the ROM as
>> expected. I've known a number of issues with VFIO and ROMs can be
>> resolved either like this or by attaching the guest GPU on an already
>> live VM (for reasons I admittedly don't quite understand myself), but
>> since getting a ROM dump can be an important part of setting up the VM
>> that will receive the device in the first place, I don't know if this
>> could be considered an option.
>>
>> I figured this had something to do with the host BIOS tainting the ROM
>> on boot, which is a well known thing on boot GPUs (even though the one
>> I'm attempting to dump isn't), and that passing the tainted GPU to a VM
>> then causes the device to power cycle and clears that, somehow? With
>> that in mind, I've attempted a number of things, but neither resetting
>> the PCI port the device is attached to
>> <https://unix.stackexchange.com/a/474378>, nor doing function-reset on
>> the device, nor going in and out of sleep mode on the guest seems to
>> resolve the issue. I've also tried using pci-stub instead of vfio-pci to
>> see if this wasn't a driver issue, but no success here either.
>>
>> Is there anyone with better technical knowledge of VFIO who could help
>> with this?
>>
>> - Nicolas
>>
>>
>> ___
>> vfio-users mailing list
>> vfio-users@redhat.com
>> https://www.redhat.com/mailman/listinfo/vfio-users
>>
> 
> ___
> vfio-users mailing list
> vfio-users@redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
> 

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] "Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff" when attempting to dump PCI ROM

[Auger Eric]

Hi Nicolas,

On 3/10/19 11:06 PM, Nicolas Roy-Renaud wrote:
> I've seen a lot of people before reccomand VFIO newcomers to flash their
> GPU if they couldn't get their passthrough working right before, and
> since I know how potentially risky and avoidable this sort of procedure
> is (since QEMU lets you just pass your own ROM to the VM to be used
> instead), and while attempting to go through the steps myself
> 
> (even though I've had a working VFIO setup for years), got something
> unexpected.
> 
> Attempting to dump the ROM from my guest card _/freshly /_/_after a
> reboot_/ results in the following error message :
> 
> cat: '/sys/bus/pci/devices/:07:00.0/rom': Input/output error
> 
> Accompanied by the following like in dmesg :
> 
> [ 1734.316429] vfio-pci :07:00.0: Invalid PCI ROM header signature: 
> expecting 0xaa55, got 0x

Last time I got this error was when a readw was attempted in
pci_map_rom/pci_get_rom_size while the Device Control memory space
access bit was disabled in PCI cfg space. At the time you attempt to
dump the rom, the guest may have disabled this bit?

Thanks

Eric
> 
> This isn't a recent thing, I started investigating this several months
> ago
> 
> and it turned out that I wasn't alone: Everyone on the VFIO discord chat
> who actually tried this got the same error message.
> 
> The interesting part is that launching and shutting down a VM using that
> device clears the issue and it then becomes possible to dump the ROM as
> expected. I've known a number of issues with VFIO and ROMs can be
> resolved either like this or by attaching the guest GPU on an already
> live VM (for reasons I admittedly don't quite understand myself), but
> since getting a ROM dump can be an important part of setting up the VM
> that will receive the device in the first place, I don't know if this
> could be considered an option.
> 
> I figured this had something to do with the host BIOS tainting the ROM
> on boot, which is a well known thing on boot GPUs (even though the one
> I'm attempting to dump isn't), and that passing the tainted GPU to a VM
> then causes the device to power cycle and clears that, somehow? With
> that in mind, I've attempted a number of things, but neither resetting
> the PCI port the device is attached to
> , nor doing function-reset on
> the device, nor going in and out of sleep mode on the guest seems to
> resolve the issue. I've also tried using pci-stub instead of vfio-pci to
> see if this wasn't a driver issue, but no success here either.
> 
> Is there anyone with better technical knowledge of VFIO who could help
> with this?
> 
> - Nicolas
> 
> 
> ___
> vfio-users mailing list
> vfio-users@redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
> 

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users