Re: [Qemu-devel] [PATCH v2 0/2] Add global device ID in virt machine

[Diana Madalina Craciun]

On 08/01/2017 05:05 AM, Michael S. Tsirkin wrote:
> On Mon, Jul 31, 2017 at 03:13:09PM +, Diana Madalina Craciun wrote:
>> On 07/31/2017 05:06 PM, Michael S. Tsirkin wrote:
>>> On Mon, Jul 31, 2017 at 01:22:45PM +, Diana Madalina Craciun wrote:
>> If we are to use a value of 0 for the constant in case of PCI devices,
>> what happens if we have multiple PCI controllers?
> I guess we'd use the PCI Segment number for that?
>
>
 Yes, we can use the PCI segment for this scenario. But this would mean
 different solutions for the same problem. The main problem is that we
 can have multiple entities in the system that are using MSIs (for now
 PCI and NXP non-PCI bus infrastructure
 (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F635905%2F=01%7C01%7Cdiana.craciun%40nxp.com%7C6b0c6c879af64718a21908d4d81d534e%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0=bpYMMqajWzgzdbdQgy%2FUYR7y%2BswyvwE%2BqFzs7wdIkkA%3D=0).
  I guess that we may have other
 platform devices that are using MSIs in the future.

 Thanks,
 Diana


>>> Don't have the time to explore NXP in depth, sorry - there's
>>> a lot of complexity there.
>>> Could you maybe stick some bits to specify bus type in there?
>>> It just looks very wrong to push low level things like this
>>> that users have no interest in up the stack.
>>>
>> Let's generalize the problem a little bit, the NXP details just does not
>> matter much. The problem we have is the following:
>>
>> The GIC-ITS, the ARM MSI controller is using deviceIDs in order to remap
>> the interrupts. Each device which is expected to send MSIs has a
>> deviceID associated with it. These deviceIDs are configured into devices
>> by software/firmware. There is support in the device tree to specify the
>> correlation between requesterID and deviceID:
>>
>> "msi-map: Maps a Requester ID to an MSI controller and associated
>>   msi-specifier data. The property is an arbitrary number of tuples of
>>   (rid-base,msi-controller,msi-base,length)"
>> (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git%2Ftree%2FDocumentation%2Fdevicetree%2Fbindings%2Fpci%2Fpci-msi.txt=01%7C01%7Cdiana.craciun%40nxp.com%7C259982f3a83b4160083708d4d881d48a%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0=QSRwrYcsTk9%2FgekD0YBYKvpWv05%2BXKiH2abPp%2FtHpes%3D=0)
>>
>> Our problem is that we have to allocate these deviceIDs in QEMU as well
>> and we have to ensure that they are unique. Currently, for PCI, the
>> assumption requesterID=deviceID is made which will no longer be true in
>> case other devices are added. So we need a way (preferable a general
>> one) to allocate these IDs to different devices in the system in a
>> consistent way which will ensure that two devices do not share the same ID.
> My question would be, do other types of devices that are there
> right now have some kind of ID like the requester ID?
> If so I would say just use that, and set high bits in the device ID
> to specify the type (e.g. 00 for pci, etc).

Would this mean extending the PCI requesterID usage as well? Or assume
that it's 00 for PCI and no changes are needed? I understand that with
this approach the allocation is  static: 00 for PCI, 01 for device type
X, etc., so the situation when we have multiple controllers (PCI or
whatever other devices) has to be solved differently. Wouldn't be this
more complex?

>
>
> IMHO if possible that is preferable to pushing this up to users.
>
>
>> The reason I put this ID into the controller itself is because on real
>> hardware is actually programmed into the controller. It is needed (for
>> example) when the MSIs are sent.
>>
>> Thanks,
>>
>> Diana
>>
> IIUC what happens on real hardware is controller maps each requester ID
> (or presumably other source ID in the request) to the device ID,
> and the mapping is internal to controller.
> If you wanted a lot of flexibility then looks like you could pass this
> mapping to controllers, but is it really necessary?
> Why don't we build a mapping that's convenient for us?
>
>
Thanks,

Diana

Re: [Qemu-devel] [PATCH v2 0/2] Add global device ID in virt machine

[Michael S. Tsirkin]

On Mon, Jul 31, 2017 at 03:13:09PM +, Diana Madalina Craciun wrote:
> On 07/31/2017 05:06 PM, Michael S. Tsirkin wrote:
> > On Mon, Jul 31, 2017 at 01:22:45PM +, Diana Madalina Craciun wrote:
>  If we are to use a value of 0 for the constant in case of PCI devices,
>  what happens if we have multiple PCI controllers?
> >>> I guess we'd use the PCI Segment number for that?
> >>>
> >>>
> >> Yes, we can use the PCI segment for this scenario. But this would mean
> >> different solutions for the same problem. The main problem is that we
> >> can have multiple entities in the system that are using MSIs (for now
> >> PCI and NXP non-PCI bus infrastructure
> >> (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F635905%2F=01%7C01%7Cdiana.craciun%40nxp.com%7C6b0c6c879af64718a21908d4d81d534e%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0=bpYMMqajWzgzdbdQgy%2FUYR7y%2BswyvwE%2BqFzs7wdIkkA%3D=0).
> >>  I guess that we may have other
> >> platform devices that are using MSIs in the future.
> >>
> >> Thanks,
> >> Diana
> >>
> >>
> > Don't have the time to explore NXP in depth, sorry - there's
> > a lot of complexity there.
> > Could you maybe stick some bits to specify bus type in there?
> > It just looks very wrong to push low level things like this
> > that users have no interest in up the stack.
> >
> Let's generalize the problem a little bit, the NXP details just does not
> matter much. The problem we have is the following:
> 
> The GIC-ITS, the ARM MSI controller is using deviceIDs in order to remap
> the interrupts. Each device which is expected to send MSIs has a
> deviceID associated with it. These deviceIDs are configured into devices
> by software/firmware. There is support in the device tree to specify the
> correlation between requesterID and deviceID:
> 
> "msi-map: Maps a Requester ID to an MSI controller and associated
>   msi-specifier data. The property is an arbitrary number of tuples of
>   (rid-base,msi-controller,msi-base,length)"
> (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings/pci/pci-msi.txt)
> 
> Our problem is that we have to allocate these deviceIDs in QEMU as well
> and we have to ensure that they are unique. Currently, for PCI, the
> assumption requesterID=deviceID is made which will no longer be true in
> case other devices are added. So we need a way (preferable a general
> one) to allocate these IDs to different devices in the system in a
> consistent way which will ensure that two devices do not share the same ID.

My question would be, do other types of devices that are there
right now have some kind of ID like the requester ID?
If so I would say just use that, and set high bits in the device ID
to specify the type (e.g. 00 for pci, etc).


IMHO if possible that is preferable to pushing this up to users.


> The reason I put this ID into the controller itself is because on real
> hardware is actually programmed into the controller. It is needed (for
> example) when the MSIs are sent.
> 
> Thanks,
> 
> Diana
> 

IIUC what happens on real hardware is controller maps each requester ID
(or presumably other source ID in the request) to the device ID,
and the mapping is internal to controller.
If you wanted a lot of flexibility then looks like you could pass this
mapping to controllers, but is it really necessary?
Why don't we build a mapping that's convenient for us?


-- 
MST

Re: [Qemu-devel] [PATCH v2 0/2] Add global device ID in virt machine

[Diana Madalina Craciun]

On 07/31/2017 05:06 PM, Michael S. Tsirkin wrote:
> On Mon, Jul 31, 2017 at 01:22:45PM +, Diana Madalina Craciun wrote:
 If we are to use a value of 0 for the constant in case of PCI devices,
 what happens if we have multiple PCI controllers?
>>> I guess we'd use the PCI Segment number for that?
>>>
>>>
>> Yes, we can use the PCI segment for this scenario. But this would mean
>> different solutions for the same problem. The main problem is that we
>> can have multiple entities in the system that are using MSIs (for now
>> PCI and NXP non-PCI bus infrastructure
>> (https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F635905%2F=01%7C01%7Cdiana.craciun%40nxp.com%7C6b0c6c879af64718a21908d4d81d534e%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0=bpYMMqajWzgzdbdQgy%2FUYR7y%2BswyvwE%2BqFzs7wdIkkA%3D=0).
>>  I guess that we may have other
>> platform devices that are using MSIs in the future.
>>
>> Thanks,
>> Diana
>>
>>
> Don't have the time to explore NXP in depth, sorry - there's
> a lot of complexity there.
> Could you maybe stick some bits to specify bus type in there?
> It just looks very wrong to push low level things like this
> that users have no interest in up the stack.
>
Let's generalize the problem a little bit, the NXP details just does not
matter much. The problem we have is the following:

The GIC-ITS, the ARM MSI controller is using deviceIDs in order to remap
the interrupts. Each device which is expected to send MSIs has a
deviceID associated with it. These deviceIDs are configured into devices
by software/firmware. There is support in the device tree to specify the
correlation between requesterID and deviceID:

"msi-map: Maps a Requester ID to an MSI controller and associated
  msi-specifier data. The property is an arbitrary number of tuples of
  (rid-base,msi-controller,msi-base,length)"
(https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings/pci/pci-msi.txt)

Our problem is that we have to allocate these deviceIDs in QEMU as well
and we have to ensure that they are unique. Currently, for PCI, the
assumption requesterID=deviceID is made which will no longer be true in
case other devices are added. So we need a way (preferable a general
one) to allocate these IDs to different devices in the system in a
consistent way which will ensure that two devices do not share the same ID.

The reason I put this ID into the controller itself is because on real
hardware is actually programmed into the controller. It is needed (for
example) when the MSIs are sent.

Thanks,

Diana

Re: [Qemu-devel] [PATCH v2 0/2] Add global device ID in virt machine

[Michael S. Tsirkin]

On Mon, Jul 31, 2017 at 01:22:45PM +, Diana Madalina Craciun wrote:
> >> If we are to use a value of 0 for the constant in case of PCI devices,
> >> what happens if we have multiple PCI controllers?
> > I guess we'd use the PCI Segment number for that?
> >
> >
> 
> Yes, we can use the PCI segment for this scenario. But this would mean
> different solutions for the same problem. The main problem is that we
> can have multiple entities in the system that are using MSIs (for now
> PCI and NXP non-PCI bus infrastructure
> (https://lwn.net/Articles/635905/). I guess that we may have other
> platform devices that are using MSIs in the future.
> 
> Thanks,
> Diana
> 
> 

Don't have the time to explore NXP in depth, sorry - there's
a lot of complexity there.
Could you maybe stick some bits to specify bus type in there?
It just looks very wrong to push low level things like this
that users have no interest in up the stack.

-- 
MST

Re: [Qemu-devel] [PATCH v2 0/2] Add global device ID in virt machine

[Diana Madalina Craciun]

On 07/06/2017 02:44 AM, Michael S. Tsirkin wrote:
> On Wed, May 31, 2017 at 12:02:56PM +, Diana Madalina Craciun wrote:
>> On 05/25/2017 01:12 AM, Michael S. Tsirkin wrote:
>>> On Tue, May 23, 2017 at 02:12:43PM +0300, Diana Craciun wrote:
 The NXP DPAA2 is a hardware architecture designed for high-speeed network
 packet processing. The DPAA2 hardware components are managed by a hardware
 component called the Management Complex (or MC) which provides an
 object-base abstraction for software drivers to use the DPAA2 hardware.
 For more details you can see: 
 https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git%2Ftree%2Fdrivers%2Fstaging%2Ffsl-mc%2FREADME.txt%3Fh%3Dv4.10=01%7C01%7Cdiana.craciun%40nxp.com%7Cce2cc4d066944ce2759308d4a2f1f3f8%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0=CJAiTF6Qnq4gklSqon7xcRby0O1HQvytTUSdPwaHuSE%3D=0

 The interrupts generated by the DPAA2 hardware components are MSIs. We 
 will add
 support for direct assigning these DPAA2 components/objects to a virtual 
 machine. However, this will add the need to expand the MSI usage in QEMU.

 Currently the MSIs in QEMU are pretty much tied to PCI. For ARM the
 GIC ITS is using a device ID for interrupt translation. Currently, for
 PCI, the requester ID is used as device ID. This will not work when
 we add another entity that needs also a device ID which is supposed to
 be unique across the system.

 My proposal is to add a static allocation in the virt machine. I considered
 that this allocation is specific to each machine/platform. Currently only
 virt machine has it, but other implementations may use the same mechanism
 as well.
 So, I used a static allocation with this formula:

 DeviceID = zero_extend( RequesterID[15:0] ) + 0x1 * Constant

 This formula was taken from SBSA spec (Appendix I: DeviceID generation and
 ITS groups). In case of QEMU the constant will be different for each 
 entity.
 In this way a unique DeviceID will be generated and the device ID will be
 derived from a requesterID (in case of PCI) or other means in case of other
 entities.

 The implementation is generic as there might be in the future other 
 non-pci devices
 that are using MSIs or IOMMU. Any architecture can use it, though currently
 only the ARM architecture is using the function that retrieves the stream 
 ID. I
 did not change all the replacements of the pci_requester_id (with 
 pci_stream_id)
 in the code (although if the constant is 0, the stream_id is equal with 
 requester_id).
 The other architectures (e.g. intel iommu code) assume that the ID is the
 requester ID.

 Tested on NXP LS2080 platform.

 History:
>>> I am confused. I get it that non-PCI things want something else
>>> in their requester ID, but why require it for PCI devices?
>>> How about using Constant == 0 for PCI? This way you do
>>> not need to touch PCI at all as DeviceID == RequesterID ...
>> It is not that other devices need something else in the requester ID,
>> but more about finding a way to provide an unique ID across the system
>> (more precisely it should be unique for all devices connected to the
>> same IOMMU/ITS). The DT already offers support to describe the
>> translation between stream IDs/device IDs to requester ID for PCI
>> devices (iommu-map for IOMMU ([1]) and msi-map for MSIs ([2]). It will
>> not change the way the requester ID is used in PCI in general, but only
>> the places that need a unique ID (which are the MSIs and IOMMU).
>>
>> If we are to use a value of 0 for the constant in case of PCI devices,
>> what happens if we have multiple PCI controllers?
> I guess we'd use the PCI Segment number for that?
>
>

Yes, we can use the PCI segment for this scenario. But this would mean
different solutions for the same problem. The main problem is that we
can have multiple entities in the system that are using MSIs (for now
PCI and NXP non-PCI bus infrastructure
(https://lwn.net/Articles/635905/). I guess that we may have other
platform devices that are using MSIs in the future.

Thanks,
Diana

Re: [Qemu-devel] [PATCH v2 0/2] Add global device ID in virt machine

[Diana Madalina Craciun]

Hi Peter,

On 07/10/2017 08:10 PM, Peter Maydell wrote:

On 23 May 2017 at 12:12, Diana Craciun 
 wrote:


The NXP DPAA2 is a hardware architecture designed for high-speeed network
packet processing. The DPAA2 hardware components are managed by a hardware
component called the Management Complex (or MC) which provides an
object-base abstraction for software drivers to use the DPAA2 hardware.
For more details you can see:
https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git%2Ftree%2Fdrivers%2Fstaging%2Ffsl-mc%2FREADME.txt%3Fh%3Dv4.10=01%7C01%7Cdiana.craciun%40nxp.com%7C87a43122f7374dd4483008d4c7b68d10%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0=Rf9Uae7gxCeBSbLbRyAs24J9R6HUT1p4%2B4K46BgxqvM%3D=0

The interrupts generated by the DPAA2 hardware components are MSIs. We will add
support for direct assigning these DPAA2 components/objects to a virtual
machine. However, this will add the need to expand the MSI usage in QEMU.

Currently the MSIs in QEMU are pretty much tied to PCI. For ARM the
GIC ITS is using a device ID for interrupt translation. Currently, for
PCI, the requester ID is used as device ID. This will not work when
we add another entity that needs also a device ID which is supposed to
be unique across the system.

My proposal is to add a static allocation in the virt machine. I considered
that this allocation is specific to each machine/platform. Currently only
virt machine has it, but other implementations may use the same mechanism
as well.
So, I used a static allocation with this formula:

DeviceID = zero_extend( RequesterID[15:0] ) + 0x1 * Constant

This formula was taken from SBSA spec (Appendix I: DeviceID generation and
ITS groups). In case of QEMU the constant will be different for each entity.
In this way a unique DeviceID will be generated and the device ID will be
derived from a requesterID (in case of PCI) or other means in case of other
entities.

The implementation is generic as there might be in the future other non-pci 
devices
that are using MSIs or IOMMU. Any architecture can use it, though currently
only the ARM architecture is using the function that retrieves the stream ID. I
did not change all the replacements of the pci_requester_id (with pci_stream_id)
in the code (although if the constant is 0, the stream_id is equal with 
requester_id).
The other architectures (e.g. intel iommu code) assume that the ID is the
requester ID.

Tested on NXP LS2080 platform.



So I'm still a bit confused about what this is for and what the
relationship is with how the equivalent task would be performed
in real hardware. Can you explain how that works, please?
Would there be a system MMU involved somewhere that would be
allocating these IDs somehow?

Let's keep the discussion to MSIs for now (assuming that there is no SMMU as 
QEMU does not have currently an SMMU emulation integrated).
Every device that sends MSIs has a deviceID associated with it. The deviceID is 
used by the GIC-ITS hardware for interrupt translation. The deviceID is 
assigned by the "system designer", so there is not an hardware entity that 
generates/allocates the deviceIDs (the device ID can be programmed into the 
device by software).

For PCI these IDs are specified in the device tree:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings/pci/pci-msi.txt

(msi-map property).



The patchset also seems to be introducing a fair amount of
mechanism without also implementing the actual use for it -- yes,
if we had two PCI controllers in the virt board we would need to
sort something out I guess, but we don't, we only have one.
Evaluating whether the mechanism makes sense without its major
user is always trickier.

Right, we have only one PCI controller in the QEMU for now. We are in the 
process of implementing device pass-through for our non-PCI bus infrastructure 
which is using MSIs as well. As Mike pointed out, the mc-bus infrastructure is 
on the point of getting out of staging:

"
The fsl-mc is a hardware resource manager that manages specialized hardware 
objects used in network-oriented packet processing applications. After the 
fsl-mc block is enabled, pools of hardware resources are available, such as 
queues, buffer pools, I/O interfaces. These resources are building blocks that 
can be used to create functional hardware objects such as network interfaces, 
crypto accelerator instances, or L2 switches.

All the fsl-mc managed hardware resources/objects are represented in a physical 
grouping mechanism called a 'container' or DPRC (data path resource container).

>From the point of view of an OS, a DPRC functions similar to a plug and play 
>bus. Using fsl-mc commands software can enumerate the contents of the DPRC 
>discovering the hardware objects present and binding them to drivers. Hardware 
>objects can be created and 

Re: [Qemu-devel] [PATCH v2 0/2] Add global device ID in virt machine

[Mike Caraman]

Peter, thanks for feedback. Diana will return from vacation next week,
I am trying clarify some of the questions.

>From: Peter Maydell <peter.mayd...@linaro.org>
>Sent: Monday, July 10, 2017 8:10 PM
>To: Diana Madalina Craciun
>Cc: QEMU Developers; Michael S. Tsirkin; Eric Auger; Mike Caraman; qemu-arm; 
>Marcel Apfelbaum; Bharat Bhushan; Christoffer Dall; Laurentiu Tudor
>Subject: Re: [Qemu-devel] [PATCH v2 0/2] Add global device ID in virt machine
>
>On 23 May 2017 at 12:12, Diana Craciun <diana.crac...@nxp.com> wrote:
>> The NXP DPAA2 is a hardware architecture designed for high-speeed network
>> packet processing. The DPAA2 hardware components are managed by a hardware
>> component called the Management Complex (or MC) which provides an
>> object-base abstraction for software drivers to use the DPAA2 hardware.
>> For more details you can see:
>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git%2Ftree%2Fdrivers%2Fstaging%2Ffsl-mc%2FREADME.txt%3Fh%3Dv4.10=01%7C01%7Cmike.caraman%40nxp.com%7Ccdb80ed25c4a4a71d86708d4c7b68cd8%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0=JfYha1w2rZ12URBU3LHgo7OwtEPEDqcGWjdAUiRCP4k%3D=0
>>
>> The interrupts generated by the DPAA2 hardware components are MSIs. We will 
>> add
>> support for direct assigning these DPAA2 components/objects to a virtual
>> machine. However, this will add the need to expand the MSI usage in QEMU.
>>
>> Currently the MSIs in QEMU are pretty much tied to PCI. For ARM the
>> GIC ITS is using a device ID for interrupt translation. Currently, for
>> PCI, the requester ID is used as device ID. This will not work when
>> we add another entity that needs also a device ID which is supposed to
>> be unique across the system.
>>
>> My proposal is to add a static allocation in the virt machine. I considered
>> that this allocation is specific to each machine/platform. Currently only
>> virt machine has it, but other implementations may use the same mechanism
>> as well.
>> So, I used a static allocation with this formula:
>>
>> DeviceID = zero_extend( RequesterID[15:0] ) + 0x1 * Constant
>>
>> This formula was taken from SBSA spec (Appendix I: DeviceID generation and
>> ITS groups). In case of QEMU the constant will be different for each entity.
>> In this way a unique DeviceID will be generated and the device ID will be
>> derived from a requesterID (in case of PCI) or other means in case of other
>> entities.
>>
>> The implementation is generic as there might be in the future other non-pci 
>> devices
>> that are using MSIs or IOMMU. Any architecture can use it, though currently
>> only the ARM architecture is using the function that retrieves the stream 
>> ID. I
>> did not change all the replacements of the pci_requester_id (with 
>> pci_stream_id)
>> in the code (although if the constant is 0, the stream_id is equal with 
>> requester_id).
>> The other architectures (e.g. intel iommu code) assume that the ID is the
>> requester ID.
>>
>> Tested on NXP LS2080 platform.
>
>So I'm still a bit confused about what this is for and what the
>relationship is with how the equivalent task would be performed
>in real hardware. Can you explain how that works, please?
>Would there be a system MMU involved somewhere that would be
>allocating these IDs somehow?

This is the first step to accommodate device pass-though with MSI support,
required for non-PCI/heterogeneous buses on ARM64 platforms with GIC-ITS.
We reached the point where our mc-bus infrastructure got the ack and
it is ready to get out of staging:
https://lkml.org/lkml/2017/7/18/708

The problem we want to address is related to guest DeviceIDs allocation
for GIC-ITS emulation. On real hardware the boot loader, in our case u-boot,
allocates the DeviceIDs and populates the PCI msi-map device tree binding.
We have a similar binding mechanism for MC devices:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings/pci/pci-msi.txt
http://git.denx.de/?p=u-boot.git;a=blob;f=drivers/pci/pcie_layerscape_fixup.c;h=9e6c2f5dfcf62e8b06935aa9ae3c40a734f9571d;hb=HEAD#l118

Qemu does not populate yet the msi-map and the DeviceIDs for PCI devices
defaults to bdf at run-time:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/of/irq.c#n599
http://elixir.free-electrons.com/linux/latest/source/drivers/pci/msi.c#L1488

The proposal is to allocate the DeviceIDs as described in SBSA and to populate 
the
device tree, as oppose of generating them at runtime, in order to stick to 
device
binding definitions and to avoid clashes.

>The patchset a

Re: [Qemu-devel] [PATCH v2 0/2] Add global device ID in virt machine

[Peter Maydell]

On 23 May 2017 at 12:12, Diana Craciun  wrote:
> The NXP DPAA2 is a hardware architecture designed for high-speeed network
> packet processing. The DPAA2 hardware components are managed by a hardware
> component called the Management Complex (or MC) which provides an
> object-base abstraction for software drivers to use the DPAA2 hardware.
> For more details you can see:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/fsl-mc/README.txt?h=v4.10
>
> The interrupts generated by the DPAA2 hardware components are MSIs. We will 
> add
> support for direct assigning these DPAA2 components/objects to a virtual
> machine. However, this will add the need to expand the MSI usage in QEMU.
>
> Currently the MSIs in QEMU are pretty much tied to PCI. For ARM the
> GIC ITS is using a device ID for interrupt translation. Currently, for
> PCI, the requester ID is used as device ID. This will not work when
> we add another entity that needs also a device ID which is supposed to
> be unique across the system.
>
> My proposal is to add a static allocation in the virt machine. I considered
> that this allocation is specific to each machine/platform. Currently only
> virt machine has it, but other implementations may use the same mechanism
> as well.
> So, I used a static allocation with this formula:
>
> DeviceID = zero_extend( RequesterID[15:0] ) + 0x1 * Constant
>
> This formula was taken from SBSA spec (Appendix I: DeviceID generation and
> ITS groups). In case of QEMU the constant will be different for each entity.
> In this way a unique DeviceID will be generated and the device ID will be
> derived from a requesterID (in case of PCI) or other means in case of other
> entities.
>
> The implementation is generic as there might be in the future other non-pci 
> devices
> that are using MSIs or IOMMU. Any architecture can use it, though currently
> only the ARM architecture is using the function that retrieves the stream ID. 
> I
> did not change all the replacements of the pci_requester_id (with 
> pci_stream_id)
> in the code (although if the constant is 0, the stream_id is equal with 
> requester_id).
> The other architectures (e.g. intel iommu code) assume that the ID is the
> requester ID.
>
> Tested on NXP LS2080 platform.

So I'm still a bit confused about what this is for and what the
relationship is with how the equivalent task would be performed
in real hardware. Can you explain how that works, please?
Would there be a system MMU involved somewhere that would be
allocating these IDs somehow?

The patchset also seems to be introducing a fair amount of
mechanism without also implementing the actual use for it -- yes,
if we had two PCI controllers in the virt board we would need to
sort something out I guess, but we don't, we only have one.
Evaluating whether the mechanism makes sense without its major
user is always trickier.

Essentially, I know very little about this use case, but my
default position is that if in doubt we should aim to match
how real hardware handles something. Virtualization-specific
code, and especially something specific to a niche use case like
this, is something I'd really prefer to avoid. The ideal is to
have something like PCI device passthrough, where the guest can
automatically probe for things and QEMU doesn't need to get
involved.

I was hoping Eric would review this, since he's more up to speed
on direct hardware passthrough than I am, but I think he's just
gone off on holiday...

thanks
-- PMM

Re: [Qemu-devel] [PATCH v2 0/2] Add global device ID in virt machine

[Michael S. Tsirkin]

On Wed, May 31, 2017 at 12:02:56PM +, Diana Madalina Craciun wrote:
> On 05/25/2017 01:12 AM, Michael S. Tsirkin wrote:
> > On Tue, May 23, 2017 at 02:12:43PM +0300, Diana Craciun wrote:
> >> The NXP DPAA2 is a hardware architecture designed for high-speeed network
> >> packet processing. The DPAA2 hardware components are managed by a hardware
> >> component called the Management Complex (or MC) which provides an
> >> object-base abstraction for software drivers to use the DPAA2 hardware.
> >> For more details you can see: 
> >> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git%2Ftree%2Fdrivers%2Fstaging%2Ffsl-mc%2FREADME.txt%3Fh%3Dv4.10=01%7C01%7Cdiana.craciun%40nxp.com%7Cce2cc4d066944ce2759308d4a2f1f3f8%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0=CJAiTF6Qnq4gklSqon7xcRby0O1HQvytTUSdPwaHuSE%3D=0
> >>
> >> The interrupts generated by the DPAA2 hardware components are MSIs. We 
> >> will add
> >> support for direct assigning these DPAA2 components/objects to a virtual 
> >> machine. However, this will add the need to expand the MSI usage in QEMU.
> >>
> >> Currently the MSIs in QEMU are pretty much tied to PCI. For ARM the
> >> GIC ITS is using a device ID for interrupt translation. Currently, for
> >> PCI, the requester ID is used as device ID. This will not work when
> >> we add another entity that needs also a device ID which is supposed to
> >> be unique across the system.
> >>
> >> My proposal is to add a static allocation in the virt machine. I considered
> >> that this allocation is specific to each machine/platform. Currently only
> >> virt machine has it, but other implementations may use the same mechanism
> >> as well.
> >> So, I used a static allocation with this formula:
> >>
> >> DeviceID = zero_extend( RequesterID[15:0] ) + 0x1 * Constant
> >>
> >> This formula was taken from SBSA spec (Appendix I: DeviceID generation and
> >> ITS groups). In case of QEMU the constant will be different for each 
> >> entity.
> >> In this way a unique DeviceID will be generated and the device ID will be
> >> derived from a requesterID (in case of PCI) or other means in case of other
> >> entities.
> >>
> >> The implementation is generic as there might be in the future other 
> >> non-pci devices
> >> that are using MSIs or IOMMU. Any architecture can use it, though currently
> >> only the ARM architecture is using the function that retrieves the stream 
> >> ID. I
> >> did not change all the replacements of the pci_requester_id (with 
> >> pci_stream_id)
> >> in the code (although if the constant is 0, the stream_id is equal with 
> >> requester_id).
> >> The other architectures (e.g. intel iommu code) assume that the ID is the
> >> requester ID.
> >>
> >> Tested on NXP LS2080 platform.
> >>
> >> History:
> > I am confused. I get it that non-PCI things want something else
> > in their requester ID, but why require it for PCI devices?
> > How about using Constant == 0 for PCI? This way you do
> > not need to touch PCI at all as DeviceID == RequesterID ...
> 
> It is not that other devices need something else in the requester ID,
> but more about finding a way to provide an unique ID across the system
> (more precisely it should be unique for all devices connected to the
> same IOMMU/ITS). The DT already offers support to describe the
> translation between stream IDs/device IDs to requester ID for PCI
> devices (iommu-map for IOMMU ([1]) and msi-map for MSIs ([2]). It will
> not change the way the requester ID is used in PCI in general, but only
> the places that need a unique ID (which are the MSIs and IOMMU).
> 
> If we are to use a value of 0 for the constant in case of PCI devices,
> what happens if we have multiple PCI controllers?

I guess we'd use the PCI Segment number for that?

> [1]
> https://www.kernel.org/doc/Documentation/devicetree/bindings/pci/pci-iommu.txt
> [2]
> https://www.kernel.org/doc/Documentation/devicetree/bindings/pci/pci-msi.txt
> 
> Thanks,
> 
> Diana
> 

Re: [Qemu-devel] [PATCH v2 0/2] Add global device ID in virt machine

[Diana Madalina Craciun]

ping

On 5/23/2017 2:12 PM, Diana Craciun wrote:
> The NXP DPAA2 is a hardware architecture designed for high-speeed network
> packet processing. The DPAA2 hardware components are managed by a hardware
> component called the Management Complex (or MC) which provides an
> object-base abstraction for software drivers to use the DPAA2 hardware.
> For more details you can see: 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/fsl-mc/README.txt?h=v4.10
>
> The interrupts generated by the DPAA2 hardware components are MSIs. We will 
> add
> support for direct assigning these DPAA2 components/objects to a virtual 
> machine. However, this will add the need to expand the MSI usage in QEMU.
>
> Currently the MSIs in QEMU are pretty much tied to PCI. For ARM the
> GIC ITS is using a device ID for interrupt translation. Currently, for
> PCI, the requester ID is used as device ID. This will not work when
> we add another entity that needs also a device ID which is supposed to
> be unique across the system.
>
> My proposal is to add a static allocation in the virt machine. I considered
> that this allocation is specific to each machine/platform. Currently only
> virt machine has it, but other implementations may use the same mechanism
> as well.
> So, I used a static allocation with this formula:
>
> DeviceID = zero_extend( RequesterID[15:0] ) + 0x1 * Constant
>
> This formula was taken from SBSA spec (Appendix I: DeviceID generation and
> ITS groups). In case of QEMU the constant will be different for each entity.
> In this way a unique DeviceID will be generated and the device ID will be
> derived from a requesterID (in case of PCI) or other means in case of other
> entities.
>
> The implementation is generic as there might be in the future other non-pci 
> devices
> that are using MSIs or IOMMU. Any architecture can use it, though currently
> only the ARM architecture is using the function that retrieves the stream ID. 
> I
> did not change all the replacements of the pci_requester_id (with 
> pci_stream_id)
> in the code (although if the constant is 0, the stream_id is equal with 
> requester_id).
> The other architectures (e.g. intel iommu code) assume that the ID is the
> requester ID.
>
> Tested on NXP LS2080 platform.
>
> History:
>
> v1->v2
> --
> - the stream ID was added as a field in the pci device structure in order
> not to traverse the PCI hierarchy each time a MSI is sent.
>
>
> Diana Craciun (2):
>   Increased the size of requester_id field from MemTxAttrs
>   Add a unique ID in the virt machine to be used as device ID
>
>  hw/arm/virt.c  | 26 ++
>  hw/i386/amd_iommu.c|  2 +-
>  hw/i386/intel_iommu.c  |  2 +-
>  hw/intc/arm_gicv3_its_common.c |  2 +-
>  hw/intc/arm_gicv3_its_kvm.c|  2 +-
>  hw/pci-host/gpex.c |  6 ++
>  hw/pci/msi.c   |  2 +-
>  hw/pci/pci.c   | 25 +
>  include/exec/memattrs.h|  4 ++--
>  include/hw/arm/virt.h  |  1 +
>  include/hw/intc/arm_gicv3_its_common.h |  2 +-
>  include/hw/pci-host/gpex.h |  2 ++
>  include/hw/pci/pci.h   |  8 
>  kvm-all.c  |  4 ++--
>  14 files changed, 78 insertions(+), 10 deletions(-)
>

Re: [Qemu-devel] [PATCH v2 0/2] Add global device ID in virt machine

[Diana Madalina Craciun]

On 05/25/2017 01:12 AM, Michael S. Tsirkin wrote:
> On Tue, May 23, 2017 at 02:12:43PM +0300, Diana Craciun wrote:
>> The NXP DPAA2 is a hardware architecture designed for high-speeed network
>> packet processing. The DPAA2 hardware components are managed by a hardware
>> component called the Management Complex (or MC) which provides an
>> object-base abstraction for software drivers to use the DPAA2 hardware.
>> For more details you can see: 
>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git%2Ftree%2Fdrivers%2Fstaging%2Ffsl-mc%2FREADME.txt%3Fh%3Dv4.10=01%7C01%7Cdiana.craciun%40nxp.com%7Cce2cc4d066944ce2759308d4a2f1f3f8%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0=CJAiTF6Qnq4gklSqon7xcRby0O1HQvytTUSdPwaHuSE%3D=0
>>
>> The interrupts generated by the DPAA2 hardware components are MSIs. We will 
>> add
>> support for direct assigning these DPAA2 components/objects to a virtual 
>> machine. However, this will add the need to expand the MSI usage in QEMU.
>>
>> Currently the MSIs in QEMU are pretty much tied to PCI. For ARM the
>> GIC ITS is using a device ID for interrupt translation. Currently, for
>> PCI, the requester ID is used as device ID. This will not work when
>> we add another entity that needs also a device ID which is supposed to
>> be unique across the system.
>>
>> My proposal is to add a static allocation in the virt machine. I considered
>> that this allocation is specific to each machine/platform. Currently only
>> virt machine has it, but other implementations may use the same mechanism
>> as well.
>> So, I used a static allocation with this formula:
>>
>> DeviceID = zero_extend( RequesterID[15:0] ) + 0x1 * Constant
>>
>> This formula was taken from SBSA spec (Appendix I: DeviceID generation and
>> ITS groups). In case of QEMU the constant will be different for each entity.
>> In this way a unique DeviceID will be generated and the device ID will be
>> derived from a requesterID (in case of PCI) or other means in case of other
>> entities.
>>
>> The implementation is generic as there might be in the future other non-pci 
>> devices
>> that are using MSIs or IOMMU. Any architecture can use it, though currently
>> only the ARM architecture is using the function that retrieves the stream 
>> ID. I
>> did not change all the replacements of the pci_requester_id (with 
>> pci_stream_id)
>> in the code (although if the constant is 0, the stream_id is equal with 
>> requester_id).
>> The other architectures (e.g. intel iommu code) assume that the ID is the
>> requester ID.
>>
>> Tested on NXP LS2080 platform.
>>
>> History:
> I am confused. I get it that non-PCI things want something else
> in their requester ID, but why require it for PCI devices?
> How about using Constant == 0 for PCI? This way you do
> not need to touch PCI at all as DeviceID == RequesterID ...

It is not that other devices need something else in the requester ID,
but more about finding a way to provide an unique ID across the system
(more precisely it should be unique for all devices connected to the
same IOMMU/ITS). The DT already offers support to describe the
translation between stream IDs/device IDs to requester ID for PCI
devices (iommu-map for IOMMU ([1]) and msi-map for MSIs ([2]). It will
not change the way the requester ID is used in PCI in general, but only
the places that need a unique ID (which are the MSIs and IOMMU).

If we are to use a value of 0 for the constant in case of PCI devices,
what happens if we have multiple PCI controllers?

[1]
https://www.kernel.org/doc/Documentation/devicetree/bindings/pci/pci-iommu.txt
[2]
https://www.kernel.org/doc/Documentation/devicetree/bindings/pci/pci-msi.txt

Thanks,

Diana

Re: [Qemu-devel] [PATCH v2 0/2] Add global device ID in virt machine

[Michael S. Tsirkin]

On Tue, May 23, 2017 at 02:12:43PM +0300, Diana Craciun wrote:
> The NXP DPAA2 is a hardware architecture designed for high-speeed network
> packet processing. The DPAA2 hardware components are managed by a hardware
> component called the Management Complex (or MC) which provides an
> object-base abstraction for software drivers to use the DPAA2 hardware.
> For more details you can see: 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/fsl-mc/README.txt?h=v4.10
> 
> The interrupts generated by the DPAA2 hardware components are MSIs. We will 
> add
> support for direct assigning these DPAA2 components/objects to a virtual 
> machine. However, this will add the need to expand the MSI usage in QEMU.
> 
> Currently the MSIs in QEMU are pretty much tied to PCI. For ARM the
> GIC ITS is using a device ID for interrupt translation. Currently, for
> PCI, the requester ID is used as device ID. This will not work when
> we add another entity that needs also a device ID which is supposed to
> be unique across the system.
> 
> My proposal is to add a static allocation in the virt machine. I considered
> that this allocation is specific to each machine/platform. Currently only
> virt machine has it, but other implementations may use the same mechanism
> as well.
> So, I used a static allocation with this formula:
> 
> DeviceID = zero_extend( RequesterID[15:0] ) + 0x1 * Constant
> 
> This formula was taken from SBSA spec (Appendix I: DeviceID generation and
> ITS groups). In case of QEMU the constant will be different for each entity.
> In this way a unique DeviceID will be generated and the device ID will be
> derived from a requesterID (in case of PCI) or other means in case of other
> entities.
> 
> The implementation is generic as there might be in the future other non-pci 
> devices
> that are using MSIs or IOMMU. Any architecture can use it, though currently
> only the ARM architecture is using the function that retrieves the stream ID. 
> I
> did not change all the replacements of the pci_requester_id (with 
> pci_stream_id)
> in the code (although if the constant is 0, the stream_id is equal with 
> requester_id).
> The other architectures (e.g. intel iommu code) assume that the ID is the
> requester ID.
> 
> Tested on NXP LS2080 platform.
> 
> History:

I am confused. I get it that non-PCI things want something else
in their requester ID, but why require it for PCI devices?
How about using Constant == 0 for PCI? This way you do
not need to touch PCI at all as DeviceID == RequesterID ...


> v1->v2
> --
> - the stream ID was added as a field in the pci device structure in order
> not to traverse the PCI hierarchy each time a MSI is sent.
> 
> 
> Diana Craciun (2):
>   Increased the size of requester_id field from MemTxAttrs
>   Add a unique ID in the virt machine to be used as device ID
> 
>  hw/arm/virt.c  | 26 ++
>  hw/i386/amd_iommu.c|  2 +-
>  hw/i386/intel_iommu.c  |  2 +-
>  hw/intc/arm_gicv3_its_common.c |  2 +-
>  hw/intc/arm_gicv3_its_kvm.c|  2 +-
>  hw/pci-host/gpex.c |  6 ++
>  hw/pci/msi.c   |  2 +-
>  hw/pci/pci.c   | 25 +
>  include/exec/memattrs.h|  4 ++--
>  include/hw/arm/virt.h  |  1 +
>  include/hw/intc/arm_gicv3_its_common.h |  2 +-
>  include/hw/pci-host/gpex.h |  2 ++
>  include/hw/pci/pci.h   |  8 
>  kvm-all.c  |  4 ++--
>  14 files changed, 78 insertions(+), 10 deletions(-)
> 
> -- 
> 2.5.5

[Qemu-devel] [PATCH v2 0/2] Add global device ID in virt machine

[Diana Craciun]

The NXP DPAA2 is a hardware architecture designed for high-speeed network
packet processing. The DPAA2 hardware components are managed by a hardware
component called the Management Complex (or MC) which provides an
object-base abstraction for software drivers to use the DPAA2 hardware.
For more details you can see: 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/fsl-mc/README.txt?h=v4.10

The interrupts generated by the DPAA2 hardware components are MSIs. We will add
support for direct assigning these DPAA2 components/objects to a virtual 
machine. However, this will add the need to expand the MSI usage in QEMU.

Currently the MSIs in QEMU are pretty much tied to PCI. For ARM the
GIC ITS is using a device ID for interrupt translation. Currently, for
PCI, the requester ID is used as device ID. This will not work when
we add another entity that needs also a device ID which is supposed to
be unique across the system.

My proposal is to add a static allocation in the virt machine. I considered
that this allocation is specific to each machine/platform. Currently only
virt machine has it, but other implementations may use the same mechanism
as well.
So, I used a static allocation with this formula:

DeviceID = zero_extend( RequesterID[15:0] ) + 0x1 * Constant

This formula was taken from SBSA spec (Appendix I: DeviceID generation and
ITS groups). In case of QEMU the constant will be different for each entity.
In this way a unique DeviceID will be generated and the device ID will be
derived from a requesterID (in case of PCI) or other means in case of other
entities.

The implementation is generic as there might be in the future other non-pci 
devices
that are using MSIs or IOMMU. Any architecture can use it, though currently
only the ARM architecture is using the function that retrieves the stream ID. I
did not change all the replacements of the pci_requester_id (with pci_stream_id)
in the code (although if the constant is 0, the stream_id is equal with 
requester_id).
The other architectures (e.g. intel iommu code) assume that the ID is the
requester ID.

Tested on NXP LS2080 platform.

History:

v1->v2
--
- the stream ID was added as a field in the pci device structure in order
not to traverse the PCI hierarchy each time a MSI is sent.


Diana Craciun (2):
  Increased the size of requester_id field from MemTxAttrs
  Add a unique ID in the virt machine to be used as device ID

 hw/arm/virt.c  | 26 ++
 hw/i386/amd_iommu.c|  2 +-
 hw/i386/intel_iommu.c  |  2 +-
 hw/intc/arm_gicv3_its_common.c |  2 +-
 hw/intc/arm_gicv3_its_kvm.c|  2 +-
 hw/pci-host/gpex.c |  6 ++
 hw/pci/msi.c   |  2 +-
 hw/pci/pci.c   | 25 +
 include/exec/memattrs.h|  4 ++--
 include/hw/arm/virt.h  |  1 +
 include/hw/intc/arm_gicv3_its_common.h |  2 +-
 include/hw/pci-host/gpex.h |  2 ++
 include/hw/pci/pci.h   |  8 
 kvm-all.c  |  4 ++--
 14 files changed, 78 insertions(+), 10 deletions(-)

-- 
2.5.5