RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-12 Thread Tian, Kevin
> From: Jason Gunthorpe > Sent: Monday, December 13, 2021 7:37 AM > > On Sun, Dec 12, 2021 at 09:55:32PM +0100, Thomas Gleixner wrote: > > Kevin, > > > > On Sun, Dec 12 2021 at 01:56, Kevin Tian wrote: > > >> From: Thomas Gleixner > > >> All I can find is drivers/iommu/virtio-iommu.c but I

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-12 Thread Jason Gunthorpe via iommu
On Sun, Dec 12, 2021 at 01:12:05AM +0100, Thomas Gleixner wrote: > PCI/MSI and PCI/MSI-X are just implementations of IMS > > Not more, not less. The fact that they have very strict rules about the > storage space and the fact that they are mutually exclusive does not > change that at all.

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-12 Thread Jason Gunthorpe via iommu
On Sun, Dec 12, 2021 at 09:55:32PM +0100, Thomas Gleixner wrote: > Kevin, > > On Sun, Dec 12 2021 at 01:56, Kevin Tian wrote: > >> From: Thomas Gleixner > >> All I can find is drivers/iommu/virtio-iommu.c but I can't find anything > >> vIR related there. > > > > Well, virtio-iommu is a

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-12 Thread Jason Gunthorpe via iommu
On Sun, Dec 12, 2021 at 08:44:46AM +0200, Mika Penttilä wrote: > > /* > > * The MSIX mappable capability informs that MSIX data of a BAR can be > > mmapped > > * which allows direct access to non-MSIX registers which happened to be > > within > > * the same system page. > > * > > *

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-12 Thread Thomas Gleixner
Kevin, On Sun, Dec 12 2021 at 01:56, Kevin Tian wrote: >> From: Thomas Gleixner >> All I can find is drivers/iommu/virtio-iommu.c but I can't find anything >> vIR related there. > > Well, virtio-iommu is a para-virtualized vIOMMU implementations. > > In reality there are also fully emulated

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-12 Thread Thomas Gleixner
Kevin, On Sun, Dec 12 2021 at 02:14, Kevin Tian wrote: >> From: Thomas Gleixner > I just continue the thought practice along that direction to see what > the host flow will be like (step by step). Looking at the current > implementation is just one necessary step in my thought practice to >

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-12 Thread Mika Penttilä
On 10.12.2021 9.36, Tian, Kevin wrote: From: Jason Gunthorpe Sent: Friday, December 10, 2021 4:59 AM On Thu, Dec 09, 2021 at 09:32:42PM +0100, Thomas Gleixner wrote: On Thu, Dec 09 2021 at 12:21, Jason Gunthorpe wrote: On Thu, Dec 09, 2021 at 09:37:06AM +0100, Thomas Gleixner wrote: If we

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-11 Thread Tian, Kevin
Hi, Thomas, > From: Thomas Gleixner > Sent: Sunday, December 12, 2021 8:12 AM > > Kevin, > > On Sat, Dec 11 2021 at 07:52, Kevin Tian wrote: > >> From: Jason Gunthorpe > >> > Then Qemu needs to find out the GSI number for the vIRTE handle. > >> > Again Qemu doesn't have such information since

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-11 Thread Tian, Kevin
> From: Thomas Gleixner > Sent: Saturday, December 11, 2021 9:05 PM > > Kevin, > > On Sat, Dec 11 2021 at 07:44, Kevin Tian wrote: > >> From: Thomas Gleixner > >> On Fri, Dec 10 2021 at 08:39, Jason Gunthorpe wrote: > >> > It is clever, we don't have an vIOMMU that supplies vIR today, so by >

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-11 Thread Thomas Gleixner
Kevin, On Sat, Dec 11 2021 at 07:52, Kevin Tian wrote: >> From: Jason Gunthorpe >> > Then Qemu needs to find out the GSI number for the vIRTE handle. >> > Again Qemu doesn't have such information since it doesn't know >> > which MSI[-X] entry points to this handle due to no trap. >> >> No this

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-11 Thread Thomas Gleixner
Kevin, On Sat, Dec 11 2021 at 07:44, Kevin Tian wrote: >> From: Thomas Gleixner >> On Fri, Dec 10 2021 at 08:39, Jason Gunthorpe wrote: >> > It is clever, we don't have an vIOMMU that supplies vIR today, so by >> > definition all guests are excluded and only bare metal works. >> >> Dammit. Now

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-11 Thread Tian, Kevin
> From: Thomas Gleixner > Sent: Friday, December 10, 2021 8:13 PM > > >> 5) It's not possible for the kernel to reliably detect whether it is > >> running on bare metal or not. Yes we talked about heuristics, but > >> that's something I really want to avoid. > > > > How would the

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-10 Thread Tian, Kevin
> From: Jason Gunthorpe > Sent: Friday, December 10, 2021 8:40 PM > > > > Then Qemu needs to find out the GSI number for the vIRTE handle. > > Again Qemu doesn't have such information since it doesn't know > > which MSI[-X] entry points to this handle due to no trap. > > No this is already

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-10 Thread Tian, Kevin
> From: Thomas Gleixner > Sent: Saturday, December 11, 2021 3:00 AM > > Jason, > > On Fri, Dec 10 2021 at 08:39, Jason Gunthorpe wrote: > > > On Fri, Dec 10, 2021 at 07:29:01AM +, Tian, Kevin wrote: > >> > 5) It's not possible for the kernel to reliably detect whether it is > >> >

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-10 Thread Thomas Gleixner
Jason, On Fri, Dec 10 2021 at 08:39, Jason Gunthorpe wrote: > On Fri, Dec 10, 2021 at 07:29:01AM +, Tian, Kevin wrote: >> > 5) It's not possible for the kernel to reliably detect whether it is >> > running on bare metal or not. Yes we talked about heuristics, but >> > that's

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-10 Thread Jason Gunthorpe via iommu
On Fri, Dec 10, 2021 at 07:29:01AM +, Tian, Kevin wrote: > > 5) It's not possible for the kernel to reliably detect whether it is > > running on bare metal or not. Yes we talked about heuristics, but > > that's something I really want to avoid. > > How would the hypercall

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-10 Thread Jason Gunthorpe via iommu
On Fri, Dec 10, 2021 at 07:36:12AM +, Tian, Kevin wrote: > /* > * The MSIX mappable capability informs that MSIX data of a BAR can be mmapped > * which allows direct access to non-MSIX registers which happened to be > within > * the same system page. > * > * Even though the userspace

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-10 Thread Thomas Gleixner
Kevin, On Fri, Dec 10 2021 at 07:29, Kevin Tian wrote: >> From: Thomas Gleixner >> 4) For the guest side we agreed that we need an hypercall because the >> host can't trap the write to the MSI[-X] entry anymore. > > To be accurate I'd like to not call it "can't trap". The host still traps

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-09 Thread Tian, Kevin
> From: Thomas Gleixner > Sent: Thursday, December 9, 2021 11:58 PM > > On Thu, Dec 09 2021 at 12:17, Kevin Tian wrote: > >> From: Thomas Gleixner > >> I think you are looking at that from the internal implementation details > >> of IDXD. But you can just model it in an IDXD implementation

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-09 Thread Tian, Kevin
> From: Jason Gunthorpe > Sent: Friday, December 10, 2021 4:59 AM > > On Thu, Dec 09, 2021 at 09:32:42PM +0100, Thomas Gleixner wrote: > > On Thu, Dec 09 2021 at 12:21, Jason Gunthorpe wrote: > > > On Thu, Dec 09, 2021 at 09:37:06AM +0100, Thomas Gleixner wrote: > > > If we keep the MSI

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-09 Thread Tian, Kevin
> From: Thomas Gleixner > Sent: Friday, December 10, 2021 8:26 AM > > On Thu, Dec 09 2021 at 23:09, Thomas Gleixner wrote: > > On Thu, Dec 09 2021 at 16:58, Jason Gunthorpe wrote: > >> Okay, I think I get it. Would be nice to have someone from intel > >> familiar with the vIOMMU protocols and

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-09 Thread Thomas Gleixner
On Thu, Dec 09 2021 at 23:09, Thomas Gleixner wrote: > On Thu, Dec 09 2021 at 16:58, Jason Gunthorpe wrote: >> Okay, I think I get it. Would be nice to have someone from intel >> familiar with the vIOMMU protocols and qemu code remark what the >> hypervisor side can look like. >> >> There is a bit

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-09 Thread Thomas Gleixner
On Thu, Dec 09 2021 at 16:58, Jason Gunthorpe wrote: > On Thu, Dec 09, 2021 at 09:32:42PM +0100, Thomas Gleixner wrote: >> That was my thought to avoid having different mechanisms. >> >> The address/data pair is computed in two places: >> >> 1) Activation of an interrupt >> 2) Affinity

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-09 Thread Jason Gunthorpe via iommu
On Thu, Dec 09, 2021 at 09:32:42PM +0100, Thomas Gleixner wrote: > On Thu, Dec 09 2021 at 12:21, Jason Gunthorpe wrote: > > On Thu, Dec 09, 2021 at 09:37:06AM +0100, Thomas Gleixner wrote: > > If we keep the MSI emulation in the hypervisor then MSI != IMS. The > > MSI code needs to write a

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-09 Thread Thomas Gleixner
On Thu, Dec 09 2021 at 12:21, Jason Gunthorpe wrote: > On Thu, Dec 09, 2021 at 09:37:06AM +0100, Thomas Gleixner wrote: > If we keep the MSI emulation in the hypervisor then MSI != IMS. The > MSI code needs to write a addr/data pair compatible with the emulation > and the IMS code needs to write

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-09 Thread Jason Gunthorpe via iommu
On Thu, Dec 09, 2021 at 09:37:06AM +0100, Thomas Gleixner wrote: > On Thu, Dec 09 2021 at 05:23, Kevin Tian wrote: > >> From: Thomas Gleixner > >> I don't see anything wrong with that. A subdevice is it's own entity and > >> VFIO can chose the most conveniant representation of it to the guest >

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-09 Thread Thomas Gleixner
On Thu, Dec 09 2021 at 12:17, Kevin Tian wrote: >> From: Thomas Gleixner >> I think you are looking at that from the internal implementation details >> of IDXD. But you can just model it in an IDXD implementation agnostic >> way: >> >> ENQCMD(PASID, IMS-ENTRY,.) > > Not exactly

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-09 Thread Tian, Kevin
> From: Thomas Gleixner > Sent: Thursday, December 9, 2021 4:37 PM > > On Thu, Dec 09 2021 at 05:23, Kevin Tian wrote: > >> From: Thomas Gleixner > >> I don't see anything wrong with that. A subdevice is it's own entity and > >> VFIO can chose the most conveniant representation of it to the

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-09 Thread Tian, Kevin
> From: Thomas Gleixner > Sent: Thursday, December 9, 2021 5:03 PM > > Kevin, > > On Thu, Dec 09 2021 at 06:26, Kevin Tian wrote: > >> From: Jason Gunthorpe > >> I don't know of any use case for more than one interrupt on a queue, > >> and if it did come up I'd probably approach it by making

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-09 Thread Thomas Gleixner
Kevin, On Thu, Dec 09 2021 at 06:26, Kevin Tian wrote: >> From: Jason Gunthorpe >> I don't know of any use case for more than one interrupt on a queue, >> and if it did come up I'd probably approach it by making the queue >> handle above also specify the 'queue relative HW index' > > We have

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-09 Thread Thomas Gleixner
On Thu, Dec 09 2021 at 05:23, Kevin Tian wrote: >> From: Thomas Gleixner >> I don't see anything wrong with that. A subdevice is it's own entity and >> VFIO can chose the most conveniant representation of it to the guest >> obviously. >> >> How that is backed on the host does not really matter.

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-08 Thread Tian, Kevin
> From: Jason Gunthorpe > Sent: Saturday, December 4, 2021 12:41 AM > > > Or has each queue and controlblock and whatever access to a shared large > > array where the messages are stored and the indices are handed out to > > the queues and controlblocks? > > > If each of them have their own

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-08 Thread Jason Wang
On Thu, Dec 9, 2021 at 1:41 PM Tian, Kevin wrote: > > > From: Jason Gunthorpe > > Sent: Thursday, December 2, 2021 9:55 PM > > > > Further, there is no reason why IMS should be reserved exclusively for > > VFIO! > > This is correct. Just as what you agreed with Thomas, the only difference >

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-08 Thread Tian, Kevin
> From: Jason Gunthorpe > Sent: Thursday, December 2, 2021 9:55 PM > > Further, there is no reason why IMS should be reserved exclusively for > VFIO! This is correct. Just as what you agreed with Thomas, the only difference between IMS and MSI is on where the messages are stored. Physically

RE: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-08 Thread Tian, Kevin
> From: Thomas Gleixner > Sent: Thursday, December 2, 2021 5:45 AM > > On Wed, Dec 01 2021 at 14:21, Dave Jiang wrote: > > On 12/1/2021 1:25 PM, Thomas Gleixner wrote: > >>> The hardware implementation does not have enough MSIX vectors for > >>> guests. There are only 9 MSIX vectors total (8 for

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-06 Thread Thomas Gleixner
On Mon, Dec 06 2021 at 17:06, Jason Gunthorpe wrote: > On Mon, Dec 06, 2021 at 09:28:47PM +0100, Thomas Gleixner wrote: >> I wish I could mask underneath for some stuff on x86. Though that would >> not help with the worst problem vs. affinity settings. See the horrible >> dance in: > > My thinking

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-06 Thread Jason Gunthorpe via iommu
On Mon, Dec 06, 2021 at 09:28:47PM +0100, Thomas Gleixner wrote: > That's already the plan in some form, but there's a long way towards > that. See below. Okay, then I think we are thinking the same sorts of things, it is good to see > Also there will be a question of how many different

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-06 Thread Thomas Gleixner
Jason, On Mon, Dec 06 2021 at 13:00, Jason Gunthorpe wrote: > On Mon, Dec 06, 2021 at 04:47:58PM +0100, Thomas Gleixner wrote: >> It will need some more than that, e.g. mask/unmask and as we discussed >> quite some time ago something like the irq_buslock/unlock pair, so you >> can handle updates

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-06 Thread Jason Gunthorpe via iommu
On Mon, Dec 06, 2021 at 04:47:58PM +0100, Thomas Gleixner wrote: > >>- The irqchip callbacks which can be implemented by these top > >> level domains are going to be restricted. > > > > OK - I think it is great that the driver will see a special ops struct > > that is 'ops for

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-06 Thread Thomas Gleixner
Jason, On Mon, Dec 06 2021 at 10:43, Jason Gunthorpe wrote: > On Sun, Dec 05, 2021 at 03:16:40PM +0100, Thomas Gleixner wrote: >> > That's not really a good idea because dev->irqdomain is a generic >> > mechanism and not restricted to the PCI use case. Special casing it for >> > PCI is just

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-06 Thread Thomas Gleixner
Jason, On Mon, Dec 06 2021 at 10:19, Jason Gunthorpe wrote: > On Sat, Dec 04, 2021 at 03:20:36PM +0100, Thomas Gleixner wrote: >> even try to make the irqchip/domain code mangled into the other device >> code. It should create the irqdomain with the associated chip and that >> creation process

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-06 Thread Jason Gunthorpe via iommu
On Sun, Dec 05, 2021 at 03:16:40PM +0100, Thomas Gleixner wrote: > On Sat, Dec 04 2021 at 15:20, Thomas Gleixner wrote: > > On Fri, Dec 03 2021 at 12:41, Jason Gunthorpe wrote: > > So I need to break that up in a way which caters for both cases, but > > does neither create a special case for PCI

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-06 Thread Jason Gunthorpe via iommu
On Sat, Dec 04, 2021 at 03:20:36PM +0100, Thomas Gleixner wrote: > Jason, > > On Fri, Dec 03 2021 at 12:41, Jason Gunthorpe wrote: > > On Fri, Dec 03, 2021 at 04:07:58PM +0100, Thomas Gleixner wrote: > > Lets do a thought experiment, lets say we forget about the current PCI > > MSI API. I've

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-05 Thread Thomas Gleixner
On Sat, Dec 04 2021 at 15:20, Thomas Gleixner wrote: > On Fri, Dec 03 2021 at 12:41, Jason Gunthorpe wrote: > So I need to break that up in a way which caters for both cases, but > does neither create a special case for PCI nor for the rest of the > universe, i.e. the 1:1 case has to be a subset

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-04 Thread Thomas Gleixner
Jason, On Fri, Dec 03 2021 at 12:41, Jason Gunthorpe wrote: > On Fri, Dec 03, 2021 at 04:07:58PM +0100, Thomas Gleixner wrote: > Lets do a thought experiment, lets say we forget about the current PCI > MSI API. > > What if it worked more like this: > > probe() > // Access the real PCI SIG

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-03 Thread Jason Gunthorpe via iommu
On Fri, Dec 03, 2021 at 04:07:58PM +0100, Thomas Gleixner wrote: > Jason, > > On Thu, Dec 02 2021 at 20:37, Jason Gunthorpe wrote: > > On Thu, Dec 02, 2021 at 11:31:11PM +0100, Thomas Gleixner wrote: > >> >> Of course we can store them in pci_dev.dev.msi.data.store. Either with a > >> >>

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-03 Thread Thomas Gleixner
Jason, On Thu, Dec 02 2021 at 20:37, Jason Gunthorpe wrote: > On Thu, Dec 02, 2021 at 11:31:11PM +0100, Thomas Gleixner wrote: >> >> Of course we can store them in pci_dev.dev.msi.data.store. Either with a >> >> dedicated xarray or by partitioning the xarray space. Both have their >> >> pro and

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-02 Thread Jason Gunthorpe via iommu
On Thu, Dec 02, 2021 at 11:31:11PM +0100, Thomas Gleixner wrote: > The software representation aka struct msi_desc is a different > story. That's what we are debating. Okay, I did mean msi_desc storage, so we are talking about the same thigns > >> Of course we can store them in

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-02 Thread Thomas Gleixner
Jason, On Thu, Dec 02 2021 at 16:00, Jason Gunthorpe wrote: > On Thu, Dec 02, 2021 at 08:25:48PM +0100, Thomas Gleixner wrote: >> We seem to have a serious problem of terminology and the understanding >> of topology which is why we continue to talk past each other forever. > > I think I

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-02 Thread Jason Gunthorpe via iommu
On Thu, Dec 02, 2021 at 08:25:48PM +0100, Thomas Gleixner wrote: > Jason, > > On Thu, Dec 02 2021 at 09:55, Jason Gunthorpe wrote: > > On Thu, Dec 02, 2021 at 01:01:42AM +0100, Thomas Gleixner wrote: > >> On Wed, Dec 01 2021 at 21:21, Thomas Gleixner wrote: > >> > On Wed, Dec 01 2021 at 14:14,

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-02 Thread Thomas Gleixner
Jason, On Thu, Dec 02 2021 at 09:55, Jason Gunthorpe wrote: > On Thu, Dec 02, 2021 at 01:01:42AM +0100, Thomas Gleixner wrote: >> On Wed, Dec 01 2021 at 21:21, Thomas Gleixner wrote: >> > On Wed, Dec 01 2021 at 14:14, Jason Gunthorpe wrote: >> > Which in turn is consistent all over the place and

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-02 Thread Jason Gunthorpe via iommu
On Thu, Dec 02, 2021 at 03:23:38PM +0100, Greg Kroah-Hartman wrote: > On Thu, Dec 02, 2021 at 09:55:02AM -0400, Jason Gunthorpe wrote: > > Further, there is no reason why IMS should be reserved exclusively for > > VFIO! Why shouldn't the cdev be able to use IMS vectors too? It is > > just a

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-02 Thread Greg Kroah-Hartman
On Thu, Dec 02, 2021 at 09:55:02AM -0400, Jason Gunthorpe wrote: > Further, there is no reason why IMS should be reserved exclusively for > VFIO! Why shouldn't the cdev be able to use IMS vectors too? It is > just a feature of the PCI device like MSI. If the queue has a PASID it > can use IDXD's

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-02 Thread Jason Gunthorpe via iommu
On Thu, Dec 02, 2021 at 01:01:42AM +0100, Thomas Gleixner wrote: > Jason, > > On Wed, Dec 01 2021 at 21:21, Thomas Gleixner wrote: > > On Wed, Dec 01 2021 at 14:14, Jason Gunthorpe wrote: > > Which in turn is consistent all over the place and does not require any > > special case for anything.

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-01 Thread Thomas Gleixner
Jason, On Wed, Dec 01 2021 at 21:21, Thomas Gleixner wrote: > On Wed, Dec 01 2021 at 14:14, Jason Gunthorpe wrote: > Which in turn is consistent all over the place and does not require any > special case for anything. Neither for interrupts nor for anything else. that said, feel free to tell me

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-01 Thread Thomas Gleixner
dave, On Wed, Dec 01 2021 at 15:53, Dave Jiang wrote: > On 12/1/2021 3:03 PM, Thomas Gleixner wrote: >> This still depends on how this overall discussion about representation >> of all of this stuff is resolved. >> What needs a subdevice to expose? >> Can you answer that too please? > >

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-01 Thread Dave Jiang
On 12/1/2021 3:03 PM, Thomas Gleixner wrote: On Wed, Dec 01 2021 at 14:49, Dave Jiang wrote: On 12/1/2021 2:44 PM, Thomas Gleixner wrote: How that is backed on the host does not really matter. You can expose MSI-X to the guest with a INTx backing as well. I'm still failing to see the

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-01 Thread Thomas Gleixner
On Wed, Dec 01 2021 at 14:49, Dave Jiang wrote: > On 12/1/2021 2:44 PM, Thomas Gleixner wrote: >> How that is backed on the host does not really matter. You can expose >> MSI-X to the guest with a INTx backing as well. >> >> I'm still failing to see the connection between the 9 MSIX vectors and >>

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-01 Thread Dave Jiang
On 12/1/2021 2:44 PM, Thomas Gleixner wrote: On Wed, Dec 01 2021 at 14:21, Dave Jiang wrote: On 12/1/2021 1:25 PM, Thomas Gleixner wrote: The hardware implementation does not have enough MSIX vectors for guests. There are only 9 MSIX vectors total (8 for queues) and 2048 IMS vectors. So if

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-01 Thread Thomas Gleixner
On Wed, Dec 01 2021 at 14:21, Dave Jiang wrote: > On 12/1/2021 1:25 PM, Thomas Gleixner wrote: >>> The hardware implementation does not have enough MSIX vectors for >>> guests. There are only 9 MSIX vectors total (8 for queues) and 2048 IMS >>> vectors. So if we are to do MSI-X for all of them,

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-01 Thread Dave Jiang
On 12/1/2021 1:25 PM, Thomas Gleixner wrote: On Wed, Dec 01 2021 at 11:47, Dave Jiang wrote: On 12/1/2021 11:41 AM, Thomas Gleixner wrote: Hi Thomas. This is actually the IDXD usage for a mediated device passed to a guest kernel when we plumb the pass through of IMS to the guest rather than

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-01 Thread Thomas Gleixner
On Wed, Dec 01 2021 at 11:47, Dave Jiang wrote: > On 12/1/2021 11:41 AM, Thomas Gleixner wrote: >>> Hi Thomas. This is actually the IDXD usage for a mediated device passed >>> to a guest kernel when we plumb the pass through of IMS to the guest >>> rather than doing previous implementation of

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-01 Thread Thomas Gleixner
Jason, On Wed, Dec 01 2021 at 14:14, Jason Gunthorpe wrote: > On Wed, Dec 01, 2021 at 06:35:35PM +0100, Thomas Gleixner wrote: >> On Wed, Dec 01 2021 at 09:00, Jason Gunthorpe wrote: >> But NTB is operating through an abstraction layer and is not a direct >> PCIe device driver. > > I'm not sure

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-01 Thread Dave Jiang
On 12/1/2021 11:41 AM, Thomas Gleixner wrote: Dave, please trim your replies. On Wed, Dec 01 2021 at 09:28, Dave Jiang wrote: On 12/1/2021 3:16 AM, Thomas Gleixner wrote: Jason, CC+ IOMMU folks On Tue, Nov 30 2021 at 20:17, Jason Gunthorpe wrote: On Tue, Nov 30, 2021 at 10:23:16PM

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-01 Thread Logan Gunthorpe
On 2021-12-01 11:14 a.m., 'Jason Gunthorpe' via linux-ntb wrote: > On Wed, Dec 01, 2021 at 06:35:35PM +0100, Thomas Gleixner wrote: >> On Wed, Dec 01 2021 at 09:00, Jason Gunthorpe wrote: >>> On Wed, Dec 01, 2021 at 11:16:47AM +0100, Thomas Gleixner wrote: Looking at the device slices as

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-01 Thread Thomas Gleixner
Dave, please trim your replies. On Wed, Dec 01 2021 at 09:28, Dave Jiang wrote: > On 12/1/2021 3:16 AM, Thomas Gleixner wrote: >> Jason, >> >> CC+ IOMMU folks >> >> On Tue, Nov 30 2021 at 20:17, Jason Gunthorpe wrote: >>> On Tue, Nov 30, 2021 at 10:23:16PM +0100, Thomas Gleixner wrote: >> >>

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-01 Thread Jason Gunthorpe via iommu
On Wed, Dec 01, 2021 at 06:35:35PM +0100, Thomas Gleixner wrote: > On Wed, Dec 01 2021 at 09:00, Jason Gunthorpe wrote: > > On Wed, Dec 01, 2021 at 11:16:47AM +0100, Thomas Gleixner wrote: > >> Looking at the device slices as subdevices with their own struct device > >> makes a lot of sense from

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-01 Thread Thomas Gleixner
On Wed, Dec 01 2021 at 09:00, Jason Gunthorpe wrote: > On Wed, Dec 01, 2021 at 11:16:47AM +0100, Thomas Gleixner wrote: >> Looking at the device slices as subdevices with their own struct device >> makes a lot of sense from the conceptual level. > > Except IMS is not just for subdevices, it should

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-01 Thread Dave Jiang
On 12/1/2021 3:16 AM, Thomas Gleixner wrote: Jason, CC+ IOMMU folks On Tue, Nov 30 2021 at 20:17, Jason Gunthorpe wrote: On Tue, Nov 30, 2021 at 10:23:16PM +0100, Thomas Gleixner wrote: The real problem is where to store the MSI descriptors because the PCI device has its own real PCI/MSI-X

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-01 Thread Jason Gunthorpe via iommu
On Wed, Dec 01, 2021 at 11:16:47AM +0100, Thomas Gleixner wrote: > Looking at the device slices as subdevices with their own struct device > makes a lot of sense from the conceptual level. Except IMS is not just for subdevices, it should be usable for any driver in any case as a general

Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()

2021-12-01 Thread Thomas Gleixner
Jason, CC+ IOMMU folks On Tue, Nov 30 2021 at 20:17, Jason Gunthorpe wrote: > On Tue, Nov 30, 2021 at 10:23:16PM +0100, Thomas Gleixner wrote: >> The real problem is where to store the MSI descriptors because the PCI >> device has its own real PCI/MSI-X interrupts which means it still shares >>