Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Thomas Gleixner]

On Thu, 5 Nov 2015, Marc Zyngier wrote:
> On 04/11/15 13:34, Thomas Gleixner wrote:
> > Marc,
> > 
> > On Fri, 16 Oct 2015, Marc Zyngier wrote:
> >> On 16/10/15 02:55, Jiang Liu wrote:
> >>>   There's a working to enable Intel VMD storage device, which
> >>> has the similar requirement. Basically a PCIe hierarchy is hidden
> >>> behind a parent PCIe device, so we need to use the PCIe irqs on parent
> >>> to de-multiple PCIe IRQs from hidden PCIe devices. Seems a chance for
> >>> consolidation here.
> >>
> >> Do you know if there is a 1-1 mapping between the interrupts seen by the
> >> parent device and those seen by the hidden devices? Or is it a case of
> >> having to demultiplex the MSIs? Looks like the former, but I'd like to
> >> be sure.
> > 
> > Yes, it's a demultiplexer. No 1:1 mapping.
> 
> Right. This doesn't exactly fit the scheme I have so far (there is a 1:1
> mapping between the wired interrupt and the MSI), but once we are able
> to expose an MSI domain, it could be possible to construct the MSI
> demultiplexer on top. That's a lot of layers! ;-)

Well for the demux case it doesn't make a lot of sense. It's not easy
to describe in a hierarchy. Having that parentless MSI domain for that
VMD case is simple enough.

Thanks,

tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Marc Zyngier]

On 04/11/15 13:34, Thomas Gleixner wrote:
> Marc,
> 
> On Fri, 16 Oct 2015, Marc Zyngier wrote:
>> On 16/10/15 02:55, Jiang Liu wrote:
>>> There's a working to enable Intel VMD storage device, which
>>> has the similar requirement. Basically a PCIe hierarchy is hidden
>>> behind a parent PCIe device, so we need to use the PCIe irqs on parent
>>> to de-multiple PCIe IRQs from hidden PCIe devices. Seems a chance for
>>> consolidation here.
>>
>> Do you know if there is a 1-1 mapping between the interrupts seen by the
>> parent device and those seen by the hidden devices? Or is it a case of
>> having to demultiplex the MSIs? Looks like the former, but I'd like to
>> be sure.
> 
> Yes, it's a demultiplexer. No 1:1 mapping.

Right. This doesn't exactly fit the scheme I have so far (there is a 1:1
mapping between the wired interrupt and the MSI), but once we are able
to expose an MSI domain, it could be possible to construct the MSI
demultiplexer on top. That's a lot of layers! ;-)

>> Sure, will do when I repost this (probably in a few weeks), and assuming
>> this fits the bill for Thomas and the MBIGEN folks.
> 
> It doesn't look that bad and the resulting mbigen stuff is way less
> horrible than it was before. So I agree this is a possible solution to
> the problem.

Cool. I'll revisit that after the merge window.

Thanks,

M.
-- 
Jazz is not dead. It just smells funny...
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Thomas Gleixner]

On Thu, 5 Nov 2015, Marc Zyngier wrote:
> On 04/11/15 13:34, Thomas Gleixner wrote:
> > Marc,
> > 
> > On Fri, 16 Oct 2015, Marc Zyngier wrote:
> >> On 16/10/15 02:55, Jiang Liu wrote:
> >>>   There's a working to enable Intel VMD storage device, which
> >>> has the similar requirement. Basically a PCIe hierarchy is hidden
> >>> behind a parent PCIe device, so we need to use the PCIe irqs on parent
> >>> to de-multiple PCIe IRQs from hidden PCIe devices. Seems a chance for
> >>> consolidation here.
> >>
> >> Do you know if there is a 1-1 mapping between the interrupts seen by the
> >> parent device and those seen by the hidden devices? Or is it a case of
> >> having to demultiplex the MSIs? Looks like the former, but I'd like to
> >> be sure.
> > 
> > Yes, it's a demultiplexer. No 1:1 mapping.
> 
> Right. This doesn't exactly fit the scheme I have so far (there is a 1:1
> mapping between the wired interrupt and the MSI), but once we are able
> to expose an MSI domain, it could be possible to construct the MSI
> demultiplexer on top. That's a lot of layers! ;-)

Well for the demux case it doesn't make a lot of sense. It's not easy
to describe in a hierarchy. Having that parentless MSI domain for that
VMD case is simple enough.

Thanks,

tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Marc Zyngier]

On 04/11/15 13:34, Thomas Gleixner wrote:
> Marc,
> 
> On Fri, 16 Oct 2015, Marc Zyngier wrote:
>> On 16/10/15 02:55, Jiang Liu wrote:
>>> There's a working to enable Intel VMD storage device, which
>>> has the similar requirement. Basically a PCIe hierarchy is hidden
>>> behind a parent PCIe device, so we need to use the PCIe irqs on parent
>>> to de-multiple PCIe IRQs from hidden PCIe devices. Seems a chance for
>>> consolidation here.
>>
>> Do you know if there is a 1-1 mapping between the interrupts seen by the
>> parent device and those seen by the hidden devices? Or is it a case of
>> having to demultiplex the MSIs? Looks like the former, but I'd like to
>> be sure.
> 
> Yes, it's a demultiplexer. No 1:1 mapping.

Right. This doesn't exactly fit the scheme I have so far (there is a 1:1
mapping between the wired interrupt and the MSI), but once we are able
to expose an MSI domain, it could be possible to construct the MSI
demultiplexer on top. That's a lot of layers! ;-)

>> Sure, will do when I repost this (probably in a few weeks), and assuming
>> this fits the bill for Thomas and the MBIGEN folks.
> 
> It doesn't look that bad and the resulting mbigen stuff is way less
> horrible than it was before. So I agree this is a possible solution to
> the problem.

Cool. I'll revisit that after the merge window.

Thanks,

M.
-- 
Jazz is not dead. It just smells funny...
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Thomas Gleixner]

Marc,

On Fri, 16 Oct 2015, Marc Zyngier wrote:
> On 16/10/15 02:55, Jiang Liu wrote:
> > There's a working to enable Intel VMD storage device, which
> > has the similar requirement. Basically a PCIe hierarchy is hidden
> > behind a parent PCIe device, so we need to use the PCIe irqs on parent
> > to de-multiple PCIe IRQs from hidden PCIe devices. Seems a chance for
> > consolidation here.
> 
> Do you know if there is a 1-1 mapping between the interrupts seen by the
> parent device and those seen by the hidden devices? Or is it a case of
> having to demultiplex the MSIs? Looks like the former, but I'd like to
> be sure.

Yes, it's a demultiplexer. No 1:1 mapping.

> Sure, will do when I repost this (probably in a few weeks), and assuming
> this fits the bill for Thomas and the MBIGEN folks.

It doesn't look that bad and the resulting mbigen stuff is way less
horrible than it was before. So I agree this is a possible solution to
the problem.

Thanks,

tglx

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Thomas Gleixner]

Marc,

On Fri, 16 Oct 2015, Marc Zyngier wrote:
> On 16/10/15 02:55, Jiang Liu wrote:
> > There's a working to enable Intel VMD storage device, which
> > has the similar requirement. Basically a PCIe hierarchy is hidden
> > behind a parent PCIe device, so we need to use the PCIe irqs on parent
> > to de-multiple PCIe IRQs from hidden PCIe devices. Seems a chance for
> > consolidation here.
> 
> Do you know if there is a 1-1 mapping between the interrupts seen by the
> parent device and those seen by the hidden devices? Or is it a case of
> having to demultiplex the MSIs? Looks like the former, but I'd like to
> be sure.

Yes, it's a demultiplexer. No 1:1 mapping.

> Sure, will do when I repost this (probably in a few weeks), and assuming
> this fits the bill for Thomas and the MBIGEN folks.

It doesn't look that bad and the resulting mbigen stuff is way less
horrible than it was before. So I agree this is a possible solution to
the problem.

Thanks,

tglx

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Marc Zyngier]

Hi Gerry,

On 16/10/15 02:55, Jiang Liu wrote:
> On 2015/10/15 23:39, Marc Zyngier wrote:
>> There seems to be a new class of interrupt controller out there whose
>> sole purpose (apart from making everybody's life a nightmare) is to
>> turn wired interrupts into MSIs.
>>
>> Instead of considering that the MSIs allocated to a device are for the
>> direct use of that device, we can turn this set of MSIs into a irq
>> domain, and use that domain to build a standard irqchip on top of
>> that.
> Hi Marc,
>   There's a working to enable Intel VMD storage device, which
> has the similar requirement. Basically a PCIe hierarchy is hidden
> behind a parent PCIe device, so we need to use the PCIe irqs on parent
> to de-multiple PCIe IRQs from hidden PCIe devices. Seems a chance for
> consolidation here.

Do you know if there is a 1-1 mapping between the interrupts seen by the
parent device and those seen by the hidden devices? Or is it a case of
having to demultiplex the MSIs? Looks like the former, but I'd like to
be sure.

I just had a quick look at the code there:
https://lkml.org/lkml/2015/8/27/674

Is there anything more recent?

>   cc Keith Busch  who is the author of
> VMD device driver.

Sure, will do when I repost this (probably in a few weeks), and assuming
this fits the bill for Thomas and the MBIGEN folks.

Thanks,

M.
-- 
Jazz is not dead. It just smells funny...
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Arnd Bergmann]

On Friday 16 October 2015 09:03:14 Marc Zyngier wrote:
> On 15/10/15 20:16, Arnd Bergmann wrote:
> > On Thursday 15 October 2015 17:01:02 Marc Zyngier wrote:
> >>
> >> "Preconfigured" is the key word. While you can do something like that if
> >> your hardware treats MSIs just as if they were wired interrupts
> >> (something like GICv2m), it becomes far more hairy if the target of MSIs
> >> is something like a GICv3 ITS (which is the case for HiSilicon mbigen).
> >>
> >> The main reason is that the ITS relies on "translation tables" kept in
> >> memory, which the OS has to configure, and handing over pre-configured
> >> tables is not something I'm looking forward to doing. From a CPU point
> >> of view, this is akin entering the kernel with the MMU already on and no
> >> idmap...
> >>
> >> The approach taken here is to make the MSI-ness explicit at the irqchip
> >> level, and keep the interrupting device oblivious of that feature. Also,
> >> this relies on the fact that we can have one MSI per wire, meaning that
> >> we don't have to multiplex anything (no nested irqchip), and that we can
> >> rely on hierarchical domains, which simplifies the code (at least for
> >> the irqchip).
> >>
> > 
> > Thanks, that already makes things much clearer. Just one more question:
> > why can't those translation tables be configured statically by the
> > irqchip driver? Is this all about being able to cut a few cycles
> > in case of virtualization? 
> 
> Having a static configuration, while doable, complicates things for
> everybody else. The LPI number used by the irqchip would need to be put
> an some "exclusion list" to make sure it is not reallocated for other
> subsystems (e.g PCI). The translation tables also define the target CPU,
> which could cause interesting problems once combined with CPU hotplug if
> the ITS is not completely in control of it.
> 
> I'm not really getting your point about virtualization though.

I think I'm mainly still confused by how MSI is implemented on
the CPU side. Your explanation makes sense though.

> > I would assume that once you have gone through the overhead of having
> > both an MSI and a normal interrupt line (with the need for
> > serialization vs DMA), you can just as well trap to user space to
> > deliver an IRQ to a guest.
> 
> The whole idea behind this bridge is to move wired interrupts to the
> periphery of a SoC. I don't think virtualization was part of the
> equation, but of course I can't speak for the "geniuses" behind the idea.
> 
> Or maybe I'm reading your question the wrong way, which is entirely
> possible given the lack of caffeine.

No, I think I get it now.

Arnd
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Marc Zyngier]

On 15/10/15 20:16, Arnd Bergmann wrote:
> On Thursday 15 October 2015 17:01:02 Marc Zyngier wrote:
>>
>> "Preconfigured" is the key word. While you can do something like that if
>> your hardware treats MSIs just as if they were wired interrupts
>> (something like GICv2m), it becomes far more hairy if the target of MSIs
>> is something like a GICv3 ITS (which is the case for HiSilicon mbigen).
>>
>> The main reason is that the ITS relies on "translation tables" kept in
>> memory, which the OS has to configure, and handing over pre-configured
>> tables is not something I'm looking forward to doing. From a CPU point
>> of view, this is akin entering the kernel with the MMU already on and no
>> idmap...
>>
>> The approach taken here is to make the MSI-ness explicit at the irqchip
>> level, and keep the interrupting device oblivious of that feature. Also,
>> this relies on the fact that we can have one MSI per wire, meaning that
>> we don't have to multiplex anything (no nested irqchip), and that we can
>> rely on hierarchical domains, which simplifies the code (at least for
>> the irqchip).
>>
> 
> Thanks, that already makes things much clearer. Just one more question:
> why can't those translation tables be configured statically by the
> irqchip driver? Is this all about being able to cut a few cycles
> in case of virtualization? 

Having a static configuration, while doable, complicates things for
everybody else. The LPI number used by the irqchip would need to be put
an some "exclusion list" to make sure it is not reallocated for other
subsystems (e.g PCI). The translation tables also define the target CPU,
which could cause interesting problems once combined with CPU hotplug if
the ITS is not completely in control of it.

I'm not really getting your point about virtualization though.

> I would assume that once you have gone through the overhead of having
> both an MSI and a normal interrupt line (with the need for
> serialization vs DMA), you can just as well trap to user space to
> deliver an IRQ to a guest.

The whole idea behind this bridge is to move wired interrupts to the
periphery of a SoC. I don't think virtualization was part of the
equation, but of course I can't speak for the "geniuses" behind the idea.

Or maybe I'm reading your question the wrong way, which is entirely
possible given the lack of caffeine.

Thanks,

M.
-- 
Jazz is not dead. It just smells funny...
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Marc Zyngier]

On 15/10/15 20:16, Arnd Bergmann wrote:
> On Thursday 15 October 2015 17:01:02 Marc Zyngier wrote:
>>
>> "Preconfigured" is the key word. While you can do something like that if
>> your hardware treats MSIs just as if they were wired interrupts
>> (something like GICv2m), it becomes far more hairy if the target of MSIs
>> is something like a GICv3 ITS (which is the case for HiSilicon mbigen).
>>
>> The main reason is that the ITS relies on "translation tables" kept in
>> memory, which the OS has to configure, and handing over pre-configured
>> tables is not something I'm looking forward to doing. From a CPU point
>> of view, this is akin entering the kernel with the MMU already on and no
>> idmap...
>>
>> The approach taken here is to make the MSI-ness explicit at the irqchip
>> level, and keep the interrupting device oblivious of that feature. Also,
>> this relies on the fact that we can have one MSI per wire, meaning that
>> we don't have to multiplex anything (no nested irqchip), and that we can
>> rely on hierarchical domains, which simplifies the code (at least for
>> the irqchip).
>>
> 
> Thanks, that already makes things much clearer. Just one more question:
> why can't those translation tables be configured statically by the
> irqchip driver? Is this all about being able to cut a few cycles
> in case of virtualization? 

Having a static configuration, while doable, complicates things for
everybody else. The LPI number used by the irqchip would need to be put
an some "exclusion list" to make sure it is not reallocated for other
subsystems (e.g PCI). The translation tables also define the target CPU,
which could cause interesting problems once combined with CPU hotplug if
the ITS is not completely in control of it.

I'm not really getting your point about virtualization though.

> I would assume that once you have gone through the overhead of having
> both an MSI and a normal interrupt line (with the need for
> serialization vs DMA), you can just as well trap to user space to
> deliver an IRQ to a guest.

The whole idea behind this bridge is to move wired interrupts to the
periphery of a SoC. I don't think virtualization was part of the
equation, but of course I can't speak for the "geniuses" behind the idea.

Or maybe I'm reading your question the wrong way, which is entirely
possible given the lack of caffeine.

Thanks,

M.
-- 
Jazz is not dead. It just smells funny...
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Arnd Bergmann]

On Friday 16 October 2015 09:03:14 Marc Zyngier wrote:
> On 15/10/15 20:16, Arnd Bergmann wrote:
> > On Thursday 15 October 2015 17:01:02 Marc Zyngier wrote:
> >>
> >> "Preconfigured" is the key word. While you can do something like that if
> >> your hardware treats MSIs just as if they were wired interrupts
> >> (something like GICv2m), it becomes far more hairy if the target of MSIs
> >> is something like a GICv3 ITS (which is the case for HiSilicon mbigen).
> >>
> >> The main reason is that the ITS relies on "translation tables" kept in
> >> memory, which the OS has to configure, and handing over pre-configured
> >> tables is not something I'm looking forward to doing. From a CPU point
> >> of view, this is akin entering the kernel with the MMU already on and no
> >> idmap...
> >>
> >> The approach taken here is to make the MSI-ness explicit at the irqchip
> >> level, and keep the interrupting device oblivious of that feature. Also,
> >> this relies on the fact that we can have one MSI per wire, meaning that
> >> we don't have to multiplex anything (no nested irqchip), and that we can
> >> rely on hierarchical domains, which simplifies the code (at least for
> >> the irqchip).
> >>
> > 
> > Thanks, that already makes things much clearer. Just one more question:
> > why can't those translation tables be configured statically by the
> > irqchip driver? Is this all about being able to cut a few cycles
> > in case of virtualization? 
> 
> Having a static configuration, while doable, complicates things for
> everybody else. The LPI number used by the irqchip would need to be put
> an some "exclusion list" to make sure it is not reallocated for other
> subsystems (e.g PCI). The translation tables also define the target CPU,
> which could cause interesting problems once combined with CPU hotplug if
> the ITS is not completely in control of it.
> 
> I'm not really getting your point about virtualization though.

I think I'm mainly still confused by how MSI is implemented on
the CPU side. Your explanation makes sense though.

> > I would assume that once you have gone through the overhead of having
> > both an MSI and a normal interrupt line (with the need for
> > serialization vs DMA), you can just as well trap to user space to
> > deliver an IRQ to a guest.
> 
> The whole idea behind this bridge is to move wired interrupts to the
> periphery of a SoC. I don't think virtualization was part of the
> equation, but of course I can't speak for the "geniuses" behind the idea.
> 
> Or maybe I'm reading your question the wrong way, which is entirely
> possible given the lack of caffeine.

No, I think I get it now.

Arnd
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Marc Zyngier]

Hi Gerry,

On 16/10/15 02:55, Jiang Liu wrote:
> On 2015/10/15 23:39, Marc Zyngier wrote:
>> There seems to be a new class of interrupt controller out there whose
>> sole purpose (apart from making everybody's life a nightmare) is to
>> turn wired interrupts into MSIs.
>>
>> Instead of considering that the MSIs allocated to a device are for the
>> direct use of that device, we can turn this set of MSIs into a irq
>> domain, and use that domain to build a standard irqchip on top of
>> that.
> Hi Marc,
>   There's a working to enable Intel VMD storage device, which
> has the similar requirement. Basically a PCIe hierarchy is hidden
> behind a parent PCIe device, so we need to use the PCIe irqs on parent
> to de-multiple PCIe IRQs from hidden PCIe devices. Seems a chance for
> consolidation here.

Do you know if there is a 1-1 mapping between the interrupts seen by the
parent device and those seen by the hidden devices? Or is it a case of
having to demultiplex the MSIs? Looks like the former, but I'd like to
be sure.

I just had a quick look at the code there:
https://lkml.org/lkml/2015/8/27/674

Is there anything more recent?

>   cc Keith Busch  who is the author of
> VMD device driver.

Sure, will do when I repost this (probably in a few weeks), and assuming
this fits the bill for Thomas and the MBIGEN folks.

Thanks,

M.
-- 
Jazz is not dead. It just smells funny...
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Jiang Liu]

On 2015/10/15 23:39, Marc Zyngier wrote:
> There seems to be a new class of interrupt controller out there whose
> sole purpose (apart from making everybody's life a nightmare) is to
> turn wired interrupts into MSIs.
> 
> Instead of considering that the MSIs allocated to a device are for the
> direct use of that device, we can turn this set of MSIs into a irq
> domain, and use that domain to build a standard irqchip on top of
> that.
Hi Marc,
There's a working to enable Intel VMD storage device, which
has the similar requirement. Basically a PCIe hierarchy is hidden
behind a parent PCIe device, so we need to use the PCIe irqs on parent
to de-multiple PCIe IRQs from hidden PCIe devices. Seems a chance for
consolidation here.
cc Keith Busch  who is the author of
VMD device driver.
Thanks!
Gerry

> 
> This requires some (slightly ugly) surgery in both the generic MSI and
> platform MSI layers, but the amount of code is actually relatively
> small (about +150 LoC so far).
> 
> On top of that, we add a dummy driver for a such a bridge, hoping that
> this will give enough information to driver authors so that they can
> use this new feature. An even more stupid client driver is provided to
> show the interrupt stack allocation:
> 
>  dummydev dummy-dev: Probing
>  dummydev dummy-dev: Allocated IRQ35
>  dummydev dummy-dev: Probing OK
>  dummydev dummy-dev: IRQ35 hwirq 5 domain msichip_domain_ops
>  dummydev dummy-dev: IRQ35 hwirq 0 domain msi_domain_ops
>  dummydev dummy-dev: IRQ35 hwirq 8192 domain its_domain_ops
>  dummydev dummy-dev: IRQ35 hwirq 8192 domain gic_irq_domain_ops
> 
> While this seems to work, it is far from being perfect:
> 
> - This is a prototype: error handling is rubbish, and there could be
>   better abstractions to use.
> 
> - It relies on not declaring this bridge with IRQCHIP_DECLARE in order
>   to prevent the DT layer from allocating interrupts behind our back.
> 
> - There is some probe ordering issues between the bridge and the wired
>   interrupt device, leading to the use of -EPROBE_DEFER.
> 
> The last two points could be addressed directly in the OF layer, as
> this is a generic device ordering issue (and people are already
> working on that).
> 
> I'd welcome any comment on that approach (though I'm going to make
> myself scarse over the next two weeks).
> 
> Marc Zyngier (7):
>   platform-msi: Allow MSIs to be allocated in chunks
>   platform-msi: Factor out allocation/free of private data
>   irqdomain: Make irq_domain_alloc_irqs_recursive available
>   genirq/msi: Make the .prepare callback reusable
>   genirq/msi: Add msi_domain_populate_irqs
>   platform-msi: Allow creation of a MSI-based stacked irq domain
>   irqchip: [Example] dummy wired interrupt/MSI bridge driver
> 
>  drivers/base/platform-msi.c   | 185 
>  drivers/irqchip/Kconfig   |   7 ++
>  drivers/irqchip/Makefile  |   1 +
>  drivers/irqchip/irq-msichip.c | 271 
> ++
>  include/linux/irqdomain.h |   3 +
>  include/linux/msi.h   |  16 +++
>  kernel/irq/irqdomain.c|   6 +-
>  kernel/irq/msi.c  |  47 +++-
>  8 files changed, 481 insertions(+), 55 deletions(-)
>  create mode 100644 drivers/irqchip/irq-msichip.c
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Arnd Bergmann]

On Thursday 15 October 2015 17:01:02 Marc Zyngier wrote:
> 
> "Preconfigured" is the key word. While you can do something like that if
> your hardware treats MSIs just as if they were wired interrupts
> (something like GICv2m), it becomes far more hairy if the target of MSIs
> is something like a GICv3 ITS (which is the case for HiSilicon mbigen).
> 
> The main reason is that the ITS relies on "translation tables" kept in
> memory, which the OS has to configure, and handing over pre-configured
> tables is not something I'm looking forward to doing. From a CPU point
> of view, this is akin entering the kernel with the MMU already on and no
> idmap...
> 
> The approach taken here is to make the MSI-ness explicit at the irqchip
> level, and keep the interrupting device oblivious of that feature. Also,
> this relies on the fact that we can have one MSI per wire, meaning that
> we don't have to multiplex anything (no nested irqchip), and that we can
> rely on hierarchical domains, which simplifies the code (at least for
> the irqchip).
> 

Thanks, that already makes things much clearer. Just one more question:
why can't those translation tables be configured statically by the
irqchip driver? Is this all about being able to cut a few cycles
in case of virtualization? I would assume that once you have gone through
the overhead of having both an MSI and a normal interrupt line (with
the need for serialization vs DMA), you can just as well trap to user
space to deliver an IRQ to a guest.

Arnd
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Marc Zyngier]

Hi Arnd,

On 15/10/15 16:46, Arnd Bergmann wrote:
> On Thursday 15 October 2015 16:39:21 Marc Zyngier wrote:
>> There seems to be a new class of interrupt controller out there whose
>> sole purpose (apart from making everybody's life a nightmare) is to
>> turn wired interrupts into MSIs.
>>
>> Instead of considering that the MSIs allocated to a device are for the
>> direct use of that device, we can turn this set of MSIs into a irq
>> domain, and use that domain to build a standard irqchip on top of
>> that.
>>
>> This requires some (slightly ugly) surgery in both the generic MSI and
>> platform MSI layers, but the amount of code is actually relatively
>> small (about +150 LoC so far).
>>
>> On top of that, we add a dummy driver for a such a bridge, hoping that
>> this will give enough information to driver authors so that they can
>> use this new feature. An even more stupid client driver is provided to
>> show the interrupt stack allocation:
> 
> I'm pretty sure you've thought of this before and it doesn't work, but
> can you explain why we can't just treat this as an edge-triggered
> nested irqchip? As long as the weird hardware can be preconfigured
> by the bootloader, the device that is attached to it shouldn't
> care how the interrupt ends up at the CPU.

"Preconfigured" is the key word. While you can do something like that if
your hardware treats MSIs just as if they were wired interrupts
(something like GICv2m), it becomes far more hairy if the target of MSIs
is something like a GICv3 ITS (which is the case for HiSilicon mbigen).

The main reason is that the ITS relies on "translation tables" kept in
memory, which the OS has to configure, and handing over pre-configured
tables is not something I'm looking forward to doing. From a CPU point
of view, this is akin entering the kernel with the MMU already on and no
idmap...

The approach taken here is to make the MSI-ness explicit at the irqchip
level, and keep the interrupting device oblivious of that feature. Also,
this relies on the fact that we can have one MSI per wire, meaning that
we don't have to multiplex anything (no nested irqchip), and that we can
rely on hierarchical domains, which simplifies the code (at least for
the irqchip).

Thanks,

M.
-- 
Jazz is not dead. It just smells funny...
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Arnd Bergmann]

On Thursday 15 October 2015 16:39:21 Marc Zyngier wrote:
> There seems to be a new class of interrupt controller out there whose
> sole purpose (apart from making everybody's life a nightmare) is to
> turn wired interrupts into MSIs.
> 
> Instead of considering that the MSIs allocated to a device are for the
> direct use of that device, we can turn this set of MSIs into a irq
> domain, and use that domain to build a standard irqchip on top of
> that.
> 
> This requires some (slightly ugly) surgery in both the generic MSI and
> platform MSI layers, but the amount of code is actually relatively
> small (about +150 LoC so far).
> 
> On top of that, we add a dummy driver for a such a bridge, hoping that
> this will give enough information to driver authors so that they can
> use this new feature. An even more stupid client driver is provided to
> show the interrupt stack allocation:

I'm pretty sure you've thought of this before and it doesn't work, but
can you explain why we can't just treat this as an edge-triggered
nested irqchip? As long as the weird hardware can be preconfigured
by the bootloader, the device that is attached to it shouldn't
care how the interrupt ends up at the CPU.

Arnd
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Arnd Bergmann]

On Thursday 15 October 2015 16:39:21 Marc Zyngier wrote:
> There seems to be a new class of interrupt controller out there whose
> sole purpose (apart from making everybody's life a nightmare) is to
> turn wired interrupts into MSIs.
> 
> Instead of considering that the MSIs allocated to a device are for the
> direct use of that device, we can turn this set of MSIs into a irq
> domain, and use that domain to build a standard irqchip on top of
> that.
> 
> This requires some (slightly ugly) surgery in both the generic MSI and
> platform MSI layers, but the amount of code is actually relatively
> small (about +150 LoC so far).
> 
> On top of that, we add a dummy driver for a such a bridge, hoping that
> this will give enough information to driver authors so that they can
> use this new feature. An even more stupid client driver is provided to
> show the interrupt stack allocation:

I'm pretty sure you've thought of this before and it doesn't work, but
can you explain why we can't just treat this as an edge-triggered
nested irqchip? As long as the weird hardware can be preconfigured
by the bootloader, the device that is attached to it shouldn't
care how the interrupt ends up at the CPU.

Arnd
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Marc Zyngier]

Hi Arnd,

On 15/10/15 16:46, Arnd Bergmann wrote:
> On Thursday 15 October 2015 16:39:21 Marc Zyngier wrote:
>> There seems to be a new class of interrupt controller out there whose
>> sole purpose (apart from making everybody's life a nightmare) is to
>> turn wired interrupts into MSIs.
>>
>> Instead of considering that the MSIs allocated to a device are for the
>> direct use of that device, we can turn this set of MSIs into a irq
>> domain, and use that domain to build a standard irqchip on top of
>> that.
>>
>> This requires some (slightly ugly) surgery in both the generic MSI and
>> platform MSI layers, but the amount of code is actually relatively
>> small (about +150 LoC so far).
>>
>> On top of that, we add a dummy driver for a such a bridge, hoping that
>> this will give enough information to driver authors so that they can
>> use this new feature. An even more stupid client driver is provided to
>> show the interrupt stack allocation:
> 
> I'm pretty sure you've thought of this before and it doesn't work, but
> can you explain why we can't just treat this as an edge-triggered
> nested irqchip? As long as the weird hardware can be preconfigured
> by the bootloader, the device that is attached to it shouldn't
> care how the interrupt ends up at the CPU.

"Preconfigured" is the key word. While you can do something like that if
your hardware treats MSIs just as if they were wired interrupts
(something like GICv2m), it becomes far more hairy if the target of MSIs
is something like a GICv3 ITS (which is the case for HiSilicon mbigen).

The main reason is that the ITS relies on "translation tables" kept in
memory, which the OS has to configure, and handing over pre-configured
tables is not something I'm looking forward to doing. From a CPU point
of view, this is akin entering the kernel with the MMU already on and no
idmap...

The approach taken here is to make the MSI-ness explicit at the irqchip
level, and keep the interrupting device oblivious of that feature. Also,
this relies on the fact that we can have one MSI per wire, meaning that
we don't have to multiplex anything (no nested irqchip), and that we can
rely on hierarchical domains, which simplifies the code (at least for
the irqchip).

Thanks,

M.
-- 
Jazz is not dead. It just smells funny...
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Jiang Liu]

On 2015/10/15 23:39, Marc Zyngier wrote:
> There seems to be a new class of interrupt controller out there whose
> sole purpose (apart from making everybody's life a nightmare) is to
> turn wired interrupts into MSIs.
> 
> Instead of considering that the MSIs allocated to a device are for the
> direct use of that device, we can turn this set of MSIs into a irq
> domain, and use that domain to build a standard irqchip on top of
> that.
Hi Marc,
There's a working to enable Intel VMD storage device, which
has the similar requirement. Basically a PCIe hierarchy is hidden
behind a parent PCIe device, so we need to use the PCIe irqs on parent
to de-multiple PCIe IRQs from hidden PCIe devices. Seems a chance for
consolidation here.
cc Keith Busch  who is the author of
VMD device driver.
Thanks!
Gerry

> 
> This requires some (slightly ugly) surgery in both the generic MSI and
> platform MSI layers, but the amount of code is actually relatively
> small (about +150 LoC so far).
> 
> On top of that, we add a dummy driver for a such a bridge, hoping that
> this will give enough information to driver authors so that they can
> use this new feature. An even more stupid client driver is provided to
> show the interrupt stack allocation:
> 
>  dummydev dummy-dev: Probing
>  dummydev dummy-dev: Allocated IRQ35
>  dummydev dummy-dev: Probing OK
>  dummydev dummy-dev: IRQ35 hwirq 5 domain msichip_domain_ops
>  dummydev dummy-dev: IRQ35 hwirq 0 domain msi_domain_ops
>  dummydev dummy-dev: IRQ35 hwirq 8192 domain its_domain_ops
>  dummydev dummy-dev: IRQ35 hwirq 8192 domain gic_irq_domain_ops
> 
> While this seems to work, it is far from being perfect:
> 
> - This is a prototype: error handling is rubbish, and there could be
>   better abstractions to use.
> 
> - It relies on not declaring this bridge with IRQCHIP_DECLARE in order
>   to prevent the DT layer from allocating interrupts behind our back.
> 
> - There is some probe ordering issues between the bridge and the wired
>   interrupt device, leading to the use of -EPROBE_DEFER.
> 
> The last two points could be addressed directly in the OF layer, as
> this is a generic device ordering issue (and people are already
> working on that).
> 
> I'd welcome any comment on that approach (though I'm going to make
> myself scarse over the next two weeks).
> 
> Marc Zyngier (7):
>   platform-msi: Allow MSIs to be allocated in chunks
>   platform-msi: Factor out allocation/free of private data
>   irqdomain: Make irq_domain_alloc_irqs_recursive available
>   genirq/msi: Make the .prepare callback reusable
>   genirq/msi: Add msi_domain_populate_irqs
>   platform-msi: Allow creation of a MSI-based stacked irq domain
>   irqchip: [Example] dummy wired interrupt/MSI bridge driver
> 
>  drivers/base/platform-msi.c   | 185 
>  drivers/irqchip/Kconfig   |   7 ++
>  drivers/irqchip/Makefile  |   1 +
>  drivers/irqchip/irq-msichip.c | 271 
> ++
>  include/linux/irqdomain.h |   3 +
>  include/linux/msi.h   |  16 +++
>  kernel/irq/irqdomain.c|   6 +-
>  kernel/irq/msi.c  |  47 +++-
>  8 files changed, 481 insertions(+), 55 deletions(-)
>  create mode 100644 drivers/irqchip/irq-msichip.c
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH RFC 0/7] Adding core support for wire-MSI bridges

[Arnd Bergmann]

On Thursday 15 October 2015 17:01:02 Marc Zyngier wrote:
> 
> "Preconfigured" is the key word. While you can do something like that if
> your hardware treats MSIs just as if they were wired interrupts
> (something like GICv2m), it becomes far more hairy if the target of MSIs
> is something like a GICv3 ITS (which is the case for HiSilicon mbigen).
> 
> The main reason is that the ITS relies on "translation tables" kept in
> memory, which the OS has to configure, and handing over pre-configured
> tables is not something I'm looking forward to doing. From a CPU point
> of view, this is akin entering the kernel with the MMU already on and no
> idmap...
> 
> The approach taken here is to make the MSI-ness explicit at the irqchip
> level, and keep the interrupting device oblivious of that feature. Also,
> this relies on the fact that we can have one MSI per wire, meaning that
> we don't have to multiplex anything (no nested irqchip), and that we can
> rely on hierarchical domains, which simplifies the code (at least for
> the irqchip).
> 

Thanks, that already makes things much clearer. Just one more question:
why can't those translation tables be configured statically by the
irqchip driver? Is this all about being able to cut a few cycles
in case of virtualization? I would assume that once you have gone through
the overhead of having both an MSI and a normal interrupt line (with
the need for serialization vs DMA), you can just as well trap to user
space to deliver an IRQ to a guest.

Arnd
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/