Re: [Xen-devel] [RFC for-4.8 0/6] xen/arm: Add support for mapping mmio-sram nodes into dom0

[Edgar E. Iglesias]

On Wed, May 25, 2016 at 03:24:39PM +0100, Julien Grall wrote:
> Hi Edgar,
> 
> On 25/05/16 14:29, Edgar E. Iglesias wrote:
> >On Tue, May 24, 2016 at 08:44:41PM +0100, Julien Grall wrote:
> >Looking a little closer, the place where the generic list of matches and
> >attributes doesn't work well is when trying to deal with the no-memory-wc
> >property available only in mmio-sram nodes.
> >
> >We'd really need an mmio-sram specific check in that case. Either
> >explicitely open coded in domain_build.c or something along the lines
> >f the .map method. Or did you have other ideas in mind?
> 
> How about extending the function dt_match_node and the structure
> dt_device_match to check the existence (or not) of a property?


Thanks Julien,

I had some time to get back to this and have created a new series
that I think is more along the lines with your suggestions.
I'll be posting an RFC soon.

Best regards,
Edgar

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC for-4.8 0/6] xen/arm: Add support for mapping mmio-sram nodes into dom0

[Julien Grall]

Hi Edgar,

On 25/05/16 14:29, Edgar E. Iglesias wrote:

On Tue, May 24, 2016 at 08:44:41PM +0100, Julien Grall wrote:
Looking a little closer, the place where the generic list of matches and
attributes doesn't work well is when trying to deal with the no-memory-wc
property available only in mmio-sram nodes.

We'd really need an mmio-sram specific check in that case. Either
explicitely open coded in domain_build.c or something along the lines
f the .map method. Or did you have other ideas in mind?


How about extending the function dt_match_node and the structure 
dt_device_match to check the existence (or not) of a property?


Regards,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC for-4.8 0/6] xen/arm: Add support for mapping mmio-sram nodes into dom0

[Edgar E. Iglesias]

On Tue, May 24, 2016 at 08:44:41PM +0100, Julien Grall wrote:
> Hi Edgar,
> 
> On 23/05/2016 16:42, Edgar E. Iglesias wrote:
> >On Mon, May 23, 2016 at 04:13:53PM +0100, Julien Grall wrote:
> >>On 23/05/16 15:02, Edgar E. Iglesias wrote:
> >>>On Mon, May 23, 2016 at 02:02:39PM +0100, Julien Grall wrote:
> (CC Wei Liu)
> 
> On 23/05/16 12:56, Edgar E. Iglesias wrote:
> >On Mon, May 23, 2016 at 11:29:31AM +0100, Julien Grall wrote:
> >>On 20/05/16 16:51, Edgar E. Iglesias wrote:
> >>>From: "Edgar E. Iglesias" 
> >>>
> >>>This series adds support for mapping mmio-sram nodes into dom0
> >>>as MEMORY, cached and with RWX perms.
> >>
> >>Can you explain why you chose to map those nodes as MEMORY, cached and 
> >>with
> >>RWX perms?
> >
> >My understanding is that these mmio-sram nodes are allowed to be treated 
> >as
> >Normal memory by the guest OS.
> >Guests could potentially do any kind of memory like operations on them.
> >
> >In our specific case, dom0 won't execute code from these regions but
> >Linux/dom0 ends up using standard memcpy/memset/x functions (not
> >memcpy_fromio and friends) on the regions.
> 
> I looked at the generic sram driver in Linux (drivers/misc/sram.c) which
> actually use memcpy_{to,from}io. So how you driver differs from the 
> generic
> one? What the SRAM will contain?
> >>>
> >>>We intend to use that same driver to map the memory but mmio-sram
> >>>nodes allow you to assign the regions to other device-drivers.
> >>>
> >>>Some examples:
> >>>Documentation/devicetree/bindings/crypto/marvell-cesa.txt
> >>>arch/arm/boot/dts/orion5x.dtsi
> >>>drivers/crypto/mv_cesa.c
> >>>
> >>>The cover letter for the sram driver had an example aswell allthough
> >>>the function names have changed since (it's of_gen_pool_get now):
> >>>https://lwn.net/Articles/537728/
> >>>
> >>>Nothing explicitely says that the regions can be assumed to be mapped
> >>>as Normal memory, but on Linux they do get mapped as Mormal WC mem
> >>>(unless the no-memory-wc prop is set on the node).
> >>>The marvell-cesa example also uses plain memset on the sram.
> >>
> >>I am a bit confused with this example. From my understanding of
> >>mv_cesa_get_ram, cp->sram can point either to a normal memory (?) area (see
> >>gen_pool_dma_alloc) or a Device_nGnRE area (see devm_ioremap_resource).
> >>
> >>However, memcpy_{from,to}io should be used when dealing with MMIO (the field
> >>sram has the __iomem attribute). See the commit 0f3304dc from Russel King
> >>related to marvell/cesa.
> >
> >
> >Yeah, I'm started to get confused too. Maybe they just forgot the memset
> >in drivers/crypto/mv_cesa.c.
> >
> >There are other examples though, that don't do fromio/toio at all.
> >Documentation/devicetree/bindings/media/coda.txt
> >drivers/media/platform/coda/coda-common.c
> >
> >Allthough ofcourse, these could also be wrong. Maybe I've missunderstood how
> >mmio-sram is supposed to be used.
> 
> I have talked about the memory attribute around me and the consensus is we
> should use the most relaxed mode that does not have any security implication
> or undefined behavior for a given device.
> 
> For SRAM it would be normal memory uncached (?) when the property
> "no-memory-wc" is not present, else TBD.
> 
> I suspect we would have to relax more MMIOs in the future. Rather than
> providing a function to map, the code is very similar except the memory
> attribute, I suggest to provide a list of compatible with the memory
> attribute to use.
> 
> All the children node would inherit the memory attribute of the parent.
> 
> What do you think?

Hi again,

Looking a little closer, the place where the generic list of matches and
attributes doesn't work well is when trying to deal with the no-memory-wc
property available only in mmio-sram nodes.

We'd really need an mmio-sram specific check in that case. Either
explicitely open coded in domain_build.c or something along the lines
f the .map method. Or did you have other ideas in mind?

Best regards,
Edgar

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC for-4.8 0/6] xen/arm: Add support for mapping mmio-sram nodes into dom0

[Edgar E. Iglesias]

On Tue, May 24, 2016 at 08:44:41PM +0100, Julien Grall wrote:
> Hi Edgar,

Hi Julien,

> 
> On 23/05/2016 16:42, Edgar E. Iglesias wrote:
> >On Mon, May 23, 2016 at 04:13:53PM +0100, Julien Grall wrote:
> >>On 23/05/16 15:02, Edgar E. Iglesias wrote:
> >>>On Mon, May 23, 2016 at 02:02:39PM +0100, Julien Grall wrote:
> (CC Wei Liu)
> 
> On 23/05/16 12:56, Edgar E. Iglesias wrote:
> >On Mon, May 23, 2016 at 11:29:31AM +0100, Julien Grall wrote:
> >>On 20/05/16 16:51, Edgar E. Iglesias wrote:
> >>>From: "Edgar E. Iglesias" 
> >>>
> >>>This series adds support for mapping mmio-sram nodes into dom0
> >>>as MEMORY, cached and with RWX perms.
> >>
> >>Can you explain why you chose to map those nodes as MEMORY, cached and 
> >>with
> >>RWX perms?
> >
> >My understanding is that these mmio-sram nodes are allowed to be treated 
> >as
> >Normal memory by the guest OS.
> >Guests could potentially do any kind of memory like operations on them.
> >
> >In our specific case, dom0 won't execute code from these regions but
> >Linux/dom0 ends up using standard memcpy/memset/x functions (not
> >memcpy_fromio and friends) on the regions.
> 
> I looked at the generic sram driver in Linux (drivers/misc/sram.c) which
> actually use memcpy_{to,from}io. So how you driver differs from the 
> generic
> one? What the SRAM will contain?
> >>>
> >>>We intend to use that same driver to map the memory but mmio-sram
> >>>nodes allow you to assign the regions to other device-drivers.
> >>>
> >>>Some examples:
> >>>Documentation/devicetree/bindings/crypto/marvell-cesa.txt
> >>>arch/arm/boot/dts/orion5x.dtsi
> >>>drivers/crypto/mv_cesa.c
> >>>
> >>>The cover letter for the sram driver had an example aswell allthough
> >>>the function names have changed since (it's of_gen_pool_get now):
> >>>https://lwn.net/Articles/537728/
> >>>
> >>>Nothing explicitely says that the regions can be assumed to be mapped
> >>>as Normal memory, but on Linux they do get mapped as Mormal WC mem
> >>>(unless the no-memory-wc prop is set on the node).
> >>>The marvell-cesa example also uses plain memset on the sram.
> >>
> >>I am a bit confused with this example. From my understanding of
> >>mv_cesa_get_ram, cp->sram can point either to a normal memory (?) area (see
> >>gen_pool_dma_alloc) or a Device_nGnRE area (see devm_ioremap_resource).
> >>
> >>However, memcpy_{from,to}io should be used when dealing with MMIO (the field
> >>sram has the __iomem attribute). See the commit 0f3304dc from Russel King
> >>related to marvell/cesa.
> >
> >
> >Yeah, I'm started to get confused too. Maybe they just forgot the memset
> >in drivers/crypto/mv_cesa.c.
> >
> >There are other examples though, that don't do fromio/toio at all.
> >Documentation/devicetree/bindings/media/coda.txt
> >drivers/media/platform/coda/coda-common.c
> >
> >Allthough ofcourse, these could also be wrong. Maybe I've missunderstood how
> >mmio-sram is supposed to be used.
> 
> I have talked about the memory attribute around me and the consensus is we
> should use the most relaxed mode that does not have any security implication
> or undefined behavior for a given device.

Yes, I agree with the principle.
The questionable part is what attributes are considered safe for specific
nodes.

> 
> For SRAM it would be normal memory uncached (?) when the property
> "no-memory-wc" is not present, else TBD.

That sounds good, it would solve our problems.


> I suspect we would have to relax more MMIOs in the future. Rather than
> providing a function to map, the code is very similar except the memory
> attribute, I suggest to provide a list of compatible with the memory
> attribute to use.
> 
> All the children node would inherit the memory attribute of the parent.
> 
> What do you think?

That sounds doable. I'll put something together addressing all your comments
and post new RFC.

Best regards,
Edgar

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC for-4.8 0/6] xen/arm: Add support for mapping mmio-sram nodes into dom0

[Stefano Stabellini]

On Wed, 25 May 2016, Julien Grall wrote:
> Hi Stefano,
> 
> On 25/05/16 10:43, Stefano Stabellini wrote:
> > > For SRAM it would be normal memory uncached (?) when the property
> > > "no-memory-wc" is not present, else TBD.
> > > 
> > > I suspect we would have to relax more MMIOs in the future. Rather than
> > > providing a function to map, the code is very similar except the memory
> > > attribute, I suggest to provide a list of compatible with the memory
> > > attribute
> > > to use.
> > > 
> > > All the children node would inherit the memory attribute of the parent.
> > > 
> > > What do you think?
> > 
> > That would work for device tree, but we still need to rely on the
> > hypercall for ACPI systems.
> > 
> > Given that it is not easy to add an additional parameter to
> > XENMEM_add_to_physmap_range, I think we'll have to provide a new
> > hypercall to allow setting attributes other than the Xen default. That
> > could be done in Xen 4.8 and Linux >= 4.9.
> 
> There is no need to introduce a new hypercall. The XENMEM_add_to_physmap_batch
> contains an unused field ('foreign_id', to be renamed) for mapping device
> MMIOs (see Jan's mail [1]).
> 
> XENMEM_add_to_physmap will always map with the default memory attribute
> (Device_nGnRnE) and if the kernel want to use another memory attribute, it
> will have to use XENMEM_add_to_physmap_batch.
> 
> With the plan suggested in [2], there are no modifications required in Linux
> for the moment.
> 
> Regards,
> 
> [1] http://lists.xenproject.org/archives/html/xen-devel/2016-05/msg02341.html
> [2] http://lists.xenproject.org/archives/html/xen-devel/2016-05/msg02347.html

I read the separate thread. Sounds good.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC for-4.8 0/6] xen/arm: Add support for mapping mmio-sram nodes into dom0

[Julien Grall]

Hi Stefano,

On 25/05/16 10:43, Stefano Stabellini wrote:

For SRAM it would be normal memory uncached (?) when the property
"no-memory-wc" is not present, else TBD.

I suspect we would have to relax more MMIOs in the future. Rather than
providing a function to map, the code is very similar except the memory
attribute, I suggest to provide a list of compatible with the memory attribute
to use.

All the children node would inherit the memory attribute of the parent.

What do you think?


That would work for device tree, but we still need to rely on the
hypercall for ACPI systems.

Given that it is not easy to add an additional parameter to
XENMEM_add_to_physmap_range, I think we'll have to provide a new
hypercall to allow setting attributes other than the Xen default. That
could be done in Xen 4.8 and Linux >= 4.9.


There is no need to introduce a new hypercall. The 
XENMEM_add_to_physmap_batch contains an unused field ('foreign_id', to 
be renamed) for mapping device MMIOs (see Jan's mail [1]).


XENMEM_add_to_physmap will always map with the default memory attribute 
(Device_nGnRnE) and if the kernel want to use another memory attribute, 
it will have to use XENMEM_add_to_physmap_batch.


With the plan suggested in [2], there are no modifications required in 
Linux for the moment.


Regards,

[1] 
http://lists.xenproject.org/archives/html/xen-devel/2016-05/msg02341.html
[2] 
http://lists.xenproject.org/archives/html/xen-devel/2016-05/msg02347.html


--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC for-4.8 0/6] xen/arm: Add support for mapping mmio-sram nodes into dom0

[Stefano Stabellini]

On Tue, 24 May 2016, Julien Grall wrote:
> On 23/05/2016 16:42, Edgar E. Iglesias wrote:
> > On Mon, May 23, 2016 at 04:13:53PM +0100, Julien Grall wrote:
> > > On 23/05/16 15:02, Edgar E. Iglesias wrote:
> > > > On Mon, May 23, 2016 at 02:02:39PM +0100, Julien Grall wrote:
> > > > > (CC Wei Liu)
> > > > > 
> > > > > On 23/05/16 12:56, Edgar E. Iglesias wrote:
> > > > > > On Mon, May 23, 2016 at 11:29:31AM +0100, Julien Grall wrote:
> > > > > > > On 20/05/16 16:51, Edgar E. Iglesias wrote:
> > > > > > > > From: "Edgar E. Iglesias" 
> > > > > > > > 
> > > > > > > > This series adds support for mapping mmio-sram nodes into dom0
> > > > > > > > as MEMORY, cached and with RWX perms.
> > > > > > > 
> > > > > > > Can you explain why you chose to map those nodes as MEMORY, cached
> > > > > > > and with
> > > > > > > RWX perms?
> > > > > > 
> > > > > > My understanding is that these mmio-sram nodes are allowed to be
> > > > > > treated as
> > > > > > Normal memory by the guest OS.
> > > > > > Guests could potentially do any kind of memory like operations on
> > > > > > them.
> > > > > > 
> > > > > > In our specific case, dom0 won't execute code from these regions but
> > > > > > Linux/dom0 ends up using standard memcpy/memset/x functions (not
> > > > > > memcpy_fromio and friends) on the regions.
> > > > > 
> > > > > I looked at the generic sram driver in Linux (drivers/misc/sram.c)
> > > > > which
> > > > > actually use memcpy_{to,from}io. So how you driver differs from the
> > > > > generic
> > > > > one? What the SRAM will contain?
> > > > 
> > > > We intend to use that same driver to map the memory but mmio-sram
> > > > nodes allow you to assign the regions to other device-drivers.
> > > > 
> > > > Some examples:
> > > > Documentation/devicetree/bindings/crypto/marvell-cesa.txt
> > > > arch/arm/boot/dts/orion5x.dtsi
> > > > drivers/crypto/mv_cesa.c
> > > > 
> > > > The cover letter for the sram driver had an example aswell allthough
> > > > the function names have changed since (it's of_gen_pool_get now):
> > > > https://lwn.net/Articles/537728/
> > > > 
> > > > Nothing explicitely says that the regions can be assumed to be mapped
> > > > as Normal memory, but on Linux they do get mapped as Mormal WC mem
> > > > (unless the no-memory-wc prop is set on the node).
> > > > The marvell-cesa example also uses plain memset on the sram.
> > > 
> > > I am a bit confused with this example. From my understanding of
> > > mv_cesa_get_ram, cp->sram can point either to a normal memory (?) area
> > > (see
> > > gen_pool_dma_alloc) or a Device_nGnRE area (see devm_ioremap_resource).
> > > 
> > > However, memcpy_{from,to}io should be used when dealing with MMIO (the
> > > field
> > > sram has the __iomem attribute). See the commit 0f3304dc from Russel King
> > > related to marvell/cesa.
> > 
> > 
> > Yeah, I'm started to get confused too. Maybe they just forgot the memset
> > in drivers/crypto/mv_cesa.c.
> > 
> > There are other examples though, that don't do fromio/toio at all.
> > Documentation/devicetree/bindings/media/coda.txt
> > drivers/media/platform/coda/coda-common.c
> > 
> > Allthough ofcourse, these could also be wrong. Maybe I've missunderstood how
> > mmio-sram is supposed to be used.
> 
> I have talked about the memory attribute around me and the consensus is we
> should use the most relaxed mode that does not have any security implication
> or undefined behavior for a given device.

I agree and it has always been the intention.


> For SRAM it would be normal memory uncached (?) when the property
> "no-memory-wc" is not present, else TBD.
> 
> I suspect we would have to relax more MMIOs in the future. Rather than
> providing a function to map, the code is very similar except the memory
> attribute, I suggest to provide a list of compatible with the memory attribute
> to use.
> 
> All the children node would inherit the memory attribute of the parent.
> 
> What do you think?

That would work for device tree, but we still need to rely on the
hypercall for ACPI systems.

Given that it is not easy to add an additional parameter to
XENMEM_add_to_physmap_range, I think we'll have to provide a new
hypercall to allow setting attributes other than the Xen default. That
could be done in Xen 4.8 and Linux >= 4.9.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC for-4.8 0/6] xen/arm: Add support for mapping mmio-sram nodes into dom0

[Stefano Stabellini]

On Mon, 23 May 2016, Julien Grall wrote:
> Note, that XENMAPSPACE_dev_mmio has been introduced in Xen 4.7 (which is due
> in a couple of weeks) and part of the stable ABI. So if it is not possible to
> relax the memory attribute, it might be worth to think fixing/reverting the
> hypercall for 4.7. Otherwise we would have to introduce a new one in the next
> release.

FYI the Linux side missed the Linux 4.7 merge window and it is now
queued for 4.8. Therefore theoretically could still be changed, but I
would be careful with any doing major changes at this point.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC for-4.8 0/6] xen/arm: Add support for mapping mmio-sram nodes into dom0

[Julien Grall]

Hi Edgar,

On 23/05/2016 16:42, Edgar E. Iglesias wrote:

On Mon, May 23, 2016 at 04:13:53PM +0100, Julien Grall wrote:

On 23/05/16 15:02, Edgar E. Iglesias wrote:

On Mon, May 23, 2016 at 02:02:39PM +0100, Julien Grall wrote:

(CC Wei Liu)

On 23/05/16 12:56, Edgar E. Iglesias wrote:

On Mon, May 23, 2016 at 11:29:31AM +0100, Julien Grall wrote:

On 20/05/16 16:51, Edgar E. Iglesias wrote:

From: "Edgar E. Iglesias" 

This series adds support for mapping mmio-sram nodes into dom0
as MEMORY, cached and with RWX perms.


Can you explain why you chose to map those nodes as MEMORY, cached and with
RWX perms?


My understanding is that these mmio-sram nodes are allowed to be treated as
Normal memory by the guest OS.
Guests could potentially do any kind of memory like operations on them.

In our specific case, dom0 won't execute code from these regions but
Linux/dom0 ends up using standard memcpy/memset/x functions (not
memcpy_fromio and friends) on the regions.


I looked at the generic sram driver in Linux (drivers/misc/sram.c) which
actually use memcpy_{to,from}io. So how you driver differs from the generic
one? What the SRAM will contain?


We intend to use that same driver to map the memory but mmio-sram
nodes allow you to assign the regions to other device-drivers.

Some examples:
Documentation/devicetree/bindings/crypto/marvell-cesa.txt
arch/arm/boot/dts/orion5x.dtsi
drivers/crypto/mv_cesa.c

The cover letter for the sram driver had an example aswell allthough
the function names have changed since (it's of_gen_pool_get now):
https://lwn.net/Articles/537728/

Nothing explicitely says that the regions can be assumed to be mapped
as Normal memory, but on Linux they do get mapped as Mormal WC mem
(unless the no-memory-wc prop is set on the node).
The marvell-cesa example also uses plain memset on the sram.


I am a bit confused with this example. From my understanding of
mv_cesa_get_ram, cp->sram can point either to a normal memory (?) area (see
gen_pool_dma_alloc) or a Device_nGnRE area (see devm_ioremap_resource).

However, memcpy_{from,to}io should be used when dealing with MMIO (the field
sram has the __iomem attribute). See the commit 0f3304dc from Russel King
related to marvell/cesa.



Yeah, I'm started to get confused too. Maybe they just forgot the memset
in drivers/crypto/mv_cesa.c.

There are other examples though, that don't do fromio/toio at all.
Documentation/devicetree/bindings/media/coda.txt
drivers/media/platform/coda/coda-common.c

Allthough ofcourse, these could also be wrong. Maybe I've missunderstood how
mmio-sram is supposed to be used.


I have talked about the memory attribute around me and the consensus is 
we should use the most relaxed mode that does not have any security 
implication or undefined behavior for a given device.


For SRAM it would be normal memory uncached (?) when the property 
"no-memory-wc" is not present, else TBD.


I suspect we would have to relax more MMIOs in the future. Rather than 
providing a function to map, the code is very similar except the memory 
attribute, I suggest to provide a list of compatible with the memory 
attribute to use.


All the children node would inherit the memory attribute of the parent.

What do you think?

Regards,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC for-4.8 0/6] xen/arm: Add support for mapping mmio-sram nodes into dom0

[Edgar E. Iglesias]

On Mon, May 23, 2016 at 04:13:53PM +0100, Julien Grall wrote:
> Hi Edgar,
> 
> On 23/05/16 15:02, Edgar E. Iglesias wrote:
> >On Mon, May 23, 2016 at 02:02:39PM +0100, Julien Grall wrote:
> >>(CC Wei Liu)
> >>
> >>On 23/05/16 12:56, Edgar E. Iglesias wrote:
> >>>On Mon, May 23, 2016 at 11:29:31AM +0100, Julien Grall wrote:
> On 20/05/16 16:51, Edgar E. Iglesias wrote:
> >From: "Edgar E. Iglesias" 
> >
> >This series adds support for mapping mmio-sram nodes into dom0
> >as MEMORY, cached and with RWX perms.
> 
> Can you explain why you chose to map those nodes as MEMORY, cached and 
> with
> RWX perms?
> >>>
> >>>My understanding is that these mmio-sram nodes are allowed to be treated as
> >>>Normal memory by the guest OS.
> >>>Guests could potentially do any kind of memory like operations on them.
> >>>
> >>>In our specific case, dom0 won't execute code from these regions but
> >>>Linux/dom0 ends up using standard memcpy/memset/x functions (not
> >>>memcpy_fromio and friends) on the regions.
> >>
> >>I looked at the generic sram driver in Linux (drivers/misc/sram.c) which
> >>actually use memcpy_{to,from}io. So how you driver differs from the generic
> >>one? What the SRAM will contain?
> >
> >We intend to use that same driver to map the memory but mmio-sram
> >nodes allow you to assign the regions to other device-drivers.
> >
> >Some examples:
> >Documentation/devicetree/bindings/crypto/marvell-cesa.txt
> >arch/arm/boot/dts/orion5x.dtsi
> >drivers/crypto/mv_cesa.c
> >
> >The cover letter for the sram driver had an example aswell allthough
> >the function names have changed since (it's of_gen_pool_get now):
> >https://lwn.net/Articles/537728/
> >
> >Nothing explicitely says that the regions can be assumed to be mapped
> >as Normal memory, but on Linux they do get mapped as Mormal WC mem
> >(unless the no-memory-wc prop is set on the node).
> >The marvell-cesa example also uses plain memset on the sram.
> 
> I am a bit confused with this example. From my understanding of
> mv_cesa_get_ram, cp->sram can point either to a normal memory (?) area (see
> gen_pool_dma_alloc) or a Device_nGnRE area (see devm_ioremap_resource).
> 
> However, memcpy_{from,to}io should be used when dealing with MMIO (the field
> sram has the __iomem attribute). See the commit 0f3304dc from Russel King
> related to marvell/cesa.


Yeah, I'm started to get confused too. Maybe they just forgot the memset
in drivers/crypto/mv_cesa.c.

There are other examples though, that don't do fromio/toio at all.
Documentation/devicetree/bindings/media/coda.txt
drivers/media/platform/coda/coda-common.c

Allthough ofcourse, these could also be wrong. Maybe I've missunderstood how
mmio-sram is supposed to be used.


> >>>We saw issues with memset doing cache operations on DEVICE memory in
> >>>the past (external data aborts). I can't find the text in the ARM ARM
> >>>regarding this at the moment but IIRC, dc ops on device mem are not
> >>>allowed.
> >>
> >>The ARM ARM (D3-1632 in ARM DDI 0487A.i) states "that the effects of the
> >>cache maintenance instructions can apply regardless of:
> >>Whether the address accessed:
> >>* Is Normal memory or Device memory.
> >>* Has the Cacheable attribute or the Non-cacheable attribute.
> >>"
> >>
> >>So I am not sure why you would get an external data aborts when executing dc
> >>ops on device memory.
> >
> >
> >OK, I found a reference to the issue we were seeing. If you look at ARM ARM
> >D3.4.9 Data cache zero instruction, you'll see that the DC ZVA insn always
> >generates an alignment fault if used on device memory.
> 
> Thinking a bit more, I find weird to use cache instructions on the SRAM
> given the region will be mapped uncacheable by Linux. Note that SRAM is
> usually very-fast so using the cache may not improve that much the
> performance.
> 
> How bad would the performance be if the processor cannot speculate access to
> the SRAM?

I don't have the setup available right now to try it out but I wouldn't
expect it to be very signifcant for our apps. In our case it had more to
do with the ability to use the remote-proc drivers as they are and really
any linux driver that potentially can take an sram as a memory region for
local use (DMA buffers or whatever), without changing the memory ops to
the _fromio/_toio versions.

Best regards,
Edgar

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC for-4.8 0/6] xen/arm: Add support for mapping mmio-sram nodes into dom0

[Julien Grall]

Hi Edgar,

On 23/05/16 15:02, Edgar E. Iglesias wrote:

On Mon, May 23, 2016 at 02:02:39PM +0100, Julien Grall wrote:

(CC Wei Liu)

On 23/05/16 12:56, Edgar E. Iglesias wrote:

On Mon, May 23, 2016 at 11:29:31AM +0100, Julien Grall wrote:

On 20/05/16 16:51, Edgar E. Iglesias wrote:

From: "Edgar E. Iglesias" 

This series adds support for mapping mmio-sram nodes into dom0
as MEMORY, cached and with RWX perms.


Can you explain why you chose to map those nodes as MEMORY, cached and with
RWX perms?


My understanding is that these mmio-sram nodes are allowed to be treated as
Normal memory by the guest OS.
Guests could potentially do any kind of memory like operations on them.

In our specific case, dom0 won't execute code from these regions but
Linux/dom0 ends up using standard memcpy/memset/x functions (not
memcpy_fromio and friends) on the regions.


I looked at the generic sram driver in Linux (drivers/misc/sram.c) which
actually use memcpy_{to,from}io. So how you driver differs from the generic
one? What the SRAM will contain?


We intend to use that same driver to map the memory but mmio-sram
nodes allow you to assign the regions to other device-drivers.

Some examples:
Documentation/devicetree/bindings/crypto/marvell-cesa.txt
arch/arm/boot/dts/orion5x.dtsi
drivers/crypto/mv_cesa.c

The cover letter for the sram driver had an example aswell allthough
the function names have changed since (it's of_gen_pool_get now):
https://lwn.net/Articles/537728/

Nothing explicitely says that the regions can be assumed to be mapped
as Normal memory, but on Linux they do get mapped as Mormal WC mem
(unless the no-memory-wc prop is set on the node).
The marvell-cesa example also uses plain memset on the sram.


I am a bit confused with this example. From my understanding of 
mv_cesa_get_ram, cp->sram can point either to a normal memory (?) area 
(see gen_pool_dma_alloc) or a Device_nGnRE area (see devm_ioremap_resource).


However, memcpy_{from,to}io should be used when dealing with MMIO (the 
field sram has the __iomem attribute). See the commit 0f3304dc from 
Russel King related to marvell/cesa.





We saw issues with memset doing cache operations on DEVICE memory in
the past (external data aborts). I can't find the text in the ARM ARM
regarding this at the moment but IIRC, dc ops on device mem are not
allowed.


The ARM ARM (D3-1632 in ARM DDI 0487A.i) states "that the effects of the
cache maintenance instructions can apply regardless of:
Whether the address accessed:
* Is Normal memory or Device memory.
* Has the Cacheable attribute or the Non-cacheable attribute.
"

So I am not sure why you would get an external data aborts when executing dc
ops on device memory.



OK, I found a reference to the issue we were seeing. If you look at ARM ARM
D3.4.9 Data cache zero instruction, you'll see that the DC ZVA insn always
generates an alignment fault if used on device memory.


Thinking a bit more, I find weird to use cache instructions on the SRAM 
given the region will be mapped uncacheable by Linux. Note that SRAM is 
usually very-fast so using the cache may not improve that much the 
performance.


How bad would the performance be if the processor cannot speculate 
access to the SRAM?


Regards,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC for-4.8 0/6] xen/arm: Add support for mapping mmio-sram nodes into dom0

[Edgar E. Iglesias]

On Mon, May 23, 2016 at 02:02:39PM +0100, Julien Grall wrote:
> (CC Wei Liu)
> 
> On 23/05/16 12:56, Edgar E. Iglesias wrote:
> >On Mon, May 23, 2016 at 11:29:31AM +0100, Julien Grall wrote:
> >>On 20/05/16 16:51, Edgar E. Iglesias wrote:
> >>>From: "Edgar E. Iglesias" 
> >>>
> >>>This series adds support for mapping mmio-sram nodes into dom0
> >>>as MEMORY, cached and with RWX perms.
> >>
> >>Can you explain why you chose to map those nodes as MEMORY, cached and with
> >>RWX perms?
> >
> >My understanding is that these mmio-sram nodes are allowed to be treated as
> >Normal memory by the guest OS.
> >Guests could potentially do any kind of memory like operations on them.
> >
> >In our specific case, dom0 won't execute code from these regions but
> >Linux/dom0 ends up using standard memcpy/memset/x functions (not
> >memcpy_fromio and friends) on the regions.
> 
> I looked at the generic sram driver in Linux (drivers/misc/sram.c) which
> actually use memcpy_{to,from}io. So how you driver differs from the generic
> one? What the SRAM will contain?

We intend to use that same driver to map the memory but mmio-sram
nodes allow you to assign the regions to other device-drivers.

Some examples:
Documentation/devicetree/bindings/crypto/marvell-cesa.txt
arch/arm/boot/dts/orion5x.dtsi
drivers/crypto/mv_cesa.c

The cover letter for the sram driver had an example aswell allthough
the function names have changed since (it's of_gen_pool_get now):
https://lwn.net/Articles/537728/

Nothing explicitely says that the regions can be assumed to be mapped
as Normal memory, but on Linux they do get mapped as Mormal WC mem
(unless the no-memory-wc prop is set on the node).

The marvell-cesa example also uses plain memset on the sram.


> >We saw issues with memset doing cache operations on DEVICE memory in
> >the past (external data aborts). I can't find the text in the ARM ARM
> >regarding this at the moment but IIRC, dc ops on device mem are not
> >allowed.
> 
> The ARM ARM (D3-1632 in ARM DDI 0487A.i) states "that the effects of the
> cache maintenance instructions can apply regardless of:
> Whether the address accessed:
>   * Is Normal memory or Device memory.
>   * Has the Cacheable attribute or the Non-cacheable attribute.
> "
> 
> So I am not sure why you would get an external data aborts when executing dc
> ops on device memory.


OK, I found a reference to the issue we were seeing. If you look at ARM ARM
D3.4.9 Data cache zero instruction, you'll see that the DC ZVA insn always
generates an alignment fault if used on device memory.

I must have remembered incorrectly about the external abort, it's an
alignment fault.


> >We could also see alignment problems, as the alignment checks for
> >ARMv8 differ somewhat between normal memory and device memory.
> >
> >A third reason is performance. The rules for speculative accesses
> >and caching are different between device and normal memory.
> >
> >So I opted for the most permissive attributes thinking that dom0 can
> >apply further restrictions if it needs to do so.
> >
> >
> >>>Dom0 can then choose to further restrict these mappings if needed.
> >>>We only look at the outer mmio-sram region. The sub-area nodes that
> >>>describe allocations within the mmio-sram region are only meaningful
> >>>to the guest AFAICT.
> >>>
> >>>In an attempt to avoid growing the already fairly large domain_build.c
> >>>file, I've tried to implement a distributed way to deal with these kind
> >>>of special/custom mapping needs. These can live outside of domain_build.c
> >>>and are registerd by means of a .map method in the device_spec.
> >>>
> >>>If this approach is not good, I'm happy to bin it and try something else.
> >>
> >>We will have a similar problem when using ACPI for DOM0 or mapping a such
> >>MMIO to the guest. The hypercalls XENMAPSPACE_dev_mmio and
> >>XEN_DOMCTL_memory_mapping do not provide enough information to know the
> >>attribute to be used for mapping.
> >>
> >>MMIO are always mapped in Stage-2 with Device_nGnRE, which is quite
> >>restrictive. This would also impact any MMIO regions, such as the video RAM
> >>buffer, that could be mapped write-combine.
> >>
> >>After reading the ARM ARM (B2.8.2 ARM DII 0486.i), I think we could relax
> >>the stage-2 mapping by using Device_GRE for all the device MMIOs but the
> >>GIC.
> >>
> >>We have to keep the GIC MMIO with the most restrictive memory attribute to
> >>avoid potential side-effect when Xen is switching between multiple vCPUs.
> >
> >I see, that makes sense.
> >
> >
> >>All the other devices will be exclusive to a specific guest, so the guest
> >>can handle the device the way it wants. This may require some extra-care
> >>when reassigning the device to another domain.
> >>
> >>Edgar, would Device_GRE be fine for you?
> >
> >Sorry, but I don't think so. We can live with performance impacts but it's
> >harder with the external data aborts and potential alignment checks.
> >
> >
> >>Note, 

Re: [Xen-devel] [RFC for-4.8 0/6] xen/arm: Add support for mapping mmio-sram nodes into dom0

[Julien Grall]

(CC Wei Liu)

On 23/05/16 12:56, Edgar E. Iglesias wrote:

On Mon, May 23, 2016 at 11:29:31AM +0100, Julien Grall wrote:

On 20/05/16 16:51, Edgar E. Iglesias wrote:

From: "Edgar E. Iglesias" 

This series adds support for mapping mmio-sram nodes into dom0
as MEMORY, cached and with RWX perms.


Can you explain why you chose to map those nodes as MEMORY, cached and with
RWX perms?


My understanding is that these mmio-sram nodes are allowed to be treated as
Normal memory by the guest OS.
Guests could potentially do any kind of memory like operations on them.

In our specific case, dom0 won't execute code from these regions but
Linux/dom0 ends up using standard memcpy/memset/x functions (not
memcpy_fromio and friends) on the regions.


I looked at the generic sram driver in Linux (drivers/misc/sram.c) which 
actually use memcpy_{to,from}io. So how you driver differs from the 
generic one? What the SRAM will contain?




We saw issues with memset doing cache operations on DEVICE memory in
the past (external data aborts). I can't find the text in the ARM ARM
regarding this at the moment but IIRC, dc ops on device mem are not
allowed.


The ARM ARM (D3-1632 in ARM DDI 0487A.i) states "that the effects of the 
cache maintenance instructions can apply regardless of:

Whether the address accessed:
* Is Normal memory or Device memory.
* Has the Cacheable attribute or the Non-cacheable attribute.
"

So I am not sure why you would get an external data aborts when 
executing dc ops on device memory.




We could also see alignment problems, as the alignment checks for
ARMv8 differ somewhat between normal memory and device memory.

A third reason is performance. The rules for speculative accesses
and caching are different between device and normal memory.

So I opted for the most permissive attributes thinking that dom0 can
apply further restrictions if it needs to do so.



Dom0 can then choose to further restrict these mappings if needed.
We only look at the outer mmio-sram region. The sub-area nodes that
describe allocations within the mmio-sram region are only meaningful
to the guest AFAICT.

In an attempt to avoid growing the already fairly large domain_build.c
file, I've tried to implement a distributed way to deal with these kind
of special/custom mapping needs. These can live outside of domain_build.c
and are registerd by means of a .map method in the device_spec.

If this approach is not good, I'm happy to bin it and try something else.


We will have a similar problem when using ACPI for DOM0 or mapping a such
MMIO to the guest. The hypercalls XENMAPSPACE_dev_mmio and
XEN_DOMCTL_memory_mapping do not provide enough information to know the
attribute to be used for mapping.

MMIO are always mapped in Stage-2 with Device_nGnRE, which is quite
restrictive. This would also impact any MMIO regions, such as the video RAM
buffer, that could be mapped write-combine.

After reading the ARM ARM (B2.8.2 ARM DII 0486.i), I think we could relax
the stage-2 mapping by using Device_GRE for all the device MMIOs but the
GIC.

We have to keep the GIC MMIO with the most restrictive memory attribute to
avoid potential side-effect when Xen is switching between multiple vCPUs.


I see, that makes sense.



All the other devices will be exclusive to a specific guest, so the guest
can handle the device the way it wants. This may require some extra-care
when reassigning the device to another domain.

Edgar, would Device_GRE be fine for you?


Sorry, but I don't think so. We can live with performance impacts but it's
harder with the external data aborts and potential alignment checks.



Note, that XENMAPSPACE_dev_mmio has been introduced in Xen 4.7 (which is due
in a couple of weeks) and part of the stable ABI. So if it is not possible
to relax the memory attribute, it might be worth to think fixing/reverting
the hypercall for 4.7. Otherwise we would have to introduce a new one in the
next release.


Yes, maybe we could add something along the lines of the pgprot arg
that Linux/arm64 has in it's __ioremap call. Even if we only support
PROT_DEVICE_nGnRE (or GRE) in 4.7 at the least we can reuse the ABI
to add more modes in 4.8?


I will bring it on a separate subject with the REST of the maintainers 
when we find out what are the possible memory attributes.


Depending on the result, this might be considered as a blocker as I do 
not think we should avoid to  introduce a new hypercall 
(XENMAPSPACE_dev_mmio) which is known to not fit for every case.


Regards,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC for-4.8 0/6] xen/arm: Add support for mapping mmio-sram nodes into dom0

[Edgar E. Iglesias]

On Mon, May 23, 2016 at 11:29:31AM +0100, Julien Grall wrote:
> Hello Edgar,
> 
> I have CCed a couple of people from ARM to get more input on it.

Thanks Julien,


> On 20/05/16 16:51, Edgar E. Iglesias wrote:
> >From: "Edgar E. Iglesias" 
> >
> >This series adds support for mapping mmio-sram nodes into dom0
> >as MEMORY, cached and with RWX perms.
> 
> Can you explain why you chose to map those nodes as MEMORY, cached and with
> RWX perms?

My understanding is that these mmio-sram nodes are allowed to be treated as
Normal memory by the guest OS.
Guests could potentially do any kind of memory like operations on them.

In our specific case, dom0 won't execute code from these regions but
Linux/dom0 ends up using standard memcpy/memset/x functions (not
memcpy_fromio and friends) on the regions.

We saw issues with memset doing cache operations on DEVICE memory in
the past (external data aborts). I can't find the text in the ARM ARM
regarding this at the moment but IIRC, dc ops on device mem are not
allowed.

We could also see alignment problems, as the alignment checks for
ARMv8 differ somewhat between normal memory and device memory.

A third reason is performance. The rules for speculative accesses
and caching are different between device and normal memory.

So I opted for the most permissive attributes thinking that dom0 can
apply further restrictions if it needs to do so.


> >Dom0 can then choose to further restrict these mappings if needed.
> >We only look at the outer mmio-sram region. The sub-area nodes that
> >describe allocations within the mmio-sram region are only meaningful
> >to the guest AFAICT.
> >
> >In an attempt to avoid growing the already fairly large domain_build.c
> >file, I've tried to implement a distributed way to deal with these kind
> >of special/custom mapping needs. These can live outside of domain_build.c
> >and are registerd by means of a .map method in the device_spec.
> >
> >If this approach is not good, I'm happy to bin it and try something else.
> 
> We will have a similar problem when using ACPI for DOM0 or mapping a such
> MMIO to the guest. The hypercalls XENMAPSPACE_dev_mmio and
> XEN_DOMCTL_memory_mapping do not provide enough information to know the
> attribute to be used for mapping.
> 
> MMIO are always mapped in Stage-2 with Device_nGnRE, which is quite
> restrictive. This would also impact any MMIO regions, such as the video RAM
> buffer, that could be mapped write-combine.
> 
> After reading the ARM ARM (B2.8.2 ARM DII 0486.i), I think we could relax
> the stage-2 mapping by using Device_GRE for all the device MMIOs but the
> GIC.
> 
> We have to keep the GIC MMIO with the most restrictive memory attribute to
> avoid potential side-effect when Xen is switching between multiple vCPUs.

I see, that makes sense.


> All the other devices will be exclusive to a specific guest, so the guest
> can handle the device the way it wants. This may require some extra-care
> when reassigning the device to another domain.
> 
> Edgar, would Device_GRE be fine for you?

Sorry, but I don't think so. We can live with performance impacts but it's
harder with the external data aborts and potential alignment checks.


> Note, that XENMAPSPACE_dev_mmio has been introduced in Xen 4.7 (which is due
> in a couple of weeks) and part of the stable ABI. So if it is not possible
> to relax the memory attribute, it might be worth to think fixing/reverting
> the hypercall for 4.7. Otherwise we would have to introduce a new one in the
> next release.

Yes, maybe we could add something along the lines of the pgprot arg
that Linux/arm64 has in it's __ioremap call. Even if we only support
PROT_DEVICE_nGnRE (or GRE) in 4.7 at the least we can reuse the ABI
to add more modes in 4.8?

Best regards,
Edgar

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC for-4.8 0/6] xen/arm: Add support for mapping mmio-sram nodes into dom0

[Julien Grall]

Hello Edgar,

I have CCed a couple of people from ARM to get more input on it.

On 20/05/16 16:51, Edgar E. Iglesias wrote:

From: "Edgar E. Iglesias" 

This series adds support for mapping mmio-sram nodes into dom0
as MEMORY, cached and with RWX perms.


Can you explain why you chose to map those nodes as MEMORY, cached and 
with RWX perms?




Dom0 can then choose to further restrict these mappings if needed.
We only look at the outer mmio-sram region. The sub-area nodes that
describe allocations within the mmio-sram region are only meaningful
to the guest AFAICT.

In an attempt to avoid growing the already fairly large domain_build.c
file, I've tried to implement a distributed way to deal with these kind
of special/custom mapping needs. These can live outside of domain_build.c
and are registerd by means of a .map method in the device_spec.

If this approach is not good, I'm happy to bin it and try something else.


We will have a similar problem when using ACPI for DOM0 or mapping a 
such MMIO to the guest. The hypercalls XENMAPSPACE_dev_mmio and 
XEN_DOMCTL_memory_mapping do not provide enough information to know the 
attribute to be used for mapping.


MMIO are always mapped in Stage-2 with Device_nGnRE, which is quite 
restrictive. This would also impact any MMIO regions, such as the video 
RAM buffer, that could be mapped write-combine.


After reading the ARM ARM (B2.8.2 ARM DII 0486.i), I think we could 
relax the stage-2 mapping by using Device_GRE for all the device MMIOs 
but the GIC.


We have to keep the GIC MMIO with the most restrictive memory attribute 
to avoid potential side-effect when Xen is switching between multiple 
vCPUs. All the other devices will be exclusive to a specific guest, so 
the guest can handle the device the way it wants. This may require some 
extra-care when reassigning the device to another domain.


Edgar, would Device_GRE be fine for you?

Note, that XENMAPSPACE_dev_mmio has been introduced in Xen 4.7 (which is 
due in a couple of weeks) and part of the stable ABI. So if it is not 
possible to relax the memory attribute, it might be worth to think 
fixing/reverting the hypercall for 4.7. Otherwise we would have to 
introduce a new one in the next release.


Regards,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [RFC for-4.8 0/6] xen/arm: Add support for mapping mmio-sram nodes into dom0

[Edgar E. Iglesias]

From: "Edgar E. Iglesias" 

This series adds support for mapping mmio-sram nodes into dom0
as MEMORY, cached and with RWX perms.

Dom0 can then choose to further restrict these mappings if needed.
We only look at the outer mmio-sram region. The sub-area nodes that
describe allocations within the mmio-sram region are only meaningful
to the guest AFAICT.

In an attempt to avoid growing the already fairly large domain_build.c
file, I've tried to implement a distributed way to deal with these kind
of special/custom mapping needs. These can live outside of domain_build.c
and are registerd by means of a .map method in the device_spec.

If this approach is not good, I'm happy to bin it and try something else.

Comments welcome!

Best regards,
Edgar

Edgar E. Iglesias (6):
  xen/arm: Add device_get_desc()
  xen/arm: Add an optional map function to the device descriptor
  xen/arm: Add a DEVICE_MEMORY class
  xen/arm: Add helper functions to map RWX memory regions
  xen/arm: Add an mmio-sram device
  xen/arm: Avoid multiple dev class lookups in handle_node

 xen/arch/arm/Makefile|  1 +
 xen/arch/arm/device.c| 15 
 xen/arch/arm/domain_build.c  | 19 +++--
 xen/arch/arm/mmio-sram.c | 92 
 xen/arch/arm/p2m.c   | 26 +
 xen/include/asm-arm/device.h | 12 ++
 xen/include/asm-arm/p2m.h| 10 +
 7 files changed, 172 insertions(+), 3 deletions(-)
 create mode 100644 xen/arch/arm/mmio-sram.c

-- 
2.5.0


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel