Re: [PATCH] Xen: Design doc for 1:1 direct-map and static allocation
On 20/05/2021 06:36, Penny Zheng wrote:
Hi Julien
Hi Penny,
+
+Later, when domain get destroyed and memory relinquished, only pages
+in `page_list` go back to heap, and pages in `reserved_page_list` shall not.
While going through the series, I could not find any code implementing this.
However, this is not enough to prevent a page to go to the heap allocator
because a domain can release memory at runtime using hypercalls like
XENMEM_remove_from_physmap.
One of the use case is when the guest decides to balloon out some memory.
This will call free_domheap_pages().
Effectively, you are treating static memory as domheap pages. So I think it
would be better if you hook in free_domheap_pages() to decide which
allocator is used.
Now, if a guest can balloon out memory, it can also balloon in memory.
There are two cases:
1) The region used to be RAM region statically allocated
2) The region used to be unallocated.
I think for 1), we need to be able to re-use the page previously. For 2), it is
not clear to me whether a guest with memory statically allocated should be
allowed to allocate "dynamic" pages.
Yeah, I share the same with you of hooking in free_domheap_pages(). I'm thinking
that if pages of PGC_reserved, we may create a new func free_staticmem_pages to
free them.
For issues on ballooning out or in, it is not supported here.
It is fine that the implementation doesn't yet implement it. However, I
think the design document should take into account ballooning. This is
because even if...
Domain on Static Allocation and 1:1 direct-map are all based on dom0-less right
now, so no PV, grant table, event channel, etc, considered.
... there is no PV support & co, a guest is still able to issue
hypercalls (they are not hidden). Therefore your guest will be able to
disturb your static allocation.
Right now, it only supports device got passthrough into the guest.
+### Memory Allocation for Domains on Static Allocation
+
+RAM regions pre-defined as static memory for one specifc domain shall
+be parsed and reserved from the beginning. And they shall never go to
+any memory allocator for any use.
Technically, you are introducing a new allocator. So do you mean they should
not be given to neither the buddy allocator nor the bot allocator?
Yes. These pre-defined RAM regions will not be given to any current
memory allocator. If be given there, there is no guarantee that it will
not be allocated for other use.
And right now, in my current design, these pre-defined RAM regions are either
for
one specific domain as guest RAM or as XEN heap.
+
+Later when allocating static memory for this specific domain, after
+acquiring those reserved regions, users need to a do set of
+verification before assigning.
+For each page there, it at least includes the following steps:
+1. Check if it is in free state and has zero reference count.
+2. Check if the page is reserved(`PGC_reserved`).
+
+Then, assigning these pages to this specific domain, and all pages go
+to one new linked page list `reserved_page_list`.
+
+At last, set up guest P2M mapping. By default, it shall be mapped to
+the fixed guest RAM address `GUEST_RAM0_BASE`, `GUEST_RAM1_BASE`,
+just like normal domains. But later in 1:1 direct-map design, if
+`direct-map` is set, the guest physical address will equal to physical
address.
+
+### Static Allocation for Xen itself
+
+### New Deivce Tree Node: `xen,reserved_heap`
s/Deivce/Device/
Thx.
+
+Static memory for Xen heap refers to parts of RAM reserved in the
+beginning for Xen heap only. The memory is pre-defined through XEN
+configuration using physical address ranges.
+
+The reserved memory for Xen heap is an optional feature and can be
+enabled by adding a device tree property in the `chosen` node.
+Currently, this feature is only supported on AArch64.
+
+Here is one example:
+
+
+chosen {
+xen,reserved-heap = <0x0 0x3000 0x0 0x4000>;
+...
+};
+
+RAM at 0x3000 of 1G size will be reserved as heap memory. Later,
+heap allocator will allocate memory only from this specific region.
This section is quite confusing. I think we need to clearly differentiate heap
vs
allocator.
In Xen we have two heaps:
1) Xen heap: It is always mapped in Xen virtual address space. This is
mainly used for xen internal allocation.
2) Domain heap: It may not always be mapped in Xen virtual address space.
This is mainly used for domain memory and mapped on-demand.
For Arm64 (and x86), two heaps are allocated from the same region. But on
Arm32, they are different.
We also have two allocator:
1) Boot allocator: This is used during boot only. There is no concept of
heap at this time.
2) Buddy allocator: This is the current runtime allocator. This can either
allocator from either heap.
AFAICT, this design is introducing a 3rd allocator that will return domain heap
pages.
Now, back to this section, are you saying you
RE: [PATCH] Xen: Design doc for 1:1 direct-map and static allocation
Hi Julien
> -Original Message-
> From: Julien Grall
> Sent: Tuesday, May 18, 2021 7:48 PM
> To: Penny Zheng ; xen-devel@lists.xenproject.org;
> sstabell...@kernel.org
> Cc: Bertrand Marquis ; Wei Chen
> ; nd
> Subject: Re: [PATCH] Xen: Design doc for 1:1 direct-map and static allocation
>
> Hi Penny,
>
> On 18/05/2021 06:07, Penny Zheng wrote:
> > Create one design doc for 1:1 direct-map and static allocation.
> > It is the first draft and aims to describe why and how we allocate
> > 1:1 direct-map(guest physical == physical) domains, and why and how we
> > let domains on static allocation.
> >
> > Signed-off-by: Penny Zheng
> > ---
> > docs/designs/static_alloc_and_direct_map.md | 239
>
> > 1 file changed, 239 insertions(+)
> > create mode 100644 docs/designs/static_alloc_and_direct_map.md
> >
> > diff --git a/docs/designs/static_alloc_and_direct_map.md
> > b/docs/designs/static_alloc_and_direct_map.md
> > new file mode 100644
> > index 00..fdda162188
> > --- /dev/null
> > +++ b/docs/designs/static_alloc_and_direct_map.md
> > @@ -0,0 +1,239 @@
> > +# Preface
> > +
> > +The document is an early draft for 1:1 direct-map memory map (`guest
> > +physical == physical`) of domUs and Static Allocation.
> > +Since the implementation of these two features shares a lot, we would
> > +like to introduce both in one design.
> > +
> > +Right now, these two features are limited to ARM architecture.
> > +
> > +This design aims to describe why and how the guest would be created
> > +as 1:1 direct-map domain, and why and what the static allocation is.
> > +
> > +This document is partly based on Stefano Stabellini's patch serie v1:
> > +[direct-map DomUs](
> > +https://lists.xenproject.org/archives/html/xen-devel/2020-
> 04/msg00707.html).
>
> While for the reviewer this is a useful information to have, I am not sure a
> future reader needs to know all the history. So I would move this to the
> commit message.
>
> > +
> > +This is a first draft and some questions are still unanswered. When
> > +this is the case, it will be included under chapter `DISCUSSION`.
> > +
> > +# Introduction on Static Allocation
> > +
> > +Static allocation refers to system or sub-system(domains) for which
> > +memory areas are pre-defined by configuration using physical address
> ranges.
> > +
> > +## Background
> > +
> > +Cases where needs static allocation:
> > +
> > + * Static allocation needed whenever a system has a pre-defined
> > +non-changing behaviour. This is usually the case in safety world
> > +where system must behave the same upon reboot, so memory resource
> for
> > +both XEN and domains should be static and pre-defined.
> > +
> > + * Static allocation needed whenever a guest wants to allocate
> > +memory from refined memory ranges. For example, a system has one
> > +high-speed RAM region, and would like to assign it to one specific domain.
> > +
> > + * Static allocation needed whenever a system needs a guest
> > +restricted to some known memory area due to hardware limitations
> > +reason. For example, some device can only do DMA to a specific part of
> the memory.
> > +
> > +Limitations:
> > + * There is no consideration for PV devices at the moment.
> > +
> > +## Design on Static Allocation
> > +
> > +Static allocation refers to system or sub-system(domains) for which
> > +memory areas are pre-defined by configuration using physical address
> ranges.
> > +
> > +These pre-defined memory, -- Static Momery, as parts of RAM reserved
> > +in the
>
> s/Momery/Memory/
>
Thx.
> > +beginning, shall never go to heap allocator or boot allocator for any use.
>
> I think you mean "buddy" rather than "heap". Looking at your code, you are
> treating static memory region as domheap pages.
>
> > +
> > +### Static Allocation for Domains
> > +
> > +### New Deivce Tree Node: `xen,static_mem`
>
> S/Deivce/
>
Thx.
> > +
> > +Here introduces new `xen,static_mem` node to define static memory
> > +nodes for one specific domain.
> > +
> > +For domains on static allocation, users need to pre-define guest RAM
> > +regions in configuration, through `xen,static_mem` node under approriate
> `domUx` node.
> > +
> > +Here is one example:
> > +
> > +
> > +domU1 {
> > +compatible = "xen,domain";
> > +#address-cells = <0x
Re: [PATCH] Xen: Design doc for 1:1 direct-map and static allocation
Hi Penny,
On 18/05/2021 06:07, Penny Zheng wrote:
Create one design doc for 1:1 direct-map and static allocation.
It is the first draft and aims to describe why and how we allocate
1:1 direct-map(guest physical == physical) domains, and why and how
we let domains on static allocation.
Signed-off-by: Penny Zheng
---
docs/designs/static_alloc_and_direct_map.md | 239
1 file changed, 239 insertions(+)
create mode 100644 docs/designs/static_alloc_and_direct_map.md
diff --git a/docs/designs/static_alloc_and_direct_map.md
b/docs/designs/static_alloc_and_direct_map.md
new file mode 100644
index 00..fdda162188
--- /dev/null
+++ b/docs/designs/static_alloc_and_direct_map.md
@@ -0,0 +1,239 @@
+# Preface
+
+The document is an early draft for 1:1 direct-map memory map
+(`guest physical == physical`) of domUs and Static Allocation.
+Since the implementation of these two features shares a lot, we would like
+to introduce both in one design.
+
+Right now, these two features are limited to ARM architecture.
+
+This design aims to describe why and how the guest would be created as 1:1
+direct-map domain, and why and what the static allocation is.
+
+This document is partly based on Stefano Stabellini's patch serie v1:
+[direct-map DomUs](
+https://lists.xenproject.org/archives/html/xen-devel/2020-04/msg00707.html).
While for the reviewer this is a useful information to have, I am not
sure a future reader needs to know all the history. So I would move this
to the commit message.
+
+This is a first draft and some questions are still unanswered. When this is
+the case, it will be included under chapter `DISCUSSION`.
+
+# Introduction on Static Allocation
+
+Static allocation refers to system or sub-system(domains) for which memory
+areas are pre-defined by configuration using physical address ranges.
+
+## Background
+
+Cases where needs static allocation:
+
+ * Static allocation needed whenever a system has a pre-defined non-changing
+behaviour. This is usually the case in safety world where system must behave
+the same upon reboot, so memory resource for both XEN and domains should be
+static and pre-defined.
+
+ * Static allocation needed whenever a guest wants to allocate memory
+from refined memory ranges. For example, a system has one high-speed RAM
+region, and would like to assign it to one specific domain.
+
+ * Static allocation needed whenever a system needs a guest restricted to some
+known memory area due to hardware limitations reason. For example, some device
+can only do DMA to a specific part of the memory.
+
+Limitations:
+ * There is no consideration for PV devices at the moment.
+
+## Design on Static Allocation
+
+Static allocation refers to system or sub-system(domains) for which memory
+areas are pre-defined by configuration using physical address ranges.
+
+These pre-defined memory, -- Static Momery, as parts of RAM reserved in the
s/Momery/Memory/
+beginning, shall never go to heap allocator or boot allocator for any use.
I think you mean "buddy" rather than "heap". Looking at your code, you
are treating static memory region as domheap pages.
+
+### Static Allocation for Domains
+
+### New Deivce Tree Node: `xen,static_mem`
S/Deivce/
+
+Here introduces new `xen,static_mem` node to define static memory nodes for
+one specific domain.
+
+For domains on static allocation, users need to pre-define guest RAM regions in
+configuration, through `xen,static_mem` node under approriate `domUx` node.
+
+Here is one example:
+
+
+domU1 {
+compatible = "xen,domain";
+#address-cells = <0x2>;
+#size-cells = <0x2>;
+cpus = <2>;
+xen,static-mem = <0x0 0xa000 0x0 0x2000>;
+...
+};
+
+RAM at 0xa000 of 512 MB are static memory reserved for domU1 as its RAM.
+
+### New Page Flag: `PGC_reserved`
+
+In order to differentiate and manage pages reserved as static memory with
+those which are allocated from heap allocator for normal domains, we shall
+introduce a new page flag `PGC_reserved` to tell.
+
+Grant pages `PGC_reserved` when initializing static memory.
+
+### New linked page list: `reserved_page_list` in `struct domain`
+
+Right now, for normal domains, on assigning pages to domain, pages allocated
+from heap allocator as guest RAM shall be inserted to one linked page
+list `page_list` for later managing and storing.
+
+So in order to tell, pages allocated from static memory, shall be inserted
+to a different linked page list `reserved_page_list`.
You already have the flag ``PGC_reserved`` to indicate whether the
memory is reserved or not. So why do you also need to link list it?
+
+Later, when domain get destroyed and memory relinquished, only pages in
+`page_list` go back to heap, and pages in `reserved_page_list` shall not.
While going through the series, I could not find any code implementing
this. However, this is not enough to prevent a
Re: [PATCH] Xen: Design doc for 1:1 direct-map and static allocation
On 18.05.2021 07:07, Penny Zheng wrote:
> +## Background
> +
> +Cases where needs static allocation:
> +
> + * Static allocation needed whenever a system has a pre-defined non-changing
> +behaviour. This is usually the case in safety world where system must behave
> +the same upon reboot, so memory resource for both XEN and domains should be
> +static and pre-defined.
> +
> + * Static allocation needed whenever a guest wants to allocate memory
> +from refined memory ranges. For example, a system has one high-speed RAM
> +region, and would like to assign it to one specific domain.
> +
> + * Static allocation needed whenever a system needs a guest restricted to
> some
> +known memory area due to hardware limitations reason. For example, some
> device
> +can only do DMA to a specific part of the memory.
This isn't a reason for fully static partitioning. Such devices also exist
in the x86 world, without there having been a need to statically partition
systems. All you want to guarantee is that for I/O purposes a domain has
_some_ memory in the accessible range.
> +Limitations:
> + * There is no consideration for PV devices at the moment.
How would PV devices be affected? Drivers better would use grant
transfers, but that's about it afaics.
> +## Design on Static Allocation
> +
> +Static allocation refers to system or sub-system(domains) for which memory
> +areas are pre-defined by configuration using physical address ranges.
> +
> +These pre-defined memory, -- Static Momery, as parts of RAM reserved in the
> +beginning, shall never go to heap allocator or boot allocator for any use.
> +
> +### Static Allocation for Domains
> +
> +### New Deivce Tree Node: `xen,static_mem`
> +
> +Here introduces new `xen,static_mem` node to define static memory nodes for
> +one specific domain.
> +
> +For domains on static allocation, users need to pre-define guest RAM regions
> in
> +configuration, through `xen,static_mem` node under approriate `domUx` node.
> +
> +Here is one example:
> +
> +
> +domU1 {
> +compatible = "xen,domain";
> +#address-cells = <0x2>;
> +#size-cells = <0x2>;
> +cpus = <2>;
> +xen,static-mem = <0x0 0xa000 0x0 0x2000>;
> +...
> +};
> +
> +RAM at 0xa000 of 512 MB are static memory reserved for domU1 as its RAM.
> +
> +### New Page Flag: `PGC_reserved`
> +
> +In order to differentiate and manage pages reserved as static memory with
> +those which are allocated from heap allocator for normal domains, we shall
> +introduce a new page flag `PGC_reserved` to tell.
This contradicts you saying higher up "shall never go to heap allocator
or boot allocator for any use" - no such flag ought to be needed of the
allocators never get to see these pages. And even if such a flag was
needed, I can't see how it would be sufficient to express the page ->
domain relationship.
> +Grant pages `PGC_reserved` when initializing static memory.
I'm afraid I don't understand this sentence at all.
> +### New linked page list: `reserved_page_list` in `struct domain`
> +
> +Right now, for normal domains, on assigning pages to domain, pages allocated
> +from heap allocator as guest RAM shall be inserted to one linked page
> +list `page_list` for later managing and storing.
> +
> +So in order to tell, pages allocated from static memory, shall be inserted
> +to a different linked page list `reserved_page_list`.
> +
> +Later, when domain get destroyed and memory relinquished, only pages in
> +`page_list` go back to heap, and pages in `reserved_page_list` shall not.
If such a domain can be destroyed (and re-created), how would the
association between memory and intended owner be retained / propagated?
Where else would the pages from reserved_page_list go (they need to go
somewhere, as the struct domain instance will go away)?
> +### Memory Allocation for Domains on Static Allocation
> +
> +RAM regions pre-defined as static memory for one specifc domain shall be
> parsed
> +and reserved from the beginning. And they shall never go to any memory
> +allocator for any use.
> +
> +Later when allocating static memory for this specific domain, after acquiring
> +those reserved regions, users need to a do set of verification before
> +assigning.
> +For each page there, it at least includes the following steps:
> +1. Check if it is in free state and has zero reference count.
> +2. Check if the page is reserved(`PGC_reserved`).
If this memory is reserved for a specific domain, why is such verification
necessary?
> +Then, assigning these pages to this specific domain, and all pages go to one
> +new linked page list `reserved_page_list`.
> +
> +At last, set up guest P2M mapping. By default, it shall be mapped to the
> fixed
> +guest RAM address `GUEST_RAM0_BASE`, `GUEST_RAM1_BASE`, just like normal
> +domains. But later in 1:1 direct-map design, if `direct-map` is set, the
> guest
> +physical address will equal to physical
[PATCH] Xen: Design doc for 1:1 direct-map and static allocation
Create one design doc for 1:1 direct-map and static allocation.
It is the first draft and aims to describe why and how we allocate
1:1 direct-map(guest physical == physical) domains, and why and how
we let domains on static allocation.
Signed-off-by: Penny Zheng
---
docs/designs/static_alloc_and_direct_map.md | 239
1 file changed, 239 insertions(+)
create mode 100644 docs/designs/static_alloc_and_direct_map.md
diff --git a/docs/designs/static_alloc_and_direct_map.md
b/docs/designs/static_alloc_and_direct_map.md
new file mode 100644
index 00..fdda162188
--- /dev/null
+++ b/docs/designs/static_alloc_and_direct_map.md
@@ -0,0 +1,239 @@
+# Preface
+
+The document is an early draft for 1:1 direct-map memory map
+(`guest physical == physical`) of domUs and Static Allocation.
+Since the implementation of these two features shares a lot, we would like
+to introduce both in one design.
+
+Right now, these two features are limited to ARM architecture.
+
+This design aims to describe why and how the guest would be created as 1:1
+direct-map domain, and why and what the static allocation is.
+
+This document is partly based on Stefano Stabellini's patch serie v1:
+[direct-map DomUs](
+https://lists.xenproject.org/archives/html/xen-devel/2020-04/msg00707.html).
+
+This is a first draft and some questions are still unanswered. When this is
+the case, it will be included under chapter `DISCUSSION`.
+
+# Introduction on Static Allocation
+
+Static allocation refers to system or sub-system(domains) for which memory
+areas are pre-defined by configuration using physical address ranges.
+
+## Background
+
+Cases where needs static allocation:
+
+ * Static allocation needed whenever a system has a pre-defined non-changing
+behaviour. This is usually the case in safety world where system must behave
+the same upon reboot, so memory resource for both XEN and domains should be
+static and pre-defined.
+
+ * Static allocation needed whenever a guest wants to allocate memory
+from refined memory ranges. For example, a system has one high-speed RAM
+region, and would like to assign it to one specific domain.
+
+ * Static allocation needed whenever a system needs a guest restricted to some
+known memory area due to hardware limitations reason. For example, some device
+can only do DMA to a specific part of the memory.
+
+Limitations:
+ * There is no consideration for PV devices at the moment.
+
+## Design on Static Allocation
+
+Static allocation refers to system or sub-system(domains) for which memory
+areas are pre-defined by configuration using physical address ranges.
+
+These pre-defined memory, -- Static Momery, as parts of RAM reserved in the
+beginning, shall never go to heap allocator or boot allocator for any use.
+
+### Static Allocation for Domains
+
+### New Deivce Tree Node: `xen,static_mem`
+
+Here introduces new `xen,static_mem` node to define static memory nodes for
+one specific domain.
+
+For domains on static allocation, users need to pre-define guest RAM regions in
+configuration, through `xen,static_mem` node under approriate `domUx` node.
+
+Here is one example:
+
+
+domU1 {
+compatible = "xen,domain";
+#address-cells = <0x2>;
+#size-cells = <0x2>;
+cpus = <2>;
+xen,static-mem = <0x0 0xa000 0x0 0x2000>;
+...
+};
+
+RAM at 0xa000 of 512 MB are static memory reserved for domU1 as its RAM.
+
+### New Page Flag: `PGC_reserved`
+
+In order to differentiate and manage pages reserved as static memory with
+those which are allocated from heap allocator for normal domains, we shall
+introduce a new page flag `PGC_reserved` to tell.
+
+Grant pages `PGC_reserved` when initializing static memory.
+
+### New linked page list: `reserved_page_list` in `struct domain`
+
+Right now, for normal domains, on assigning pages to domain, pages allocated
+from heap allocator as guest RAM shall be inserted to one linked page
+list `page_list` for later managing and storing.
+
+So in order to tell, pages allocated from static memory, shall be inserted
+to a different linked page list `reserved_page_list`.
+
+Later, when domain get destroyed and memory relinquished, only pages in
+`page_list` go back to heap, and pages in `reserved_page_list` shall not.
+
+### Memory Allocation for Domains on Static Allocation
+
+RAM regions pre-defined as static memory for one specifc domain shall be parsed
+and reserved from the beginning. And they shall never go to any memory
+allocator for any use.
+
+Later when allocating static memory for this specific domain, after acquiring
+those reserved regions, users need to a do set of verification before
+assigning.
+For each page there, it at least includes the following steps:
+1. Check if it is in free state and has zero reference count.
+2. Check if the page is reserved(`PGC_reserved`).
+
+Then, assigning these pages to this specific domain, and all pages go to
