Re: [Xen-devel] [ARM] Handling CMA pool device nodes in Dom0

[Peng Fan]

On Tue, Nov 29, 2016 at 01:49:51PM +, Julien Grall wrote:
>(CC Stefano)
>
>On 25/11/16 12:19, Iurii Mykhalskyi wrote:
>>Hello!
>
>Hi Iurii,
>
>>
>>I'm working under Renesas Gen3 H3 board with 4GB RAM (Salvator-X)
>>support in Xen mainline.
>>
>>Salvator-X has several  CMA pool nodes, for example:
>>
>>1:
>>adsp_reserved: linux,adsp {
>>compatible = "shared-dma-pool";
>>reusable;
>>reg = <0x 0x5700 0x0 0x0100>;
>>};
>>
>>2:
>>linux,cma {
>>compatible = "shared-dma-pool";
>>reusable;
>>reg = <0x 0x5800 0x0 0x1800>;
>>linux,cma-default;
>>};
>>
>>During Dom0 allocation, we can't guarantee, that allocated memory will
>>contain mentioned regions.
>>In second сase, we can actually hardcode mapped region by using separate
>>DTS for Dom0 with changed memory regions.
>>But for first one, this in not an option - this pool is used for audio
>>DSP and its firmware relies on this addresses.
>>
>>What is the correct way to solve this situation?
>>Does Xen has some mechanism to handle such cases?

How about using "alloc-ranges"?

Regards,
Peng.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Big.LITTLE support (WAS Re: Xen ARM community call)

[Peng Fan]

Hi Julien,
On Tue, Nov 22, 2016 at 02:28:39PM +, Julien Grall wrote:
>Hello Anastassios,
>
>On 09/11/16 22:50, Anastassios Nanos wrote:
>>Hi Julien, all,
>>
>>>I would like to start organizing a recurring community call to discuss and
>>>sync-up on upcoming features for Xen ARM.
>>
>>great idea, I'd like to join too (GMT).
>>
>>>Example of features that could be discussed:
>>>- Sharing co-processor between guests
>>>- PCI passthrough
>>
>>Another issue to discuss, at some point, could be big.LITTLE support
>>(based on 
>>https://lists.xenproject.org/archives/html/xen-devel/2016-09/msg01802.html).
>
>Good point. Looking at the discussion, many ideas was suggested and I don't
>remember whether we all agreed on what to do. I would recommend to summarize
>the discussion in a mail so we can come up with an agreement.
>
>Peng, would you be up to do this?

Sorry for late reply. I am a bit busy on other things. I appreciate if you or 
anyone else
can help summarize the ideas. I may come to this big.little stuff at the 
beginning
of Dec.

Thanks,
Peng.

>
>Regards,
>
>-- 
>Julien Grall
>
>___
>Xen-devel mailing list
>Xen-devel@lists.xen.org
>https://lists.xen.org/xen-devel

-- 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH V4] xen/arm: domain_build: allocate lowmem for dom0 as much as possible

[Peng Fan]

On Thu, Nov 10, 2016 at 01:01:38PM +, Julien Grall wrote:
>(CC Wei as release manager)
>
>On 10/11/16 08:30, Peng Fan wrote:
>>Hi Julien,
>
>Hi Peng,
>
>>On Tue, Nov 01, 2016 at 02:42:06PM +, Julien Grall wrote:
>>>Hi Peng,
>>>
>>>Sorry for the late answer.
>>>
>>>On 23/09/2016 03:55, Peng Fan wrote:
>>>>On AArch64 SoCs, some IPs may only have the capability to access
>>>>32 bits address space. The physical memory assigned for Dom0 maybe
>>>>not in 4GB address space, then the IPs will not work properly.
>>>>So need to allocate memory under 4GB for Dom0.
>>>>
>>>>There is no restriction that how much lowmem needs to be allocated for
>>>>Dom0 ,so allocate lowmem as much as possible for Dom0.
>>>>
>>>>This patch does not affect 32-bit domain, because Variable "lowmem" is
>>>>set to true at the beginning. If failed to allocate bank0 under 4GB,
>>>>need to panic for 32-bit domain, because 32-bit domain requires bank0
>>>>be allocated under 4GB.
>>>>
>>>>For 64-bit domain, set "lowmem" to false, and continue allocating
>>>>memory from above 4GB.
>>>>
>>>>Signed-off-by: Peng Fan <peng@nxp.com>
>>>>Cc: Stefano Stabellini <sstabell...@kernel.org>
>>>>Cc: Julien Grall <julien.gr...@arm.com>
>>>
>>>Reviewed-by: Julien Grall <julien.gr...@arm.com>
>>>
>>>I am undecided whether this should be considered as a bug fix for Xen 4.8.
>>>Are you aware of any ARM64 platform we currently support requiring allocation
>>>of memory below 4GB?
>>
>>I have no idea about this (:, but I think this is a bug fix. Alought current
>>supported platforms works well, users may choose 4.8 to support their
>>new platform which has the limitation to access 64bit address.
>
>We are already late in the release process (rc5) for Xen 4.8. So we need to
>be careful when including a bug fix and evaluate the pros and cons.
>
>This patch is modifying the DOM0 memory layout for all 64-bit platforms. So
>it could potentially break one of the platform we  officially support (see
>[1] for a non-exhaustive list). We don't have a test suite running
>automatically for ARM64 at the moment (it is been working on), this means
>that manual testing needs to be done. I am not aware of any platform, in the
>list we supports, having this issue so I prefer to stay on the safe side and
>defer this patch for Xen 4.9.

Ok. Defer it for 4.9 to avoid breaking any platforms. :)

>
>If a user cares about Xen 4.8 for their platforms, then they could request
>the patch to be backported in Xen 4.8 after the release and after extensive
>testing in staging.

Yeah. Agree

Thanks,
Peng.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH V4] xen/arm: domain_build: allocate lowmem for dom0 as much as possible

[Peng Fan]

Hi Julien,

Sorry for late reply.

On Tue, Nov 01, 2016 at 02:42:06PM +, Julien Grall wrote:
>Hi Peng,
>
>Sorry for the late answer.
>
>On 23/09/2016 03:55, Peng Fan wrote:
>>On AArch64 SoCs, some IPs may only have the capability to access
>>32 bits address space. The physical memory assigned for Dom0 maybe
>>not in 4GB address space, then the IPs will not work properly.
>>So need to allocate memory under 4GB for Dom0.
>>
>>There is no restriction that how much lowmem needs to be allocated for
>>Dom0 ,so allocate lowmem as much as possible for Dom0.
>>
>>This patch does not affect 32-bit domain, because Variable "lowmem" is
>>set to true at the beginning. If failed to allocate bank0 under 4GB,
>>need to panic for 32-bit domain, because 32-bit domain requires bank0
>>be allocated under 4GB.
>>
>>For 64-bit domain, set "lowmem" to false, and continue allocating
>>memory from above 4GB.
>>
>>Signed-off-by: Peng Fan <peng@nxp.com>
>>Cc: Stefano Stabellini <sstabell...@kernel.org>
>>Cc: Julien Grall <julien.gr...@arm.com>
>
>Reviewed-by: Julien Grall <julien.gr...@arm.com>
>
>I am undecided whether this should be considered as a bug fix for Xen 4.8.
>Are you aware of any ARM64 platform we currently support requiring allocation
>of memory below 4GB?

I have no idea about this (:, but I think this is a bug fix. Alought current
supported platforms works well, users may choose 4.8 to support their
new platform which has the limitation to access 64bit address.

Regards,
Peng.

>
>Regards,
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH V4] xen/arm: domain_build: allocate lowmem for dom0 as much as possible

[Peng Fan]

Hi Stefano, Julien

Any comments on this v4 patch?

Thanks,
Peng
On Fri, Sep 23, 2016 at 10:55:34AM +0800, Peng Fan wrote:
>On AArch64 SoCs, some IPs may only have the capability to access
>32 bits address space. The physical memory assigned for Dom0 maybe
>not in 4GB address space, then the IPs will not work properly.
>So need to allocate memory under 4GB for Dom0.
>
>There is no restriction that how much lowmem needs to be allocated for
>Dom0 ,so allocate lowmem as much as possible for Dom0.
>
>This patch does not affect 32-bit domain, because Variable "lowmem" is
>set to true at the beginning. If failed to allocate bank0 under 4GB,
>need to panic for 32-bit domain, because 32-bit domain requires bank0
>be allocated under 4GB.
>
>For 64-bit domain, set "lowmem" to false, and continue allocating
>memory from above 4GB.
>
>Signed-off-by: Peng Fan <peng@nxp.com>
>Cc: Stefano Stabellini <sstabell...@kernel.org>
>Cc: Julien Grall <julien.gr...@arm.com>
>---
>
>This patch is to resolve the issue mentioned in
>https://lists.xen.org/archives/html/xen-devel/2016-09/msg00235.html
>
> Tested results:
> (XEN) Allocating 1:1 mappings totalling 2048MB for dom0:
> (XEN) BANK[0] 0x008800-0x00f800 (1792MB)
> (XEN) BANK[1] 0x09e000-0x09f000 (256MB)
> 1792M allocated in 4GB address space.
>
>V4:
> Address comments in V3: 
> https://lists.xen.org/archives/html/xen-devel/2016-09/msg02499.html
> Drop uneccessary check when failed to allocate memory under 4GB
> Refine comments according to Julien's suggestion in V3.
> Keep "bits <= (lowmem ? 32 : PADDR_BITS)", but not changed to "bits <= 32"
>
>V3:
> Add more commit log
> Add more comments
> Add back panic if failed to allocate bank0 under 4GB for 32-bit domain.
>
>V2:
> Remove the bootargs dom0_lowmem introduced in V1.
> Following 
> "https://lists.xenproject.org/archives/html/xen-devel/2016-09/msg01459.html;
> to allocate as much as possible lowmem.
>
> xen/arch/arm/domain_build.c | 33 ++---
> 1 file changed, 22 insertions(+), 11 deletions(-)
>
>diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>index 35ab08d..6b5ac8d 100644
>--- a/xen/arch/arm/domain_build.c
>+++ b/xen/arch/arm/domain_build.c
>@@ -195,9 +195,9 @@ fail:
>  *bank. Partly this is just easier for us to deal with, but also
>  *the ramdisk and DTB must be placed within a certain proximity of
>  *the kernel within RAM.
>- * 3. For 32-bit dom0 we want to place as much of the RAM as we
>- *reasonably can below 4GB, so that it can be used by non-LPAE
>- *enabled kernels.
>+ * 3. For dom0 we want to place as much of the RAM as we reasonably can
>+ *below 4GB, so that it can be used by non-LPAE enabled kernels (32-bit)
>+ *or when a device assigned to dom0 can only do 32-bit DMA access.
>  * 4. For 32-bit dom0 the kernel must be located below 4GB.
>  * 5. We want to have a few largers banks rather than many smaller ones.
>  *
>@@ -230,7 +230,8 @@ fail:
>  * we give up.
>  *
>  * For 32-bit domain we require that the initial allocation for the
>- * first bank is under 4G. Then for the subsequent allocations we
>+ * first bank is under 4G. For 64-bit domain, the first bank is preferred
>+ * to be allocated under 4G. Then for the subsequent allocations we
>  * initially allocate memory only from below 4GB. Once that runs out
>  * (as described above) we allow higher allocations and continue until
>  * that runs out (or we have allocated sufficient dom0 memory).
>@@ -244,7 +245,7 @@ static void allocate_memory(struct domain *d, struct 
>kernel_info *kinfo)
> unsigned int order = get_11_allocation_size(kinfo->unassigned_mem);
> int i;
> 
>-bool_t lowmem = is_32bit_domain(d);
>+bool_t lowmem = true;
> unsigned int bits;
> 
> /*
>@@ -269,20 +270,30 @@ static void allocate_memory(struct domain *d, struct 
>kernel_info *kinfo)
> {
> pg = alloc_domheap_pages(d, order, MEMF_bits(bits));
> if ( pg != NULL )
>+{
>+if ( !insert_11_bank(d, kinfo, pg, order) )
>+BUG(); /* Cannot fail for first bank */
>+
> goto got_bank0;
>+}
> }
> order--;
> }
> 
>-panic("Unable to allocate first memory bank");
>-
>- got_bank0:
>+/* Failed to allocate bank0 under 4GB */
>+if ( is_32bit_domain(d) )
>+panic("Unable to allocate first memory bank.");
> 
>-if ( !insert_11_bank(d, kinfo, pg, order) )
>-BUG(); /* Cannot fail for first bank 

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

On Fri, Sep 23, 2016 at 10:24:37AM +0100, Julien Grall wrote:
>Hello Peng,
>
>On 23/09/16 03:14, Peng Fan wrote:
>>On Thu, Sep 22, 2016 at 07:54:02PM +0100, Julien Grall wrote:
>>>Hi Stefano,
>>>
>>>On 22/09/2016 18:31, Stefano Stabellini wrote:
>>>>On Thu, 22 Sep 2016, Julien Grall wrote:
>>>>>Hello Peng,
>>>>>
>>>>>On 22/09/16 10:27, Peng Fan wrote:
>>>>>>On Thu, Sep 22, 2016 at 10:50:23AM +0200, Dario Faggioli wrote:
>>>>>>>On Thu, 2016-09-22 at 14:49 +0800, Peng Fan wrote:
>>>>>>>>On Wed, Sep 21, 2016 at 08:11:43PM +0100, Julien Grall wrote:
>>>>>>>A feature like `xl cpupool-biglittle-split' can still be interesting,
>>>>>>
>>>>>>"cpupool-cluster-split" maybe a better name?
>>>>>
>>>>>You seem to assume that a cluster, from the MPIDR point of view, can only
>>>>>contain the same set of CPUs. I don't think this is part of the 
>>>>>architecture,
>>>>>so this may not be true in the future.
>>>>
>>>>Interesting. I also understood that a cluster can only have one kind if
>>>>cpus. Honestly it would be a little insane for it to be otherwise :-)
>>>
>>>I don't think this is insane (or maybe I am insane :)). Cluster usually
>>>doesn't share all L2 cache (assuming L1 is local to each core) and L3 cache
>>>may not be present, so if you move a task from one cluster to another you
>>>will add latency because the new L2 cache has to be refilled.
>>>
>>>The use case of big.LITTLE is big cores are used for short period of burst
>>>and little core are used for the rest (e.g listening audio, fetching
>>>mail...). If you want to reduce latency when switch between big and little
>>>CPUs, you may want to put them within the same cluster.
>>>
>>>Also, as mentioned in another thread, you may have a platform with the same
>>>micro-architecture (e.g Cortex A-53) but different silicon implementation
>>>(e.g to have a different frequency, power efficiency). Here the concept of
>>>big.LITTLE is more blurred.
>>
>>That is possible that in one cluster, different pcpus runs with different cpu
>>frequency. This depends on hardware design. Some may require all the cores in
>>one cluster runs at the same frequency, some may have more complicated design 
>>that
>>supports different cores runs at different frequency.
>>
>>This is just like you have a smp system, but different cores can run at
>>different cpu frequency. I think this is not what bit.LITTLE means.
>
>big.LITTLE is a generic term to have "power hungry and powerful core
>powerful" (big) with slower and battery-saving cores (LITTLE).
>
>It is not mandatory to have different micro-architectures between big and
>LITTLE cores.
>
>In any case, the interface should not be big.LITTLE specific. We don't want
>to tie us to one specific architecture.

If all the cores have the same micro-architecture, but for some reason,
they are put in different clusters or cpus in one cluster support running
at different cpu freq.

We still can introduce cpupool-cluster-split or as Juergen suggested,
use "cpupool-slit feature=xx"  to split the cluster or cpuclasses
into different cpupools. This is just a feature that better to have, I think.

The reason to include cpupool-cluster-split or else is to split the big and 
little
cores into different cpupools. And now big and little cores are in different cpu
clusters from the hardware[1] I can see. I think assigning cores from
different clusters into one cpupool is not a good idea.

I have no idea about future hardware.


If cluster is not prefered, cpuclass maybe a choice, but I personally perfer
"cluster" split for ARM.

Thanks,
Peng.

[1] https://en.wikipedia.org/wiki/ARM_big.LITTLE

>
>Regards,
>
>-- 
>Julien Grall

-- 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [PATCH V4] xen/arm: domain_build: allocate lowmem for dom0 as much as possible

[Peng Fan]

On AArch64 SoCs, some IPs may only have the capability to access
32 bits address space. The physical memory assigned for Dom0 maybe
not in 4GB address space, then the IPs will not work properly.
So need to allocate memory under 4GB for Dom0.

There is no restriction that how much lowmem needs to be allocated for
Dom0 ,so allocate lowmem as much as possible for Dom0.

This patch does not affect 32-bit domain, because Variable "lowmem" is
set to true at the beginning. If failed to allocate bank0 under 4GB,
need to panic for 32-bit domain, because 32-bit domain requires bank0
be allocated under 4GB.

For 64-bit domain, set "lowmem" to false, and continue allocating
memory from above 4GB.

Signed-off-by: Peng Fan <peng@nxp.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
Cc: Julien Grall <julien.gr...@arm.com>
---

This patch is to resolve the issue mentioned in
https://lists.xen.org/archives/html/xen-devel/2016-09/msg00235.html

 Tested results:
 (XEN) Allocating 1:1 mappings totalling 2048MB for dom0:
 (XEN) BANK[0] 0x008800-0x00f800 (1792MB)
 (XEN) BANK[1] 0x09e000-0x09f000 (256MB)
 1792M allocated in 4GB address space.

V4:
 Address comments in V3: 
https://lists.xen.org/archives/html/xen-devel/2016-09/msg02499.html
 Drop uneccessary check when failed to allocate memory under 4GB
 Refine comments according to Julien's suggestion in V3.
 Keep "bits <= (lowmem ? 32 : PADDR_BITS)", but not changed to "bits <= 32"

V3:
 Add more commit log
 Add more comments
 Add back panic if failed to allocate bank0 under 4GB for 32-bit domain.

V2:
 Remove the bootargs dom0_lowmem introduced in V1.
 Following 
"https://lists.xenproject.org/archives/html/xen-devel/2016-09/msg01459.html;
 to allocate as much as possible lowmem.

 xen/arch/arm/domain_build.c | 33 ++---
 1 file changed, 22 insertions(+), 11 deletions(-)

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 35ab08d..6b5ac8d 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -195,9 +195,9 @@ fail:
  *bank. Partly this is just easier for us to deal with, but also
  *the ramdisk and DTB must be placed within a certain proximity of
  *the kernel within RAM.
- * 3. For 32-bit dom0 we want to place as much of the RAM as we
- *reasonably can below 4GB, so that it can be used by non-LPAE
- *enabled kernels.
+ * 3. For dom0 we want to place as much of the RAM as we reasonably can
+ *below 4GB, so that it can be used by non-LPAE enabled kernels (32-bit)
+ *or when a device assigned to dom0 can only do 32-bit DMA access.
  * 4. For 32-bit dom0 the kernel must be located below 4GB.
  * 5. We want to have a few largers banks rather than many smaller ones.
  *
@@ -230,7 +230,8 @@ fail:
  * we give up.
  *
  * For 32-bit domain we require that the initial allocation for the
- * first bank is under 4G. Then for the subsequent allocations we
+ * first bank is under 4G. For 64-bit domain, the first bank is preferred
+ * to be allocated under 4G. Then for the subsequent allocations we
  * initially allocate memory only from below 4GB. Once that runs out
  * (as described above) we allow higher allocations and continue until
  * that runs out (or we have allocated sufficient dom0 memory).
@@ -244,7 +245,7 @@ static void allocate_memory(struct domain *d, struct 
kernel_info *kinfo)
 unsigned int order = get_11_allocation_size(kinfo->unassigned_mem);
 int i;
 
-bool_t lowmem = is_32bit_domain(d);
+bool_t lowmem = true;
 unsigned int bits;
 
 /*
@@ -269,20 +270,30 @@ static void allocate_memory(struct domain *d, struct 
kernel_info *kinfo)
 {
 pg = alloc_domheap_pages(d, order, MEMF_bits(bits));
 if ( pg != NULL )
+{
+if ( !insert_11_bank(d, kinfo, pg, order) )
+BUG(); /* Cannot fail for first bank */
+
 goto got_bank0;
+}
 }
 order--;
 }
 
-panic("Unable to allocate first memory bank");
-
- got_bank0:
+/* Failed to allocate bank0 under 4GB */
+if ( is_32bit_domain(d) )
+panic("Unable to allocate first memory bank.");
 
-if ( !insert_11_bank(d, kinfo, pg, order) )
-BUG(); /* Cannot fail for first bank */
+/* Try to allocate memory from above 4GB */
+printk(XENLOG_INFO "No bank has been allocated below 4GB.\n");
+lowmem = false;
 
-/* Now allocate more memory and fill in additional banks */
+ got_bank0:
 
+/*
+ * If we failed to allocate bank0 under 4GB, continue allocating
+ * memory from above 4GB and fill in banks.
+ */
 order = get_11_allocation_size(kinfo->unassigned_mem);
 while ( kinfo->unassigned_mem && kinfo->mem.nr_banks < NR_MEM_BANKS )
 {
-- 
2.6.6


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

On Thu, Sep 22, 2016 at 12:21:00PM +0100, Julien Grall wrote:

According to George's comments,
Then, I think we could use affinity to restrict little vcpus be scheduled 
on little vcpus,
and restrict big vcpus on big vcpus. Seems no need to consider soft 
affinity, use hard
affinity is to handle this.

We may need to provide some interface to let xl can get the information 
such as
big.little or smp. if it is big.little, which is big and which is little.

For how to differentiate cpus, I am looking the linaro eas cpu topology 
code,
The code has not been upstreamed (:, but merged into google android kernel.
I only plan to take some necessary code, such as device tree parse and
cpu topology build, because we only need to know the computing capacity of 
each pcpu.

Some doc about eas piece, including dts node examples:
https://git.linaro.org/arm/eas/kernel.git/blob/refs/heads/lsk-v4.4-eas-v5.2:/Documentation/devicetree/bindings/scheduler/sched-energy-costs.txt
>>>
>>>I am reluctant to take any non-upstreamed bindings in Xen. There is a similar
>>>series going on the lklm [1].
>>
>>For how to differentiate cpu classes, how about directly use
>>compatible property of each cpu node?
>
>What do you mean by cpu classes? If it is power, then the compatible will not
>help here. You may have a platform with the same core (e.g cortex A53) but
>different silicon implementation, so the power efficiency will be different.

cpu classes, I mean cpu clusters. I checked the cpu capacity code[1] you listed,
it use dmips from dhystone. But now what I plan to implement to block vcpu
from being scheduled within big.LITTLE.

In my case, vcpu will be restricted in A53 or A72.

In the same cluster, different cores may run at different cpu freq or all the 
cores
run at the same freq. This depends on soc implementation.

This needs xen to choose which pcpu to run the vcpu, and no local migration 
scheduling vcpu on the cpus in one cluster.

Considering power for future, dmips needs to be used, but also need to 
differentiate
cpus from different clusters. So "dmips + compatible" both needs to be 
considered.

For cpus in one cluster, also need to take the dmips info for xen scheduling
vcpu between pcpu effectively.

Thanks,
Peng.

[1]https://lwn.net/Articles/699569/
>
>Regards,
>
>-- 
>Julien Grall

-- 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

On Thu, Sep 22, 2016 at 07:54:02PM +0100, Julien Grall wrote:
>Hi Stefano,
>
>On 22/09/2016 18:31, Stefano Stabellini wrote:
>>On Thu, 22 Sep 2016, Julien Grall wrote:
>>>Hello Peng,
>>>
>>>On 22/09/16 10:27, Peng Fan wrote:
>>>>On Thu, Sep 22, 2016 at 10:50:23AM +0200, Dario Faggioli wrote:
>>>>>On Thu, 2016-09-22 at 14:49 +0800, Peng Fan wrote:
>>>>>>On Wed, Sep 21, 2016 at 08:11:43PM +0100, Julien Grall wrote:
>>>>>A feature like `xl cpupool-biglittle-split' can still be interesting,
>>>>
>>>>"cpupool-cluster-split" maybe a better name?
>>>
>>>You seem to assume that a cluster, from the MPIDR point of view, can only
>>>contain the same set of CPUs. I don't think this is part of the architecture,
>>>so this may not be true in the future.
>>
>>Interesting. I also understood that a cluster can only have one kind if
>>cpus. Honestly it would be a little insane for it to be otherwise :-)
>
>I don't think this is insane (or maybe I am insane :)). Cluster usually
>doesn't share all L2 cache (assuming L1 is local to each core) and L3 cache
>may not be present, so if you move a task from one cluster to another you
>will add latency because the new L2 cache has to be refilled.
>
>The use case of big.LITTLE is big cores are used for short period of burst
>and little core are used for the rest (e.g listening audio, fetching
>mail...). If you want to reduce latency when switch between big and little
>CPUs, you may want to put them within the same cluster.
>
>Also, as mentioned in another thread, you may have a platform with the same
>micro-architecture (e.g Cortex A-53) but different silicon implementation
>(e.g to have a different frequency, power efficiency). Here the concept of
>big.LITTLE is more blurred.

That is possible that in one cluster, different pcpus runs with different cpu
frequency. This depends on hardware design. Some may require all the cores in
one cluster runs at the same frequency, some may have more complicated design 
that
supports different cores runs at different frequency.

This is just like you have a smp system, but different cores can run at
different cpu frequency. I think this is not what bit.LITTLE means.

For the pcpus in one cluster, xen needs to choose which pcpu for vcpu
for power or etc.


Thanks,
Peng.

>
>That's why I am quite reluctant to name (even if it may be more handy to the
>user) "big" and "little" the different CPU set.
>
>Cheers,
>
>-- 
>Julien Grall

-- 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

On Thu, Sep 22, 2016 at 12:29:53PM +0100, Julien Grall wrote:
>Hello Peng,
>
>On 22/09/16 10:27, Peng Fan wrote:
>>On Thu, Sep 22, 2016 at 10:50:23AM +0200, Dario Faggioli wrote:
>>>On Thu, 2016-09-22 at 14:49 +0800, Peng Fan wrote:
>>>>On Wed, Sep 21, 2016 at 08:11:43PM +0100, Julien Grall wrote:
>>>A feature like `xl cpupool-biglittle-split' can still be interesting,
>>
>>"cpupool-cluster-split" maybe a better name?
>
>You seem to assume that a cluster, from the MPIDR point of view, can only
>contain the same set of CPUs. I don't think this is part of the architecture,
>so this may not be true in the future.
>
>>
>>>completely orthogonally and independently from the affinity based work,
>>>and this series looks like it can be used to implement that. :-)
>>
>>Agree. All pcpus default can be assigned into cpupool0 based on the affinity 
>>work.
>
>What do you mean by affinity? From MPIDR?

vcpu hard affinity. When allocate or initialize vcpu, the hard affinity needs
to be inititialized.

Thanks,
Peng.

>
>>We could add one like "cpupool-numa-split" to split different classes cpu
>>into different cpupools.
>
>Regards,
>
>-- 
>Julien Grall

-- 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH V3] xen/arm: domain_build: allocate lowmem for dom0 as much as possible

[Peng Fan]

On Thu, Sep 22, 2016 at 04:23:05PM +0100, Julien Grall wrote:
>Hello Peng,
>
>On 22/09/16 10:16, Peng Fan wrote:
>>On AArch64 SoCs, some IPs may only have the capability to access
>>32 bits address space. The physical memory assigned for Dom0 maybe
>>not in 4GB address space, then the IPs will not work properly.
>>So need to allocate memory under 4GB for Dom0.
>>
>>There is no restriction that how much lowmem needs to be allocated for
>>Dom0. Dom0 now use 1:1 mapping, but DomU does not use 1:1 mapping,
>>there is no need to reserve lowmem for DomU, so allocate lowmem as much
>>as possible for Dom0.
>
>The last sentence is confusing. I would avoid to mention domU use case here.

Drop it in V4.

>
>>
>>This patch does not affect 32-bit domain, because Variable "lowmem" is
>>set to true at the beginning. If failed to allocate bank0 under 4GB,
>>need to panic for 32-bit domain, because 32-bit domain requires bank0
>>be allocated under 4GB.
>>
>>For 64-bit domain, set "lowmem" to false, and continue allocating
>>memory from higher memory space.
>>
>>Signed-off-by: Peng Fan <peng@nxp.com>
>>Cc: Stefano Stabellini <sstabell...@kernel.org>
>>Cc: Julien Grall <julien.gr...@arm.com>
>>---
>>
>>This patch is to resolve the issue mentioned in
>>https://lists.xen.org/archives/html/xen-devel/2016-09/msg00235.html
>>
>>V3:
>> Add more commit log
>> Add more comments
>> Add back panic if failed to allocate bank0 under 4GB for 32-bit domain.
>>
>>V2:
>> Remove the bootargs dom0_lowmem introduced in V1.
>> Following 
>> "https://lists.xenproject.org/archives/html/xen-devel/2016-09/msg01459.html;
>> to allocate as much as possible lowmem.
>>
>> Tested results:
>> (XEN) Allocating 1:1 mappings totalling 2048MB for dom0:
>> (XEN) BANK[0] 0x008800-0x00f800 (1792MB)
>> (XEN) BANK[1] 0x09e000-0x09f000 (256MB)
>> 1792M allocated in 4GB address space.
>>
>> xen/arch/arm/domain_build.c | 44 +++-
>> 1 file changed, 31 insertions(+), 13 deletions(-)
>>
>>diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>>index 35ab08d..a9c37c4 100644
>>--- a/xen/arch/arm/domain_build.c
>>+++ b/xen/arch/arm/domain_build.c
>>@@ -195,9 +195,10 @@ fail:
>>  *bank. Partly this is just easier for us to deal with, but also
>>  *the ramdisk and DTB must be placed within a certain proximity of
>>  *the kernel within RAM.
>>- * 3. For 32-bit dom0 we want to place as much of the RAM as we
>>- *reasonably can below 4GB, so that it can be used by non-LPAE
>>- *enabled kernels.
>>+ * 3. For dom0 we want to place as much of the RAM as we reasonably can
>>+ *below 4GB, so that the devices have the limitation to access 64 bits
>>+ *address space and work properly and it can be used by non-LPAE enabled
>>+ *kernels.
>
>I would say: "For dom0 we want to place as much of the RAM as we reasonably
>can below 4GB, so that it can be used by non-LPAE enabled kernels (32-bit) or
>when a device assigned to dom0 can only do 32-bit DMA access".

Thanks. Fix in V4.

>
>>  * 4. For 32-bit dom0 the kernel must be located below 4GB.
>>  * 5. We want to have a few largers banks rather than many smaller ones.
>>  *
>>@@ -230,7 +231,8 @@ fail:
>>  * we give up.
>>  *
>>  * For 32-bit domain we require that the initial allocation for the
>>- * first bank is under 4G. Then for the subsequent allocations we
>>+ * first bank is under 4G. For 64-bit domain, the first bank is preferred
>>+ * to be allocated under 4G. Then for the subsequent allocations we
>>  * initially allocate memory only from below 4GB. Once that runs out
>>  * (as described above) we allow higher allocations and continue until
>>  * that runs out (or we have allocated sufficient dom0 memory).
>>@@ -240,11 +242,11 @@ static void allocate_memory(struct domain *d, struct 
>>kernel_info *kinfo)
>> const unsigned int min_low_order =
>> get_order_from_bytes(min_t(paddr_t, dom0_mem, MB(128)));
>> const unsigned int min_order = get_order_from_bytes(MB(4));
>>-struct page_info *pg;
>>+struct page_info *pg = NULL;
>> unsigned int order = get_11_allocation_size(kinfo->unassigned_mem);
>> int i;
>>
>>-bool_t lowmem = is_32bit_domain(d);
>>+bool_t lowmem = true;
>> unsigned int bits;
>>
>> /*
>>@@ -265,24 +267,40 @@ static 

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

On Thu, Sep 22, 2016 at 10:51:04AM +0100, George Dunlap wrote:
>On 22/09/16 10:27, Peng Fan wrote:
>> On Thu, Sep 22, 2016 at 10:50:23AM +0200, Dario Faggioli wrote:
>>> On Thu, 2016-09-22 at 14:49 +0800, Peng Fan wrote:
>>>> On Wed, Sep 21, 2016 at 08:11:43PM +0100, Julien Grall wrote:
>>>>>
>>>>> Hi Stefano,
>>>>>
>>>>> On 21/09/2016 19:13, Stefano Stabellini wrote:
>>>>>>
>>>>>> On Wed, 21 Sep 2016, Julien Grall wrote:
>>>>>>>
>>>>>>> (CC a couple of ARM folks)
>>>>>>>
>>>>>>> On 21/09/16 11:22, George Dunlap wrote:
>>>>>>>>
>>>>>>>> On 21/09/16 11:09, Julien Grall wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 20/09/16 21:17, Stefano Stabellini wrote:
>>>>>>>>>>
>>>>>>>>>> On Tue, 20 Sep 2016, Julien Grall wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Stefano,
>>>>>>>>>>>
>>>>>>>>>>> On 20/09/2016 20:09, Stefano Stabellini wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, 20 Sep 2016, Julien Grall wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 20/09/2016 12:27, George Dunlap wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Sep 20, 2016 at 11:03 AM, Peng Fan
>>>>>>>>>>>>>> <van.free...@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Sep 20, 2016 at 02:54:06AM +0200, Dario
>>>>>>>>>>>>>>> Faggioli
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, 2016-09-19 at 17:01 -0700, Stefano
>>>>>>>>>>>>>>>> Stabellini wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, 20 Sep 2016, Dario Faggioli wrote:
>>>>>>>>>>>>>>> I'd like to add a computing capability in
>>>>>>>>>>>>>>> xen/arm, like this:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> struct compute_capatiliby
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>> ?? char *core_name;
>>>>>>>>>>>>>>> ?? uint32_t rank;
>>>>>>>>>>>>>>> ?? uint32_t cpu_partnum;
>>>>>>>>>>>>>>> };
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> struct compute_capatiliby cc=
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>> ??{"A72", 4, 0xd08},
>>>>>>>>>>>>>>> ??{"A57", 3, 0},
>>>>>>>>>>>>>>> ??{"A53", 2, 0xd03},
>>>>>>>>>>>>>>> ??{"A35", 1, ...},
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Then when identify cpu, we decide which cpu is
>>>>>>>>>>>>>>> big and which
>>>>>>>>>>>>>>> cpu is
>>>>>>>>>>>>>>> little
>>>>>>>>>>>>>>> according to the computing rank.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Any comments?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think we definitely need to have Xen have some
>>>>>>>&g

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

On Thu, Sep 22, 2016 at 10:50:23AM +0200, Dario Faggioli wrote:
>On Thu, 2016-09-22 at 14:49 +0800, Peng Fan wrote:
>> On Wed, Sep 21, 2016 at 08:11:43PM +0100, Julien Grall wrote:
>> > 
>> > Hi Stefano,
>> > 
>> > On 21/09/2016 19:13, Stefano Stabellini wrote:
>> > > 
>> > > On Wed, 21 Sep 2016, Julien Grall wrote:
>> > > > 
>> > > > (CC a couple of ARM folks)
>> > > > 
>> > > > On 21/09/16 11:22, George Dunlap wrote:
>> > > > > 
>> > > > > On 21/09/16 11:09, Julien Grall wrote:
>> > > > > > 
>> > > > > > 
>> > > > > > 
>> > > > > > On 20/09/16 21:17, Stefano Stabellini wrote:
>> > > > > > > 
>> > > > > > > On Tue, 20 Sep 2016, Julien Grall wrote:
>> > > > > > > > 
>> > > > > > > > Hi Stefano,
>> > > > > > > > 
>> > > > > > > > On 20/09/2016 20:09, Stefano Stabellini wrote:
>> > > > > > > > > 
>> > > > > > > > > On Tue, 20 Sep 2016, Julien Grall wrote:
>> > > > > > > > > > 
>> > > > > > > > > > Hi,
>> > > > > > > > > > 
>> > > > > > > > > > On 20/09/2016 12:27, George Dunlap wrote:
>> > > > > > > > > > > 
>> > > > > > > > > > > On Tue, Sep 20, 2016 at 11:03 AM, Peng Fan
>> > > > > > > > > > > <van.free...@gmail.com>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > > 
>> > > > > > > > > > > > On Tue, Sep 20, 2016 at 02:54:06AM +0200, Dario
>> > > > > > > > > > > > Faggioli
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > 
>> > > > > > > > > > > > > On Mon, 2016-09-19 at 17:01 -0700, Stefano
>> > > > > > > > > > > > > Stabellini wrote:
>> > > > > > > > > > > > > > 
>> > > > > > > > > > > > > > On Tue, 20 Sep 2016, Dario Faggioli wrote:
>> > > > > > > > > > > > I'd like to add a computing capability in
>> > > > > > > > > > > > xen/arm, like this:
>> > > > > > > > > > > > 
>> > > > > > > > > > > > struct compute_capatiliby
>> > > > > > > > > > > > {
>> > > > > > > > > > > > ?? char *core_name;
>> > > > > > > > > > > > ?? uint32_t rank;
>> > > > > > > > > > > > ?? uint32_t cpu_partnum;
>> > > > > > > > > > > > };
>> > > > > > > > > > > > 
>> > > > > > > > > > > > struct compute_capatiliby cc=
>> > > > > > > > > > > > {
>> > > > > > > > > > > > ??{"A72", 4, 0xd08},
>> > > > > > > > > > > > ??{"A57", 3, 0},
>> > > > > > > > > > > > ??{"A53", 2, 0xd03},
>> > > > > > > > > > > > ??{"A35", 1, ...},
>> > > > > > > > > > > > }
>> > > > > > > > > > > > 
>> > > > > > > > > > > > Then when identify cpu, we decide which cpu is
>> > > > > > > > > > > > big and which
>> > > > > > > > > > > > cpu is
>> > > > > > > > > > > > little
>> > > > > > > > > > > > according to the computing rank.
>> > > > > > > > > > > > 
>> > > > > > > > > > > > Any comments?
>> > > > > > > > > > > 
>> > > > > > > > > > > I think we definitely need to have Xen have some
>> > > > > > > > > > > kind of idea
>> > > >

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

On Wed, Sep 21, 2016 at 11:15:35AM +0100, Julien Grall wrote:
>Hello Peng,
>
>On 21/09/16 09:38, Peng Fan wrote:
>>On Tue, Sep 20, 2016 at 01:17:04PM -0700, Stefano Stabellini wrote:
>>>On Tue, 20 Sep 2016, Julien Grall wrote:
>>>>On 20/09/2016 20:09, Stefano Stabellini wrote:
>>>>>On Tue, 20 Sep 2016, Julien Grall wrote:
>>>>>>On 20/09/2016 12:27, George Dunlap wrote:
>>>>>>>On Tue, Sep 20, 2016 at 11:03 AM, Peng Fan <van.free...@gmail.com>
>>>>>>>wrote:
>>>>>>>>On Tue, Sep 20, 2016 at 02:54:06AM +0200, Dario Faggioli wrote:
>>>>>>>>>On Mon, 2016-09-19 at 17:01 -0700, Stefano Stabellini wrote:
>>>>>>>>>>On Tue, 20 Sep 2016, Dario Faggioli wrote:
>>>>>It is harder to figure out which one is supposed to be
>>>>>big and which one LITTLE. Regardless, we could default to using the
>>>>>first cluster (usually big), which is also the cluster of the boot cpu,
>>>>>and utilize the second cluster only when the user demands it.
>>>>
>>>>Why do you think the boot CPU will usually be a big one? In the case of Juno
>>>>platform it is configurable, and the boot CPU is a little core on r2 by
>>>>default.
>>>>
>>>>In any case, what we care about is differentiate between two set of CPUs. I
>>>>don't think Xen should care about migrating a guest vCPU between big and
>>>>LITTLE cpus. So I am not sure why we would want to know that.
>>>
>>>No, it is not about migrating (at least yet). It is about giving useful
>>>information to the user. It would be nice if the user had to choose
>>>between "big" and "LITTLE" rather than "class 0x1" and "class 0x100", or
>>>even "A7" or "A15".
>>
>>As Dario mentioned in previous email,
>>for dom0 provide like this:
>>
>>dom0_vcpus_big = 4
>>dom0_vcpus_little = 2
>>
>>to dom0.
>>
>>If these two no provided, we could let dom0 runs on big pcpus or big.little.
>>Anyway this is not the important point for dom0 only big or big.little.
>>
>>For domU, provide "vcpus.big" and "vcpus.little" in xl configuration file.
>>Such as:
>>
>>vcpus.big = 2
>>vcpus.litle = 4
>>
>>
>>According to George's comments,
>>Then, I think we could use affinity to restrict little vcpus be scheduled on 
>>little vcpus,
>>and restrict big vcpus on big vcpus. Seems no need to consider soft affinity, 
>>use hard
>>affinity is to handle this.
>>
>>We may need to provide some interface to let xl can get the information such 
>>as
>>big.little or smp. if it is big.little, which is big and which is little.
>>
>>For how to differentiate cpus, I am looking the linaro eas cpu topology code,
>>The code has not been upstreamed (:, but merged into google android kernel.
>>I only plan to take some necessary code, such as device tree parse and
>>cpu topology build, because we only need to know the computing capacity of 
>>each pcpu.
>>
>>Some doc about eas piece, including dts node examples:
>>https://git.linaro.org/arm/eas/kernel.git/blob/refs/heads/lsk-v4.4-eas-v5.2:/Documentation/devicetree/bindings/scheduler/sched-energy-costs.txt
>
>I am reluctant to take any non-upstreamed bindings in Xen. There is a similar
>series going on the lklm [1].

For how to differentiate cpu classes, how about directly use
compatible property of each cpu node?

A57_0: cpu@0 {
compatible = "arm,cortex-a57","arm,armv8";
reg = <0x0 0x0>;
...
};

A53_0: cpu@100 {
compatible = "arm,cortex-a53","arm,armv8";
reg = <0x0 0x100>;
.
}

Thanks,
Peng.

>
>But it sounds like it is a lot of works for little benefits (i.e giving a
>better name to the set of CPUs). The naming will also not fit if in the
>future hardware will have more than 2 kind of CPUs.
>
>[...]
>
>>I am not sure, but we may also need to handle mpidr for ARM, because big and 
>>little vcpus are supported.
>
>I am not sure to understand what you mean here.
>
>Regards,
>
>[1] https://lwn.net/Articles/699569/
>
>-- 
>Julien Grall

-- 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

On Thu, Sep 22, 2016 at 10:50:23AM +0200, Dario Faggioli wrote:
>On Thu, 2016-09-22 at 14:49 +0800, Peng Fan wrote:
>> On Wed, Sep 21, 2016 at 08:11:43PM +0100, Julien Grall wrote:
>> > 
>> > Hi Stefano,
>> > 
>> > On 21/09/2016 19:13, Stefano Stabellini wrote:
>> > > 
>> > > On Wed, 21 Sep 2016, Julien Grall wrote:
>> > > > 
>> > > > (CC a couple of ARM folks)
>> > > > 
>> > > > On 21/09/16 11:22, George Dunlap wrote:
>> > > > > 
>> > > > > On 21/09/16 11:09, Julien Grall wrote:
>> > > > > > 
>> > > > > > 
>> > > > > > 
>> > > > > > On 20/09/16 21:17, Stefano Stabellini wrote:
>> > > > > > > 
>> > > > > > > On Tue, 20 Sep 2016, Julien Grall wrote:
>> > > > > > > > 
>> > > > > > > > Hi Stefano,
>> > > > > > > > 
>> > > > > > > > On 20/09/2016 20:09, Stefano Stabellini wrote:
>> > > > > > > > > 
>> > > > > > > > > On Tue, 20 Sep 2016, Julien Grall wrote:
>> > > > > > > > > > 
>> > > > > > > > > > Hi,
>> > > > > > > > > > 
>> > > > > > > > > > On 20/09/2016 12:27, George Dunlap wrote:
>> > > > > > > > > > > 
>> > > > > > > > > > > On Tue, Sep 20, 2016 at 11:03 AM, Peng Fan
>> > > > > > > > > > > <van.free...@gmail.com>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > > 
>> > > > > > > > > > > > On Tue, Sep 20, 2016 at 02:54:06AM +0200, Dario
>> > > > > > > > > > > > Faggioli
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > 
>> > > > > > > > > > > > > On Mon, 2016-09-19 at 17:01 -0700, Stefano
>> > > > > > > > > > > > > Stabellini wrote:
>> > > > > > > > > > > > > > 
>> > > > > > > > > > > > > > On Tue, 20 Sep 2016, Dario Faggioli wrote:
>> > > > > > > > > > > > I'd like to add a computing capability in
>> > > > > > > > > > > > xen/arm, like this:
>> > > > > > > > > > > > 
>> > > > > > > > > > > > struct compute_capatiliby
>> > > > > > > > > > > > {
>> > > > > > > > > > > > ?? char *core_name;
>> > > > > > > > > > > > ?? uint32_t rank;
>> > > > > > > > > > > > ?? uint32_t cpu_partnum;
>> > > > > > > > > > > > };
>> > > > > > > > > > > > 
>> > > > > > > > > > > > struct compute_capatiliby cc=
>> > > > > > > > > > > > {
>> > > > > > > > > > > > ??{"A72", 4, 0xd08},
>> > > > > > > > > > > > ??{"A57", 3, 0},
>> > > > > > > > > > > > ??{"A53", 2, 0xd03},
>> > > > > > > > > > > > ??{"A35", 1, ...},
>> > > > > > > > > > > > }
>> > > > > > > > > > > > 
>> > > > > > > > > > > > Then when identify cpu, we decide which cpu is
>> > > > > > > > > > > > big and which
>> > > > > > > > > > > > cpu is
>> > > > > > > > > > > > little
>> > > > > > > > > > > > according to the computing rank.
>> > > > > > > > > > > > 
>> > > > > > > > > > > > Any comments?
>> > > > > > > > > > > 
>> > > > > > > > > > > I think we definitely need to have Xen have some
>> > > > > > > > > > > kind of idea
>> > > >

[Xen-devel] [PATCH V3] xen/arm: domain_build: allocate lowmem for dom0 as much as possible

[Peng Fan]

On AArch64 SoCs, some IPs may only have the capability to access
32 bits address space. The physical memory assigned for Dom0 maybe
not in 4GB address space, then the IPs will not work properly.
So need to allocate memory under 4GB for Dom0.

There is no restriction that how much lowmem needs to be allocated for
Dom0. Dom0 now use 1:1 mapping, but DomU does not use 1:1 mapping,
there is no need to reserve lowmem for DomU, so allocate lowmem as much
as possible for Dom0.

This patch does not affect 32-bit domain, because Variable "lowmem" is
set to true at the beginning. If failed to allocate bank0 under 4GB,
need to panic for 32-bit domain, because 32-bit domain requires bank0
be allocated under 4GB.

For 64-bit domain, set "lowmem" to false, and continue allocating
memory from higher memory space.

Signed-off-by: Peng Fan <peng@nxp.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
Cc: Julien Grall <julien.gr...@arm.com>
---

This patch is to resolve the issue mentioned in
https://lists.xen.org/archives/html/xen-devel/2016-09/msg00235.html

V3:
 Add more commit log
 Add more comments
 Add back panic if failed to allocate bank0 under 4GB for 32-bit domain.

V2:
 Remove the bootargs dom0_lowmem introduced in V1.
 Following 
"https://lists.xenproject.org/archives/html/xen-devel/2016-09/msg01459.html;
 to allocate as much as possible lowmem.

 Tested results:
 (XEN) Allocating 1:1 mappings totalling 2048MB for dom0:
 (XEN) BANK[0] 0x008800-0x00f800 (1792MB)
 (XEN) BANK[1] 0x09e000-0x09f000 (256MB)
 1792M allocated in 4GB address space.

 xen/arch/arm/domain_build.c | 44 +++-
 1 file changed, 31 insertions(+), 13 deletions(-)

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 35ab08d..a9c37c4 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -195,9 +195,10 @@ fail:
  *bank. Partly this is just easier for us to deal with, but also
  *the ramdisk and DTB must be placed within a certain proximity of
  *the kernel within RAM.
- * 3. For 32-bit dom0 we want to place as much of the RAM as we
- *reasonably can below 4GB, so that it can be used by non-LPAE
- *enabled kernels.
+ * 3. For dom0 we want to place as much of the RAM as we reasonably can
+ *below 4GB, so that the devices have the limitation to access 64 bits
+ *address space and work properly and it can be used by non-LPAE enabled
+ *kernels.
  * 4. For 32-bit dom0 the kernel must be located below 4GB.
  * 5. We want to have a few largers banks rather than many smaller ones.
  *
@@ -230,7 +231,8 @@ fail:
  * we give up.
  *
  * For 32-bit domain we require that the initial allocation for the
- * first bank is under 4G. Then for the subsequent allocations we
+ * first bank is under 4G. For 64-bit domain, the first bank is preferred
+ * to be allocated under 4G. Then for the subsequent allocations we
  * initially allocate memory only from below 4GB. Once that runs out
  * (as described above) we allow higher allocations and continue until
  * that runs out (or we have allocated sufficient dom0 memory).
@@ -240,11 +242,11 @@ static void allocate_memory(struct domain *d, struct 
kernel_info *kinfo)
 const unsigned int min_low_order =
 get_order_from_bytes(min_t(paddr_t, dom0_mem, MB(128)));
 const unsigned int min_order = get_order_from_bytes(MB(4));
-struct page_info *pg;
+struct page_info *pg = NULL;
 unsigned int order = get_11_allocation_size(kinfo->unassigned_mem);
 int i;
 
-bool_t lowmem = is_32bit_domain(d);
+bool_t lowmem = true;
 unsigned int bits;
 
 /*
@@ -265,24 +267,40 @@ static void allocate_memory(struct domain *d, struct 
kernel_info *kinfo)
  */
 while ( order >= min_low_order )
 {
-for ( bits = order ; bits <= (lowmem ? 32 : PADDR_BITS); bits++ )
+for ( bits = order ; bits <= 32 ; bits++ )
 {
 pg = alloc_domheap_pages(d, order, MEMF_bits(bits));
 if ( pg != NULL )
+{
+if ( !insert_11_bank(d, kinfo, pg, order) )
+BUG(); /* Cannot fail for first bank */
+
 goto got_bank0;
+   }
 }
 order--;
 }
 
-panic("Unable to allocate first memory bank");
-
- got_bank0:
+/* Failed to allocate bank0 under 4GB */
+if ( pg == NULL )
+{
+if ( is_32bit_domain(d) )
+panic("Unable to allocate first memory bank.");
 
-if ( !insert_11_bank(d, kinfo, pg, order) )
-BUG(); /* Cannot fail for first bank */
+/* Try allocating higher memory */
+printk(XENLOG_INFO "No bank has been allocated below 4GB.\n");
+lowmem = false;
+}
 
-/* Now allocate more memory and fill in additional banks */
+ got_bank0:
 
+/*
+ * If failed to allocate bank0 und

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

On Wed, Sep 21, 2016 at 08:11:43PM +0100, Julien Grall wrote:
>Hi Stefano,
>
>On 21/09/2016 19:13, Stefano Stabellini wrote:
>>On Wed, 21 Sep 2016, Julien Grall wrote:
>>>(CC a couple of ARM folks)
>>>
>>>On 21/09/16 11:22, George Dunlap wrote:
>>>>On 21/09/16 11:09, Julien Grall wrote:
>>>>>
>>>>>
>>>>>On 20/09/16 21:17, Stefano Stabellini wrote:
>>>>>>On Tue, 20 Sep 2016, Julien Grall wrote:
>>>>>>>Hi Stefano,
>>>>>>>
>>>>>>>On 20/09/2016 20:09, Stefano Stabellini wrote:
>>>>>>>>On Tue, 20 Sep 2016, Julien Grall wrote:
>>>>>>>>>Hi,
>>>>>>>>>
>>>>>>>>>On 20/09/2016 12:27, George Dunlap wrote:
>>>>>>>>>>On Tue, Sep 20, 2016 at 11:03 AM, Peng Fan
>>>>>>>>>><van.free...@gmail.com>
>>>>>>>>>>wrote:
>>>>>>>>>>>On Tue, Sep 20, 2016 at 02:54:06AM +0200, Dario Faggioli
>>>>>>>>>>>wrote:
>>>>>>>>>>>>On Mon, 2016-09-19 at 17:01 -0700, Stefano Stabellini wrote:
>>>>>>>>>>>>>On Tue, 20 Sep 2016, Dario Faggioli wrote:
>>>>>>>>>>>I'd like to add a computing capability in xen/arm, like this:
>>>>>>>>>>>
>>>>>>>>>>>struct compute_capatiliby
>>>>>>>>>>>{
>>>>>>>>>>>   char *core_name;
>>>>>>>>>>>   uint32_t rank;
>>>>>>>>>>>   uint32_t cpu_partnum;
>>>>>>>>>>>};
>>>>>>>>>>>
>>>>>>>>>>>struct compute_capatiliby cc=
>>>>>>>>>>>{
>>>>>>>>>>>  {"A72", 4, 0xd08},
>>>>>>>>>>>  {"A57", 3, 0},
>>>>>>>>>>>  {"A53", 2, 0xd03},
>>>>>>>>>>>  {"A35", 1, ...},
>>>>>>>>>>>}
>>>>>>>>>>>
>>>>>>>>>>>Then when identify cpu, we decide which cpu is big and which
>>>>>>>>>>>cpu is
>>>>>>>>>>>little
>>>>>>>>>>>according to the computing rank.
>>>>>>>>>>>
>>>>>>>>>>>Any comments?
>>>>>>>>>>
>>>>>>>>>>I think we definitely need to have Xen have some kind of idea
>>>>>>>>>>the
>>>>>>>>>>order between processors, so that the user doesn't need to
>>>>>>>>>>figure out
>>>>>>>>>>which class / pool is big and which pool is LITTLE.  Whether
>>>>>>>>>>this
>>>>>>>>>>sort
>>>>>>>>>>of enumeration is the best way to do that I'll let Julien and
>>>>>>>>>>Stefano
>>>>>>>>>>give their opinion.
>>>>>>>>>
>>>>>>>>>I don't think an hardcoded list of processor in Xen is the right
>>>>>>>>>solution.
>>>>>>>>>There are many existing processors and combinations for big.LITTLE
>>>>>>>>>so it
>>>>>>>>>will
>>>>>>>>>nearly be impossible to keep updated.
>>>>>>>>>
>>>>>>>>>I would expect the firmware table (device tree, ACPI) to provide
>>>>>>>>>relevant
>>>>>>>>>data
>>>>>>>>>for each processor and differentiate big from LITTLE core.
>>>>>>>>>Note that I haven't looked at it for now. A good place to start is
>>>>>>>>>looking
>>>>>>>>>at
>>>>>>>>>how Linux does.
>>>>>>>>
>>>>>>>>That's right, see Documentation/devicetree/bindings/arm/cpus.txt. It
>>>>>>>>is
>>>>>>>>trivial to identify the two different CPU classes and which cores
>>>>>>>>belong
>>>>>>>>to which class.t, as
>>

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

On Wed, Sep 21, 2016 at 08:28:32PM +0100, Julien Grall wrote:
>Hi Dario,
>
>On 21/09/2016 16:45, Dario Faggioli wrote:
>>On Wed, 2016-09-21 at 14:06 +0100, Julien Grall wrote:
>>>(CC a couple of ARM folks)
>>>
>>Yay, thanks for this! :-)
>>
>>>I had few discussions and  more thought about big.LITTLE support in
>>>Xen.
>>>The main goal of big.LITTLE is power efficiency by moving task
>>>around
>>>and been able to idle one cluster. All the solutions suggested
>>>(including mine) so far, can be replicated by hand (except the VPIDR)
>>>so
>>>they are mostly an automatic way.
>>>
>>I'm sorry, how is this (going to be) handled in Linux? Is it that any
>>arbitrary task executing any arbitrary binary code can be run on both
>>big and LITTLE pcpus, depending on the scheduler's and energy
>>management's decisions?
>>
>>This does not seem to match with what has been said at some point in
>>this thread... And if it's like that, how's that possible, if the
>>pcpus' ISAs are (even only slightly) different?
>
>Right, at some point I mentioned that the set of errata and features will be
>different between processor.
>
>However, it is possible to sanitize the feature registers to expose a common
>set to the guest. This is what is done in Linux at boot time, only the
>features common to all the CPUs will be enabled.
>
>This allows a task to migrate between big and LITTLE CPUs seamlessly.
>
>>
>>>This will also remove the real
>>>benefits of big.LITTLE because Xen will not be able to migrate vCPU
>>>across cluster for power efficiency.
>>>
>>>If we care about power efficiency, we would have to handle
>>>seamlessly
>>>big.LITTLE in Xen (i.e a guess would only see a kind of CPU).
>>>
>>Well, I'm a big fan of an approach that leaves the guests' scheduler
>>dumb about things like these (i.e., load balancing, energy efficiency,
>>etc), and hence puts Xen in charge. In fact, on a Xen system, it is
>>only Xen that has all the info necessary to make wise decisions (e.g.,
>>the load of the _whole_ host, the effect of any decisions on the
>>_whole_ host, etc).
>>
>>But this case may be a LITTLE.bit ( :-PP ) different.
>>
>>Anyway, I guess I'll way your reply to my question above before
>>commenting more.
>>
>>>This arise
>>>quite few problem, nothing insurmountable, similar to migration
>>>across
>>>two platforms with different micro-architecture (e.g processors):
>>>errata, features supported... The guest would have to know the union
>>>of
>>>all the errata (this is done so far via the MIDR, so we would a PV
>>>way
>>>to do it), and only the intersection of features would be exposed to
>>>the
>>>guest. This also means the scheduler would have to be modified to
>>>handle
>>>power efficiency (not strictly necessary at the beginning).
>>>
>>>I agree that a such solution would require some work to implement,
>>>although Xen will have a better control of the energy consumption of
>>>the
>>>platform.
>>>
>>>So the question here, is what do we want to achieve with big.LITTLE?
>>>
>>Just thinking out loud here. So, instead of "just", as George
>>suggested:
>>
>> vcpuclass=["0-1:A35","2-5:A53", "6-7:A72"]
>>
>>we can allow something like the following (note that I'm tossing out
>>random numbers next to the 'A's):
>>
>> vcpuclass = ["0-1:A35", "2-5:A53,A17", "6-7:A72,A24,A31", "12-13:A8"]
>>
>>with the following meaning:
>> - vcpus 0, 1 can only run on pcpus of class A35
>> - vcpus 2,3,4,5 can run on pcpus of class A53 _and_ on pcpus of class
>>   A17
>> - vcpus 6,7 can run on pcpus of class A72, A24, A31
>> - vcpus 8,9,10,11 --since they're not mentioned, can run on pcpus of
>>   any class
>> - vcpus 12,13 can only run on pcpus of class A8
>>
>>This will set the "boundaries", for each vcpu. Then, within these
>>boundaries, once in the (Xen's) scheduler, we can implement whatever
>>complex/magic/silly logic we want, e.g.:
>> - only use a pcpu of class A53 for vcpus that have an average load
>>   above 50%
>> - only use a pcpu of class A31 if there are no idle pcpus of class A24
>> - only use a pcpu of class A17 for a vcpu if the total system load
>>   divided by the vcpu ID give 42 as result
>> - whatever
>>
>>This allows us to achieve both the following goals:
>> - allow Xen to take smart decisions, considering the load and the
>>   efficiency of the host as a whole
>> - allow the guest to take smart decisions, like running lightweight
>>   tasks on low power vcpus (which then Xen will run on low
>>   power pcpus, at least on a properly configured system)
>>
>>Of course this **requires** that, for instance, vcpu 6 must be able to
>>run on A72, A24 and A31 just fine, i.e., it must be possible for it to
>>block on I/O when executing on an A72 pcpu, and, later, after wakeup,
>>restart executing on an A24 pcpu.
>
>With a bit of work in Xen, it would be possible to do move the vCPU between
>big and LITTLE cpus. As mentioned above, we could sanitize the features to
>only enable a common set. You can view the big.LITTLE problem as a local live

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

On Wed, Sep 21, 2016 at 11:09:11AM +0100, Julien Grall wrote:
>
>
>On 20/09/16 21:17, Stefano Stabellini wrote:
>>On Tue, 20 Sep 2016, Julien Grall wrote:
>>>Hi Stefano,
>>>
>>>On 20/09/2016 20:09, Stefano Stabellini wrote:
>>>>On Tue, 20 Sep 2016, Julien Grall wrote:
>>>>>Hi,
>>>>>
>>>>>On 20/09/2016 12:27, George Dunlap wrote:
>>>>>>On Tue, Sep 20, 2016 at 11:03 AM, Peng Fan <van.free...@gmail.com>
>>>>>>wrote:
>>>>>>>On Tue, Sep 20, 2016 at 02:54:06AM +0200, Dario Faggioli wrote:
>>>>>>>>On Mon, 2016-09-19 at 17:01 -0700, Stefano Stabellini wrote:
>>>>>>>>>On Tue, 20 Sep 2016, Dario Faggioli wrote:
>>>>>>>I'd like to add a computing capability in xen/arm, like this:
>>>>>>>
>>>>>>>struct compute_capatiliby
>>>>>>>{
>>>>>>>   char *core_name;
>>>>>>>   uint32_t rank;
>>>>>>>   uint32_t cpu_partnum;
>>>>>>>};
>>>>>>>
>>>>>>>struct compute_capatiliby cc=
>>>>>>>{
>>>>>>>  {"A72", 4, 0xd08},
>>>>>>>  {"A57", 3, 0},
>>>>>>>  {"A53", 2, 0xd03},
>>>>>>>  {"A35", 1, ...},
>>>>>>>}
>>>>>>>
>>>>>>>Then when identify cpu, we decide which cpu is big and which cpu is
>>>>>>>little
>>>>>>>according to the computing rank.
>>>>>>>
>>>>>>>Any comments?
>>>>>>
>>>>>>I think we definitely need to have Xen have some kind of idea the
>>>>>>order between processors, so that the user doesn't need to figure out
>>>>>>which class / pool is big and which pool is LITTLE.  Whether this sort
>>>>>>of enumeration is the best way to do that I'll let Julien and Stefano
>>>>>>give their opinion.
>>>>>
>>>>>I don't think an hardcoded list of processor in Xen is the right solution.
>>>>>There are many existing processors and combinations for big.LITTLE so it
>>>>>will
>>>>>nearly be impossible to keep updated.
>>>>>
>>>>>I would expect the firmware table (device tree, ACPI) to provide relevant
>>>>>data
>>>>>for each processor and differentiate big from LITTLE core.
>>>>>Note that I haven't looked at it for now. A good place to start is looking
>>>>>at
>>>>>how Linux does.
>>>>
>>>>That's right, see Documentation/devicetree/bindings/arm/cpus.txt. It is
>>>>trivial to identify the two different CPU classes and which cores belong
>>>>to which class.t, as
>>>
>>>The class of the CPU can be found from the MIDR, there is no need to use the
>>>device tree/acpi for that. Note that I don't think there is an easy way in
>>>ACPI (i.e not in AML) to find out the class.
>>>
>>>>It is harder to figure out which one is supposed to be
>>>>big and which one LITTLE. Regardless, we could default to using the
>>>>first cluster (usually big), which is also the cluster of the boot cpu,
>>>>and utilize the second cluster only when the user demands it.
>>>
>>>Why do you think the boot CPU will usually be a big one? In the case of Juno
>>>platform it is configurable, and the boot CPU is a little core on r2 by
>>>default.
>>>
>>>In any case, what we care about is differentiate between two set of CPUs. I
>>>don't think Xen should care about migrating a guest vCPU between big and
>>>LITTLE cpus. So I am not sure why we would want to know that.
>>
>>No, it is not about migrating (at least yet). It is about giving useful
>>information to the user. It would be nice if the user had to choose
>>between "big" and "LITTLE" rather than "class 0x1" and "class 0x100", or
>>even "A7" or "A15".
>
>I don't think it is wise to assume that we may have only 2 kind of CPUs on
>the platform. We may have more in the future, if so how would you name them?

Consider more than 2 kinds of physical cpus,
"vcpuclass=["0-1:A35","2-5:A53", "6-7:A72"]" seems easier to be handled

Regards,
Peng.

>
>IHMO, asking the user to specify the type of CPUs he wants would be the
>easiest way (though a bit difficult for the user) and avoid us to rely on
>non-upstreamed bindings.
>
>Regards,
>
>-- 
>Julien Grall

-- 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

On Wed, Sep 21, 2016 at 10:22:14AM +0100, George Dunlap wrote:
>On 21/09/16 09:38, Peng Fan wrote:
>> On Tue, Sep 20, 2016 at 01:17:04PM -0700, Stefano Stabellini wrote:
>>> On Tue, 20 Sep 2016, Julien Grall wrote:
>>>> Hi Stefano,
>>>>
>>>> On 20/09/2016 20:09, Stefano Stabellini wrote:
>>>>> On Tue, 20 Sep 2016, Julien Grall wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On 20/09/2016 12:27, George Dunlap wrote:
>>>>>>> On Tue, Sep 20, 2016 at 11:03 AM, Peng Fan <van.free...@gmail.com>
>>>>>>> wrote:
>>>>>>>> On Tue, Sep 20, 2016 at 02:54:06AM +0200, Dario Faggioli wrote:
>>>>>>>>> On Mon, 2016-09-19 at 17:01 -0700, Stefano Stabellini wrote:
>>>>>>>>>> On Tue, 20 Sep 2016, Dario Faggioli wrote:
>>>>>>>> I'd like to add a computing capability in xen/arm, like this:
>>>>>>>>
>>>>>>>> struct compute_capatiliby
>>>>>>>> {
>>>>>>>>char *core_name;
>>>>>>>>uint32_t rank;
>>>>>>>>uint32_t cpu_partnum;
>>>>>>>> };
>>>>>>>>
>>>>>>>> struct compute_capatiliby cc=
>>>>>>>> {
>>>>>>>>   {"A72", 4, 0xd08},
>>>>>>>>   {"A57", 3, 0},
>>>>>>>>   {"A53", 2, 0xd03},
>>>>>>>>   {"A35", 1, ...},
>>>>>>>> }
>>>>>>>>
>>>>>>>> Then when identify cpu, we decide which cpu is big and which cpu is
>>>>>>>> little
>>>>>>>> according to the computing rank.
>>>>>>>>
>>>>>>>> Any comments?
>>>>>>>
>>>>>>> I think we definitely need to have Xen have some kind of idea the
>>>>>>> order between processors, so that the user doesn't need to figure out
>>>>>>> which class / pool is big and which pool is LITTLE.  Whether this sort
>>>>>>> of enumeration is the best way to do that I'll let Julien and Stefano
>>>>>>> give their opinion.
>>>>>>
>>>>>> I don't think an hardcoded list of processor in Xen is the right 
>>>>>> solution.
>>>>>> There are many existing processors and combinations for big.LITTLE so it
>>>>>> will
>>>>>> nearly be impossible to keep updated.
>>>>>>
>>>>>> I would expect the firmware table (device tree, ACPI) to provide relevant
>>>>>> data
>>>>>> for each processor and differentiate big from LITTLE core.
>>>>>> Note that I haven't looked at it for now. A good place to start is 
>>>>>> looking
>>>>>> at
>>>>>> how Linux does.
>>>>>
>>>>> That's right, see Documentation/devicetree/bindings/arm/cpus.txt. It is
>>>>> trivial to identify the two different CPU classes and which cores belong
>>>>> to which class.t, as
>>>>
>>>> The class of the CPU can be found from the MIDR, there is no need to use 
>>>> the
>>>> device tree/acpi for that. Note that I don't think there is an easy way in
>>>> ACPI (i.e not in AML) to find out the class.
>>>>
>>>>> It is harder to figure out which one is supposed to be
>>>>> big and which one LITTLE. Regardless, we could default to using the
>>>>> first cluster (usually big), which is also the cluster of the boot cpu,
>>>>> and utilize the second cluster only when the user demands it.
>>>>
>>>> Why do you think the boot CPU will usually be a big one? In the case of 
>>>> Juno
>>>> platform it is configurable, and the boot CPU is a little core on r2 by
>>>> default.
>>>>
>>>> In any case, what we care about is differentiate between two set of CPUs. I
>>>> don't think Xen should care about migrating a guest vCPU between big and
>>>> LITTLE cpus. So I am not sure why we would want to know that.
>>>
>>> No, it is not about migrating (at least yet). It is about giving useful
>>> information to the user. It would be nice if the user had to choose
>>> between "big" and "LITTLE

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

On Wed, Sep 21, 2016 at 11:15:35AM +0100, Julien Grall wrote:
>Hello Peng,
>
>On 21/09/16 09:38, Peng Fan wrote:
>>On Tue, Sep 20, 2016 at 01:17:04PM -0700, Stefano Stabellini wrote:
>>>On Tue, 20 Sep 2016, Julien Grall wrote:
>>>>On 20/09/2016 20:09, Stefano Stabellini wrote:
>>>>>On Tue, 20 Sep 2016, Julien Grall wrote:
>>>>>>On 20/09/2016 12:27, George Dunlap wrote:
>>>>>>>On Tue, Sep 20, 2016 at 11:03 AM, Peng Fan <van.free...@gmail.com>
>>>>>>>wrote:
>>>>>>>>On Tue, Sep 20, 2016 at 02:54:06AM +0200, Dario Faggioli wrote:
>>>>>>>>>On Mon, 2016-09-19 at 17:01 -0700, Stefano Stabellini wrote:
>>>>>>>>>>On Tue, 20 Sep 2016, Dario Faggioli wrote:
>>>>>It is harder to figure out which one is supposed to be
>>>>>big and which one LITTLE. Regardless, we could default to using the
>>>>>first cluster (usually big), which is also the cluster of the boot cpu,
>>>>>and utilize the second cluster only when the user demands it.
>>>>
>>>>Why do you think the boot CPU will usually be a big one? In the case of Juno
>>>>platform it is configurable, and the boot CPU is a little core on r2 by
>>>>default.
>>>>
>>>>In any case, what we care about is differentiate between two set of CPUs. I
>>>>don't think Xen should care about migrating a guest vCPU between big and
>>>>LITTLE cpus. So I am not sure why we would want to know that.
>>>
>>>No, it is not about migrating (at least yet). It is about giving useful
>>>information to the user. It would be nice if the user had to choose
>>>between "big" and "LITTLE" rather than "class 0x1" and "class 0x100", or
>>>even "A7" or "A15".
>>
>>As Dario mentioned in previous email,
>>for dom0 provide like this:
>>
>>dom0_vcpus_big = 4
>>dom0_vcpus_little = 2
>>
>>to dom0.
>>
>>If these two no provided, we could let dom0 runs on big pcpus or big.little.
>>Anyway this is not the important point for dom0 only big or big.little.
>>
>>For domU, provide "vcpus.big" and "vcpus.little" in xl configuration file.
>>Such as:
>>
>>vcpus.big = 2
>>vcpus.litle = 4
>>
>>
>>According to George's comments,
>>Then, I think we could use affinity to restrict little vcpus be scheduled on 
>>little vcpus,
>>and restrict big vcpus on big vcpus. Seems no need to consider soft affinity, 
>>use hard
>>affinity is to handle this.
>>
>>We may need to provide some interface to let xl can get the information such 
>>as
>>big.little or smp. if it is big.little, which is big and which is little.
>>
>>For how to differentiate cpus, I am looking the linaro eas cpu topology code,
>>The code has not been upstreamed (:, but merged into google android kernel.
>>I only plan to take some necessary code, such as device tree parse and
>>cpu topology build, because we only need to know the computing capacity of 
>>each pcpu.
>>
>>Some doc about eas piece, including dts node examples:
>>https://git.linaro.org/arm/eas/kernel.git/blob/refs/heads/lsk-v4.4-eas-v5.2:/Documentation/devicetree/bindings/scheduler/sched-energy-costs.txt
>
>I am reluctant to take any non-upstreamed bindings in Xen. There is a similar

Yeah. I understand :)

>series going on the lklm [1].

I'll have a look at this series, seems simpler than the code in linaro tree.

Whether the EAS cpu topology code or the series you listed, this is to let us
differentiate the cpu classes. This is not the hard point, just what
information to get from dts.

We need to reach a point that how to arrange the different cpu classes, I think.

Think we get dmips/cap from dts for each cpu, put the info into cpu_data for 
each cpu?

>
>But it sounds like it is a lot of works for little benefits (i.e giving a
>better name to the set of CPUs). The naming will also not fit if in the
>future hardware will have more than 2 kind of CPUs.

Oh. Yeah. There is possibility that an soc contains such as A35 + A53 + A72..
Then xx.big and xx.little seems not enough.

On such SoC, we still need to support big.little guest? We may not call it
big.little guest, if guest also needs A35 + A53 + A72 vcpu..

Use this in xl cfg file?
vcpuclass=["0-1:A35","2-5:A53", "6-7:A72"] ?

I am not sure. If there are more kinds of CPUs, how to handle guest vcpus,
as we discussed in this thread, we tend to support different classes of vcpu
for guest. But if there are many kinds of physical CPUs, we also need to let
guest have so many kinds of virtual cpus?

Anyway the first step for me is to differentiate the physical cpus and
add the info to cpu_data or else.

>
>[...]
>
>>I am not sure, but we may also need to handle mpidr for ARM, because big and 
>>little vcpus are supported.
>
>I am not sure to understand what you mean here.

For big.little guest, which vcpu is in cluster 0 , which is in cluster 1, also 
need
to fill related value for MPIDR for guest.

Regards,
Peng.
>
>Regards,
>
>[1] https://lwn.net/Articles/699569/
>
>-- 
>Julien Grall

-- 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH V2] xen/arm: domain_build: allocate lowmem for dom0 as much as possible

[Peng Fan]

On Wed, Sep 21, 2016 at 11:39:11AM +0100, Julien Grall wrote:
>
>
>On 20/09/16 10:10, Peng Fan wrote:
>>Hello Julien,
>
>Hello Peng,
>
>>On Tue, Sep 20, 2016 at 10:36:27AM +0200, Julien Grall wrote:
>>>Hello Peng,
>>>
>>>On 20/09/2016 07:52, van.free...@gmail.com wrote:
>>>>From: Peng Fan <peng@nxp.com>
>>>>
>>>>On AArch64 SoCs, some IPs may only have the capability to access
>>>>32bits address space. The physical memory assigned for Dom0 maybe
>>>
>>>s/32bits/32 bits/
>>
>>Fix in V3.
>>
>>>
>>>>not in 4GB address space, then the IPs will not work properly.
>>>>So need to allocate memory under 4GB for Dom0.
>>>>
>>>>There is no restriction that how much lowmem needs to be allocated for
>>>>Dom0. Dom0 now use 1:1 mapping, but DomU does not use 1:1 mapping,
>>>>there is no need to reserve lowmem for DomU, so allocate lowmem as much
>>>>as possible for Dom0.
>>>>
>>>>Signed-off-by: Peng Fan <peng@nxp.com>
>>>>Cc: Stefano Stabellini <sstabell...@kernel.org>
>>>>Cc: Julien Grall <julien.gr...@arm.com>
>>>>---
>>>>
>>>>This patch is to resolve the issue mentioned in
>>>>https://lists.xen.org/archives/html/xen-devel/2016-09/msg00235.html
>>>>
>>>>V2:
>>>>Remove the bootargs dom0_lowmem introduced in V1.
>>>>Following 
>>>>"https://lists.xenproject.org/archives/html/xen-devel/2016-09/msg01459.html;
>>>>to allocate as much as possible lowmem.
>>>>
>>>>Tested results:
>>>>(XEN) Allocating 1:1 mappings totalling 2048MB for dom0:
>>>>(XEN) BANK[0] 0x008800-0x00f800 (1792MB)
>>>>(XEN) BANK[1] 0x09e000-0x09f000 (256MB)
>>>>1792M allocated in 4GB address space.
>>>>
>>>>xen/arch/arm/domain_build.c | 20 +---
>>>>1 file changed, 13 insertions(+), 7 deletions(-)
>>>>
>>>>diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>>>>index 35ab08d..0b9b85c 100644
>>>>--- a/xen/arch/arm/domain_build.c
>>>>+++ b/xen/arch/arm/domain_build.c
>>>>@@ -240,11 +240,11 @@ static void allocate_memory(struct domain *d, struct 
>>>>kernel_info *kinfo)
>>>
>>>As mentioned on V1, please update the comment above this function.
>>
>>About this part:
>>* 3. For 32-bit dom0 we want to place as much of the RAM as we
>>*reasonably can below 4GB, so that it can be used by non-LPAE
>>*enabled kernels.
>>
>>Changed to this V3:
>>For dom0 we want to place as much of the RAM as we reasonably can
>>below 4GB, so the devices have the limitation to access 64 bits
>>address space can work properly and it can also be used by
>>non-LPAE enabled kernels for 32-bit dom0.
>>
>>About
>>"
>> * For 32-bit domain we require that the initial allocation for the
>> * first bank is under 4G.
>>"
>>
>>Changed to this in V3:
>>
>>"
>>For 32-bit domain we require that the initial allocation for the
>>first bank is under 4G. For 64-bit domain, the first bank is preferred
>>to be allocated under 4GB.
>>"
>>
>>Is this ok?
>
>I think so.

Thanks. Update in v3.

>
>>
>>>
>>>>const unsigned int min_low_order =
>>>>get_order_from_bytes(min_t(paddr_t, dom0_mem, MB(128)));
>>>>const unsigned int min_order = get_order_from_bytes(MB(4));
>>>>-struct page_info *pg;
>>>>+struct page_info *pg = NULL;
>>>>unsigned int order = get_11_allocation_size(kinfo->unassigned_mem);
>>>>int i;
>>>>
>>>>-bool_t lowmem = is_32bit_domain(d);
>>>>+bool_t lowmem = true;
>>>>unsigned int bits;
>>>>
>>>>/*
>>>>@@ -265,22 +265,28 @@ static void allocate_memory(struct domain *d, struct 
>>>>kernel_info *kinfo)
>>>> */
>>>>while ( order >= min_low_order )
>>>>{
>>>>-for ( bits = order ; bits <= (lowmem ? 32 : PADDR_BITS); bits++ )
>>>>+for ( bits = order ; bits <= 32 ; bits++ )
>>>>{
>>>>pg = alloc_domheap_pages(d, order, MEMF_bits(bits));
>>>>if ( pg != NULL )
>>>>+{
>>>>+

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

On Tue, Sep 20, 2016 at 01:17:04PM -0700, Stefano Stabellini wrote:
>On Tue, 20 Sep 2016, Julien Grall wrote:
>> Hi Stefano,
>> 
>> On 20/09/2016 20:09, Stefano Stabellini wrote:
>> > On Tue, 20 Sep 2016, Julien Grall wrote:
>> > > Hi,
>> > > 
>> > > On 20/09/2016 12:27, George Dunlap wrote:
>> > > > On Tue, Sep 20, 2016 at 11:03 AM, Peng Fan <van.free...@gmail.com>
>> > > > wrote:
>> > > > > On Tue, Sep 20, 2016 at 02:54:06AM +0200, Dario Faggioli wrote:
>> > > > > > On Mon, 2016-09-19 at 17:01 -0700, Stefano Stabellini wrote:
>> > > > > > > On Tue, 20 Sep 2016, Dario Faggioli wrote:
>> > > > > I'd like to add a computing capability in xen/arm, like this:
>> > > > > 
>> > > > > struct compute_capatiliby
>> > > > > {
>> > > > >char *core_name;
>> > > > >uint32_t rank;
>> > > > >uint32_t cpu_partnum;
>> > > > > };
>> > > > > 
>> > > > > struct compute_capatiliby cc=
>> > > > > {
>> > > > >   {"A72", 4, 0xd08},
>> > > > >   {"A57", 3, 0},
>> > > > >   {"A53", 2, 0xd03},
>> > > > >   {"A35", 1, ...},
>> > > > > }
>> > > > > 
>> > > > > Then when identify cpu, we decide which cpu is big and which cpu is
>> > > > > little
>> > > > > according to the computing rank.
>> > > > > 
>> > > > > Any comments?
>> > > > 
>> > > > I think we definitely need to have Xen have some kind of idea the
>> > > > order between processors, so that the user doesn't need to figure out
>> > > > which class / pool is big and which pool is LITTLE.  Whether this sort
>> > > > of enumeration is the best way to do that I'll let Julien and Stefano
>> > > > give their opinion.
>> > > 
>> > > I don't think an hardcoded list of processor in Xen is the right 
>> > > solution.
>> > > There are many existing processors and combinations for big.LITTLE so it
>> > > will
>> > > nearly be impossible to keep updated.
>> > > 
>> > > I would expect the firmware table (device tree, ACPI) to provide relevant
>> > > data
>> > > for each processor and differentiate big from LITTLE core.
>> > > Note that I haven't looked at it for now. A good place to start is 
>> > > looking
>> > > at
>> > > how Linux does.
>> > 
>> > That's right, see Documentation/devicetree/bindings/arm/cpus.txt. It is
>> > trivial to identify the two different CPU classes and which cores belong
>> > to which class.t, as
>> 
>> The class of the CPU can be found from the MIDR, there is no need to use the
>> device tree/acpi for that. Note that I don't think there is an easy way in
>> ACPI (i.e not in AML) to find out the class.
>> 
>> > It is harder to figure out which one is supposed to be
>> > big and which one LITTLE. Regardless, we could default to using the
>> > first cluster (usually big), which is also the cluster of the boot cpu,
>> > and utilize the second cluster only when the user demands it.
>> 
>> Why do you think the boot CPU will usually be a big one? In the case of Juno
>> platform it is configurable, and the boot CPU is a little core on r2 by
>> default.
>> 
>> In any case, what we care about is differentiate between two set of CPUs. I
>> don't think Xen should care about migrating a guest vCPU between big and
>> LITTLE cpus. So I am not sure why we would want to know that.
>
>No, it is not about migrating (at least yet). It is about giving useful
>information to the user. It would be nice if the user had to choose
>between "big" and "LITTLE" rather than "class 0x1" and "class 0x100", or
>even "A7" or "A15".

As Dario mentioned in previous email,
for dom0 provide like this:

dom0_vcpus_big = 4
dom0_vcpus_little = 2

to dom0.

If these two no provided, we could let dom0 runs on big pcpus or big.little.
Anyway this is not the important point for dom0 only big or big.little.

For domU, provide "vcpus.big" and "vcpus.little" in xl configuration file.
Such as:

vcpus.big = 2
vcpus.litle = 4


According to George's comments,
Then, I think we c

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

Hi Dario,
On Tue, Sep 20, 2016 at 02:54:06AM +0200, Dario Faggioli wrote:
>On Mon, 2016-09-19 at 17:01 -0700, Stefano Stabellini wrote:
>> On Tue, 20 Sep 2016, Dario Faggioli wrote:
>> > And this would work even if/when there is only one cpupool, or in
>> > general for domains that are in a pool that has both big and LITTLE
>> > pcpus. Furthermore, big.LITTLE support and cpupools will be
>> > orthogonal,
>> > just like pinning and cpupools are orthogonal right now. I.e., once
>> > we
>> > will have what I described above, nothing prevents us from
>> > implementing
>> > per-vcpu cpupool membership, and either create the two (or more!)
>> > big
>> > and LITTLE pools, or from mixing things even more, for more complex
>> > and
>> > specific use cases. :-)
>> 
>> I think that everybody agrees that this is the best long term
>> solution.
>> 
>Well, no, that wasn't obvious to me. If that's the case, it's already
>something! :-)
>
>> > 
>> > Actually, with the cpupool solution, if you want a guest (or dom0)
>> > to
>> > actually have both big and LITTLE vcpus, you necessarily have to
>> > implement per-vcpu (rather than per-domain, as it is now) cpupool
>> > membership. I said myself it's not impossible, but certainly it's
>> > some
>> > work... with the scheduler solution you basically get that for
>> > free!
>> > 
>> > So, basically, if we use cpupools for the basics of big.LITTLE
>> > support,
>> > there's no way out of it (apart from going implementing scheduling
>> > support afterwords, but that looks backwards to me, especially when
>> > thinking at it with the code in mind).
>> 
>> The question is: what is the best short-term solution we can ask Peng
>> to
>> implement that allows Xen to run on big.LITTLE systems today?
>> Possibly
>> getting us closer to the long term solution, or at least not farther
>> from it?
>> 
>So, I still have to look closely at the patches in these series. But,
>with Credit2 in mind, if one:
>
>??- take advantage of the knowledge of what arch a pcpu belongs inside??

>?? ??the code that arrange the pcpus in runqueues, which means we'll end??
>?? ??up with big runqueues and LITTLE runqueues. I re-wrote that code, I
>?? ??can provide pointers and help, if necessary;
>??- tweak the one or two instance of for_each_runqueue() [*] that there
>?? ??are in the code into a for_each_runqueue_of_same_class(), i.e.:

Do you have plan to add this support for big.LITTLE?

I admit that this is the first time I look into the scheduler part.
If I understand wrongly, please correct me.

There is a runqueue for each physical cpu, and there are several vcpus in the 
runqueue.
The scheduler will pick a vcpu in the runqueue to run on the physical cpu.

A vcpu is bind to a physical cpu when alloc_vcpu, but the vcpu can be scheduled
or migrated to a different physical cpu.

Settings cpu soft affinity and hard affinity to restrict vcpus be scheduled
on specific cpus. Then is there a need to introuduce more runqueues?

This seems more complicated than cpupool (:

>
>??if (is_big(this_cpu))
>??{
>?? ??for_each_big_runqueue()
>?? ??{
>?? ?? ?? ..
>?? ??}
>??}
>??else
>??{
>?? ??for_each_LITTLE_runqueue()
>?? ??{
>?? ?? ??..
>?? ??}
>??}??
>
>then big.LITTLE support in Credit2 would be done already, and all it
>would be left is support for the syntax of new config switches in xl,
>and a way of telling, from xl/libxl down to Xen, what arch a vcpu
>belongs to, so that it can be associated with one runqueue of the
>proper class.
>
>Thinking to Credit1, we need to make sure thet, in load_balance() and
>runq_steal(), a LITTLE cpu *only* ever try to steal work from another
>LITTLE cpu, and __never__ from a big cpu (and vice versa). And also
>that when a vcpu wakes up, and what it has in its v->processor is a
>LITTLE pcpu, that only LITTLE processors are considered for being
>tickled (I'm less certain of this last part, but it should be more or
>less like this).
>
>Then, of course the the same glue and vcpu classification code.
>
>However, in Credit1, it's possible that a trick like that would affect
>the accounting and credit algorithm, and hence provide unfair, or in
>general, unexpected results. Credit2 should, OTOH, be a lot mere
>resilient, wrt that.
>
>> > > The whole process would be more explicit and obvious if we used
>> > > cpupools. It would be easier for users to know what it is going
>> > > on --
>> > > they just need to issue an `xl cpupool-list' command and they
>> > > would
>> > > see
>> > > two clearly named pools (something like big-pool and LITTLE-
>> > > pool).??
>> > > 
>> > Well, I guess that, as part of big.LITTLE support, there will be a
>> > way
>> > to tell what pcpus are big and which are LITTLE anyway, probably
>> > both
>> > from `xl info' and from `xl cpupool-list -c' (and most likely in
>> > other
>> > ways too).
>> 
>> Sure, but it needs to be very clear. We cannot ask people to spot
>> architecture specific flags among the output of `xl info' to be able
>> to
>> appropriately start 

Re: [Xen-devel] [PATCH V2] xen/arm: domain_build: allocate lowmem for dom0 as much as possible

[Peng Fan]

Hello Julien,

On Tue, Sep 20, 2016 at 10:36:27AM +0200, Julien Grall wrote:
>Hello Peng,
>
>On 20/09/2016 07:52, van.free...@gmail.com wrote:
>>From: Peng Fan <peng@nxp.com>
>>
>>On AArch64 SoCs, some IPs may only have the capability to access
>>32bits address space. The physical memory assigned for Dom0 maybe
>
>s/32bits/32 bits/

Fix in V3.

>
>>not in 4GB address space, then the IPs will not work properly.
>>So need to allocate memory under 4GB for Dom0.
>>
>>There is no restriction that how much lowmem needs to be allocated for
>>Dom0. Dom0 now use 1:1 mapping, but DomU does not use 1:1 mapping,
>>there is no need to reserve lowmem for DomU, so allocate lowmem as much
>>as possible for Dom0.
>>
>>Signed-off-by: Peng Fan <peng@nxp.com>
>>Cc: Stefano Stabellini <sstabell...@kernel.org>
>>Cc: Julien Grall <julien.gr...@arm.com>
>>---
>>
>>This patch is to resolve the issue mentioned in
>>https://lists.xen.org/archives/html/xen-devel/2016-09/msg00235.html
>>
>>V2:
>> Remove the bootargs dom0_lowmem introduced in V1.
>> Following 
>> "https://lists.xenproject.org/archives/html/xen-devel/2016-09/msg01459.html;
>> to allocate as much as possible lowmem.
>>
>> Tested results:
>> (XEN) Allocating 1:1 mappings totalling 2048MB for dom0:
>> (XEN) BANK[0] 0x008800-0x00f800 (1792MB)
>> (XEN) BANK[1] 0x09e000-0x09f000 (256MB)
>> 1792M allocated in 4GB address space.
>>
>> xen/arch/arm/domain_build.c | 20 +---
>> 1 file changed, 13 insertions(+), 7 deletions(-)
>>
>>diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>>index 35ab08d..0b9b85c 100644
>>--- a/xen/arch/arm/domain_build.c
>>+++ b/xen/arch/arm/domain_build.c
>>@@ -240,11 +240,11 @@ static void allocate_memory(struct domain *d, struct 
>>kernel_info *kinfo)
>
>As mentioned on V1, please update the comment above this function.

About this part:
* 3. For 32-bit dom0 we want to place as much of the RAM as we
*reasonably can below 4GB, so that it can be used by non-LPAE
*enabled kernels.

Changed to this V3:
For dom0 we want to place as much of the RAM as we reasonably can
below 4GB, so the devices have the limitation to access 64 bits
address space can work properly and it can also be used by
non-LPAE enabled kernels for 32-bit dom0.

About
"
 * For 32-bit domain we require that the initial allocation for the
 * first bank is under 4G.
"

Changed to this in V3:

"
For 32-bit domain we require that the initial allocation for the
first bank is under 4G. For 64-bit domain, the first bank is preferred
to be allocated under 4GB.
"

Is this ok?

>
>> const unsigned int min_low_order =
>> get_order_from_bytes(min_t(paddr_t, dom0_mem, MB(128)));
>> const unsigned int min_order = get_order_from_bytes(MB(4));
>>-struct page_info *pg;
>>+struct page_info *pg = NULL;
>> unsigned int order = get_11_allocation_size(kinfo->unassigned_mem);
>> int i;
>>
>>-bool_t lowmem = is_32bit_domain(d);
>>+bool_t lowmem = true;
>> unsigned int bits;
>>
>> /*
>>@@ -265,22 +265,28 @@ static void allocate_memory(struct domain *d, struct 
>>kernel_info *kinfo)
>>  */
>> while ( order >= min_low_order )
>> {
>>-for ( bits = order ; bits <= (lowmem ? 32 : PADDR_BITS); bits++ )
>>+for ( bits = order ; bits <= 32 ; bits++ )
>> {
>> pg = alloc_domheap_pages(d, order, MEMF_bits(bits));
>> if ( pg != NULL )
>>+{
>>+if ( !insert_11_bank(d, kinfo, pg, order) )
>>+BUG(); /* Cannot fail for first bank */
>>+
>> goto got_bank0;
>>+ }
>> }
>> order--;
>> }
>>
>>-panic("Unable to allocate first memory bank");
>>+if ( pg == NULL )
>>+{
>>+printk(XENLOG_INFO "No bank has been allocated below 4GB.\n");
>>+lowmem = false;
>>+}
>
>This new behavior should be documented in the commit message, with some
>rationale why it is fine to do it for ARM32.

Consider the comments of the function:
  1. The dom0 kernel should be loaded within the first 128MB of RAM. This
 is necessary at least for Linux zImage kernels, which are all we
 support today.
 4. For 32-bit dom0 the kernel must be located below 4GB.

I think I may need to add back the panic when first bank is failed to be 
allocated
under 4GB.

Will use the f

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

On Tue, Sep 20, 2016 at 02:11:04AM +0200, Dario Faggioli wrote:
>On Mon, 2016-09-19 at 21:33 +0800, Peng Fan wrote:
>> On Mon, Sep 19, 2016 at 11:33:58AM +0100, George Dunlap wrote:
>> >??
>> > No, I think it would be a lot simpler to just teach the scheduler
>> > about
>> > different classes of cpus.credit1 would probably need to be
>> > modified
>> > so that its credit algorithm would be per-class rather than pool-
>> > wide;
>> > but credit2 shouldn't need much modification at all, other than to
>> > make
>> > sure that a given runqueue doesn't include more than one class; and
>> > to
>> > do load-balancing only with runqueues of the same class.
>> 
>> I try to follow.
>> ??- scheduler needs to be aware of different classes of cpus. ARM
>> big.Little cpus.
>>
>Yes, I think this is essential.
>
>> ??- scheduler schedules vcpus on different physical cpus in one
>> cpupool.
>>
>Yep, that's what the scheduler does. And personally, I'd start
>implementing big.LITTLE support for a situation where both big and
>LITTLE cpus coexists in the same pool.

It's great if you have plan to work on the scheduler part.

>
>> ??- different cpu classes needs to be in different runqueue.
>> 
>Yes. So, basically, imagine to use vcpu pinning to support big.LITTLE.
>I've spoken briefly about this in my reply to Juergen. You probably can
>even get something like this up-&-running by writing very few or zero
>code (you'll need --for now-- max_dom0_vcpus, dom0_vcpus_pin, and then,
>in domain config files, "cpus='...'").
>
>Then, the real goal, would be to achieve the same behavior
>automatically, by acting on runqueues' arrangement and load balancing
>logic in the scheduler(s).
>
>Anyway, sorry for my ignorance on big.LITTLE, but there's something I'm
>missing: _when_ is it that it is (or needs to be) decided whether a
>vcpu will run on a big or LITTLE core?

Big cores are more powerful than little cores, but consumes more power.
In Linux kernel, linaro is working on EAS scheduler to take advantage of 
big.LITTLE.
http://www.linaro.org/blog/core-dump/energy-aware-scheduling-eas-project/

As discussed, for big.little guest os that have big vcpu and little vcpu,
we only need to take care of big vcpu scheduled on big physical cpus, and little
vcpu sheduled on little physical cpus.
So a vcpu is not be scheduled between big and little physical cpus.

>
>Thinking to a bare metal system, I think that cpu X is, for instance, big, and 
>will always be like that; similarly, cpu Y is LITTLE.
>
>This makes me think that, for a virtual machine, it is ok to choose/specify at 
>_domain_creation_ time, which vcpus are big and which vcpus are LITTLE, is 
>this correct?
>If yes, this also means that --whatever way we find to make this happen, 
>cpupools, scheduler, etc-- the vcpus that we decided they are big, must only 
>be scheduled on actual big pcpus, and pcpus that we decided they are LITTLE, 
>must only be scheduled on actual LITTLE pcpus, correct again?
>
>> Then for implementation.
>> ??- When create a guest, specific physical cpus that the guest will be
>> run on.
>>
>I'd actually do that the other way round. I'd ask the user to specify
>how many --and, if that's important-- vcpus are big and how many/which
>are LITTLE.
>
>Knowing that, we also know whether the domain is a big only, LITTLE
>only or big.LITTLE one. And we also know on which set of pcpus each set
>of vcpus should be restrict to.
>
>So, basically (but it's just an example) something like this, in the xl
>config file of a guest:
>
>1) big.LITTLE guest, with 2 big and 2 LITTLE pcpus. User doesn't care ??
>?? ??which is which, so a default could be 0,1 big and 2,3 LITTLE:
>
>??vcpus = 4
>??vcpus.big = 2
>
>2) big.LITTLE guest, with 8 vcpus, of which 0,2,4 and 6 are big:
>
>vcpus = 8
>vcpus.big = [0, 2, 4, 6]
>
>Which would be the same as
>
>vcpus = 8
>vcpus.little = [1, 3, 5, 7]
>
>3) guest with 4 vcpus, all big:
>
>vcpus = 4
>vcpus.big = "all"
>
>Which would be the same as:
>
>vcpus = 4
>vcpus.little = "none"
>
>And also the same as just:
>
>vcpus = 4
>
>
>Or something like this
>
>> ??- If the physical cpus are different cpus, indicate the guest would
>> like to be a big.little guest.
>> ??And have big vcpus and little vcpus.
>>
>Not liking this as _the_ way of specifying the guest topology, wrt
>big.LITTLE-ness (see alternative proposal right above. :-))
>
>However, right now we support pinning/affinity already. We certainly
>need to decide what to do if, e.g., no vcpus.big or

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

On Mon, Sep 19, 2016 at 11:33:58AM +0100, George Dunlap wrote:
>On 19/09/16 11:06, Julien Grall wrote:
>> Hi George,
>> 
>> On 19/09/2016 11:45, George Dunlap wrote:
>>> On Mon, Sep 19, 2016 at 9:53 AM, Julien Grall 
>>> wrote:
>> As mentioned in the mail you pointed above, this series is not
>> enough to
>> make
>> big.LITTLE working on then. Xen is always using the boot CPU to detect
>> the
>> list of features. With big.LITTLE features may not be the same.
>>
>> And I would prefer to see Xen supporting big.LITTLE correctly before
>> beginning to think to expose big.LITTLE to the userspace (via cpupool)
>> automatically.
>
>
> Do you mean vcpus be scheduled between big and little cpus freely?


 By supporting big.LITTLE correctly I meant Xen thinks that all the
 cores has
 the same set of features. So the feature detection is only done the boot
 CPU. See processor_setup for instance...

 Moving vCPUs between big and little cores would be a hard task (cache
 line
 issue, and possibly feature) and I don't expect to ever cross this in
 Xen.
 However, I am expecting to see big.LITTLE exposed to the guest (i.e
 having
 big and little vCPUs).
>>>
>>> So it sounds like the big and LITTLE cores are architecturally
>>> different enough that software must be aware of which one it's running
>>> on?
>> 
>> That's correct. Each big and LITTLE cores may have different errata,
>> different features...
>> 
>> It has also the advantage to let the guest dealing itself with its own
>> power efficiency without introducing a specific Xen interface.
>
>Well in theory there would be advantages either way -- either to
>allowing Xen to automatically add power-saving "smarts" to guests which
>weren't programmed for them, or to exposing the power-saving abilities
>to guests which were.  But it sounds like automatically migrating
>between them isn't really an option (or would be a lot more trouble than
>it's worth).
>
 I care about having a design allowing an easy use of big.LITTLE on
 Xen. Your
 solution requires the administrator to know the underlying platform and
 create the pool.

 In the solution I suggested, the pools would be created by Xen (and
 the info
 exposed to the userspace for the admin).
>>>
>>> FWIW another approach could be the one taken by "xl
>>> cpupool-numa-split": you could have "xl cpupool-bigLITTLE-split" or
>>> something that would automatically set up the pools.
>>>
>>> But expanding the schedulers to know about different classes of cpus,
>>> and having vcpus specified as running only on specific types of pcpus,
>>> seems like a more flexible approach.
>> 
>> So, if I understand correctly, you would not recommend to extend the
>> number of CPU pool per domain, correct?
>
>Well imagine trying to set the scheduling parameters, such as weight,
>which in the past have been per-domain.  Now you have to specify
>parameters for a domain in each of the cpupools that its' in.
>
>No, I think it would be a lot simpler to just teach the scheduler about
>different classes of cpus.  credit1 would probably need to be modified
>so that its credit algorithm would be per-class rather than pool-wide;
>but credit2 shouldn't need much modification at all, other than to make
>sure that a given runqueue doesn't include more than one class; and to
>do load-balancing only with runqueues of the same class.

I try to follow.
 - scheduler needs to be aware of different classes of cpus. ARM big.Little 
cpus.
 - scheduler schedules vcpus on different physical cpus in one cpupool.
 - different cpu classes needs to be in different runqueue.

Then for implementation.
 - When create a guest, specific physical cpus that the guest will be run on.
 - If the physical cpus are different cpus, indicate the guest would like to be 
a big.little guest.
   And have big vcpus and little vcpus.
 - If no physical cpus specificed, then the guest may runs on big cpus or on 
little cpus. But not both.
   How to decide runs on big or little physical cpus?
 - For Dom0, I am still not sure,default big.little or else?

If use scheduler to handle the different classes cpu, we do not need to use 
cpupool
to block vcpus be scheduled onto different physical cpus. And using scheudler 
to handle this
gives an opportunity to support big.little guest.

Thanks,
Peng.

>
> -George

-- 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

On Mon, Sep 19, 2016 at 11:59:05AM +0200, Julien Grall wrote:
>
>
>On 19/09/2016 11:38, Peng Fan wrote:
>>On Mon, Sep 19, 2016 at 10:53:56AM +0200, Julien Grall wrote:
>>>Hello,
>>>
>>>On 19/09/2016 10:36, Peng Fan wrote:
>>>>On Mon, Sep 19, 2016 at 10:09:06AM +0200, Julien Grall wrote:
>>>>>Hello Peng,
>>>>>
>>>>>On 19/09/2016 04:08, van.free...@gmail.com wrote:
>>>>>>From: Peng Fan <peng@nxp.com>
>>>>>>
>>>>>>This patchset is to support XEN run on big.little SoC.
>>>>>>The idea of the patch is from
>>>>>>"https://lists.xenproject.org/archives/html/xen-devel/2016-05/msg00465.html;
>>>>>>
>>>>>>There are some changes to cpupool and add x86 stub functions to avoid 
>>>>>>build
>>>>>>break. Sending The RFC patchset out is to request for comments to see 
>>>>>>whether
>>>>>>this implementation is acceptable or not. Patchset have been tested based 
>>>>>>on
>>>>>>xen-4.8 unstable on NXP i.MX8.
>>>>>>
>>>>>>I use Big/Little CPU and cpupool to explain the idea.
>>>>>>A pool contains Big CPUs is called Big Pool.
>>>>>>A pool contains Little CPUs is called Little Pool.
>>>>>>If a pool does not contains any physical cpus, Little CPUs or Big CPUs
>>>>>>can be added to the cpupool. But the cpupool can not contain both Little
>>>>>>and Big CPUs. The CPUs in a cpupool must have the same cpu type(midr 
>>>>>>value for ARM).
>>>>>>CPUs can not be added to the cpupool which contains cpus that have 
>>>>>>different cpu type.
>>>>>>Little CPUs can not be moved to Big Pool if there are Big CPUs in Big 
>>>>>>Pool,
>>>>>>and versa. Domain in Big Pool can not be migrated to Little Pool, and 
>>>>>>versa.
>>>>>>When XEN tries to bringup all the CPUs, only add CPUs with the same cpu 
>>>>>>type(same midr value)
>>>>>>into cpupool0.
>>>>>
>>>>>As mentioned in the mail you pointed above, this series is not enough to 
>>>>>make
>>>>>big.LITTLE working on then. Xen is always using the boot CPU to detect the
>>>>>list of features. With big.LITTLE features may not be the same.
>>>>>
>>>>>And I would prefer to see Xen supporting big.LITTLE correctly before
>>>>>beginning to think to expose big.LITTLE to the userspace (via cpupool)
>>>>>automatically.
>>>>
>>>>Do you mean vcpus be scheduled between big and little cpus freely?
>>>
>>>By supporting big.LITTLE correctly I meant Xen thinks that all the cores has
>>>the same set of features. So the feature detection is only done the boot CPU.
>>>See processor_setup for instance...
>>>
>>>Moving vCPUs between big and little cores would be a hard task (cache line
>>>issue, and possibly feature) and I don't expect to ever cross this in Xen.
>>>However, I am expecting to see big.LITTLE exposed to the guest (i.e having
>>>big and little vCPUs).
>>
>>big vCPUs scheduled on big Physical CPUs and little vCPUs scheduled on little
>>physical cpus, right?
>>If it is, is there is a need to let Xen think all the cores has the same set
>>of features?
>
>I think you missed my point. The feature registers on big and little cores
>may be different. Currently, Xen is reading the feature registers of the CPU
>boot and wrongly assumes that those features will exists on all CPUs. This is
>not the case and should be fixed before we are getting in trouble.
>
>>
>>Developing big.little guest support, I am not sure how much efforts needed.
>>Is this really needed?
>
>This is not necessary at the moment, although I have seen some interest about
>it. Running a guest only on a little core is a nice beginning, but a guest
>may want to take advantage of big.LITTLE (running hungry app on big one and
>little on small one).
>
>>
>>>
>>>>
>>>>This patchset is to use cpupool to block the vcpu be scheduled between big 
>>>>and
>>>>little cpus.
>>>>
>>>>>
>>>>>See for instance v->arch.actlr = READ_SYSREG32(ACTLR_EL1).
>>>>
>>>>Thanks for this. I only expose cpuid to guest, missed actlr. I'll check
>>>>the A53 and A72 TRM

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

Hello Julien,
On Mon, Sep 19, 2016 at 10:53:56AM +0200, Julien Grall wrote:
>Hello,
>
>On 19/09/2016 10:36, Peng Fan wrote:
>>On Mon, Sep 19, 2016 at 10:09:06AM +0200, Julien Grall wrote:
>>>Hello Peng,
>>>
>>>On 19/09/2016 04:08, van.free...@gmail.com wrote:
>>>>From: Peng Fan <peng@nxp.com>
>>>>
>>>>This patchset is to support XEN run on big.little SoC.
>>>>The idea of the patch is from
>>>>"https://lists.xenproject.org/archives/html/xen-devel/2016-05/msg00465.html;
>>>>
>>>>There are some changes to cpupool and add x86 stub functions to avoid build
>>>>break. Sending The RFC patchset out is to request for comments to see 
>>>>whether
>>>>this implementation is acceptable or not. Patchset have been tested based on
>>>>xen-4.8 unstable on NXP i.MX8.
>>>>
>>>>I use Big/Little CPU and cpupool to explain the idea.
>>>>A pool contains Big CPUs is called Big Pool.
>>>>A pool contains Little CPUs is called Little Pool.
>>>>If a pool does not contains any physical cpus, Little CPUs or Big CPUs
>>>>can be added to the cpupool. But the cpupool can not contain both Little
>>>>and Big CPUs. The CPUs in a cpupool must have the same cpu type(midr value 
>>>>for ARM).
>>>>CPUs can not be added to the cpupool which contains cpus that have 
>>>>different cpu type.
>>>>Little CPUs can not be moved to Big Pool if there are Big CPUs in Big Pool,
>>>>and versa. Domain in Big Pool can not be migrated to Little Pool, and versa.
>>>>When XEN tries to bringup all the CPUs, only add CPUs with the same cpu 
>>>>type(same midr value)
>>>>into cpupool0.
>>>
>>>As mentioned in the mail you pointed above, this series is not enough to make
>>>big.LITTLE working on then. Xen is always using the boot CPU to detect the
>>>list of features. With big.LITTLE features may not be the same.
>>>
>>>And I would prefer to see Xen supporting big.LITTLE correctly before
>>>beginning to think to expose big.LITTLE to the userspace (via cpupool)
>>>automatically.
>>
>>Do you mean vcpus be scheduled between big and little cpus freely?
>
>By supporting big.LITTLE correctly I meant Xen thinks that all the cores has
>the same set of features. So the feature detection is only done the boot CPU.
>See processor_setup for instance...
>
>Moving vCPUs between big and little cores would be a hard task (cache line
>issue, and possibly feature) and I don't expect to ever cross this in Xen.
>However, I am expecting to see big.LITTLE exposed to the guest (i.e having
>big and little vCPUs).
>
>>
>>This patchset is to use cpupool to block the vcpu be scheduled between big and
>>little cpus.
>>
>>>
>>>See for instance v->arch.actlr = READ_SYSREG32(ACTLR_EL1).

Back to this.
In xen/arch/arm/traps.c, I found that
"
WRITE_SYSREG(HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
 HCR_TWE|HCR_TWI|HCR_TSC|HCR_TAC|HCR_SWIO|HCR_TIDCP|HCR_FB,
 HCR_EL2);
"

HCR_TACR, HCR_TIDx is not set. HCR_TIDCP is set, but this is used to trap
implementation defined registers.

So accessing the actlr and cpu feature registers(exclude the implementation 
defined ones)
in guest os will not trap to xen, right?
If this is true, the actlr and cpu feature registers for DomU in Pool-A72 in my 
case
should be correct.

Thanks,
Peng.

>>
>>Thanks for this. I only expose cpuid to guest, missed actlr. I'll check
>>the A53 and A72 TRM about AArch64 implementationd defined registers.
>>This actlr can be added to the cpupool_arch_info as midr.
>>
>>Reading "vcpu_initialise", seems only MIDR and ACTLR needs to be handled.
>>Please advise if I missed anything else.
>
>Have you check the register emulation?


>
>>
>>>
>>>>
>>>>Thinking an SoC with 4 A53(cpu[0-3]) + 2 A72(cpu[4-5]), cpu0 is the first 
>>>>one
>>>>that boots up. When XEN tries to bringup secondary CPUs, add cpu[0-3] to
>>>>cpupool0 and leave cpu[4-5] not in any cpupool. Then when Dom0 boots up,
>>>>`xl cpupool-list -c` will show cpu[0-3] in Pool-0.
>>>>
>>>>Then use the following script to create a new cpupool and add cpu[4-5] to
>>>>the cpupool.
>>>>#xl cpupool-create name=\"Pool-A72\" sched=\"credit2\"
>>>>#xl cpupool-cpu-add Pool-A72 4
>>>>#xl cpupool-cpu-add Pool-A72 5
>>>>#xl create -d /root/xen/domu-test pool=\"Pool-A72\"

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

On Mon, Sep 19, 2016 at 10:53:56AM +0200, Julien Grall wrote:
>Hello,
>
>On 19/09/2016 10:36, Peng Fan wrote:
>>On Mon, Sep 19, 2016 at 10:09:06AM +0200, Julien Grall wrote:
>>>Hello Peng,
>>>
>>>On 19/09/2016 04:08, van.free...@gmail.com wrote:
>>>>From: Peng Fan <peng@nxp.com>
>>>>
>>>>This patchset is to support XEN run on big.little SoC.
>>>>The idea of the patch is from
>>>>"https://lists.xenproject.org/archives/html/xen-devel/2016-05/msg00465.html;
>>>>
>>>>There are some changes to cpupool and add x86 stub functions to avoid build
>>>>break. Sending The RFC patchset out is to request for comments to see 
>>>>whether
>>>>this implementation is acceptable or not. Patchset have been tested based on
>>>>xen-4.8 unstable on NXP i.MX8.
>>>>
>>>>I use Big/Little CPU and cpupool to explain the idea.
>>>>A pool contains Big CPUs is called Big Pool.
>>>>A pool contains Little CPUs is called Little Pool.
>>>>If a pool does not contains any physical cpus, Little CPUs or Big CPUs
>>>>can be added to the cpupool. But the cpupool can not contain both Little
>>>>and Big CPUs. The CPUs in a cpupool must have the same cpu type(midr value 
>>>>for ARM).
>>>>CPUs can not be added to the cpupool which contains cpus that have 
>>>>different cpu type.
>>>>Little CPUs can not be moved to Big Pool if there are Big CPUs in Big Pool,
>>>>and versa. Domain in Big Pool can not be migrated to Little Pool, and versa.
>>>>When XEN tries to bringup all the CPUs, only add CPUs with the same cpu 
>>>>type(same midr value)
>>>>into cpupool0.
>>>
>>>As mentioned in the mail you pointed above, this series is not enough to make
>>>big.LITTLE working on then. Xen is always using the boot CPU to detect the
>>>list of features. With big.LITTLE features may not be the same.
>>>
>>>And I would prefer to see Xen supporting big.LITTLE correctly before
>>>beginning to think to expose big.LITTLE to the userspace (via cpupool)
>>>automatically.
>>
>>Do you mean vcpus be scheduled between big and little cpus freely?
>
>By supporting big.LITTLE correctly I meant Xen thinks that all the cores has
>the same set of features. So the feature detection is only done the boot CPU.
>See processor_setup for instance...
>
>Moving vCPUs between big and little cores would be a hard task (cache line
>issue, and possibly feature) and I don't expect to ever cross this in Xen.
>However, I am expecting to see big.LITTLE exposed to the guest (i.e having
>big and little vCPUs).

big vCPUs scheduled on big Physical CPUs and little vCPUs scheduled on little
physical cpus, right?
If it is, is there is a need to let Xen think all the cores has the same set
of features?

Developing big.little guest support, I am not sure how much efforts needed.
Is this really needed? 

>
>>
>>This patchset is to use cpupool to block the vcpu be scheduled between big and
>>little cpus.
>>
>>>
>>>See for instance v->arch.actlr = READ_SYSREG32(ACTLR_EL1).
>>
>>Thanks for this. I only expose cpuid to guest, missed actlr. I'll check
>>the A53 and A72 TRM about AArch64 implementationd defined registers.
>>This actlr can be added to the cpupool_arch_info as midr.
>>
>>Reading "vcpu_initialise", seems only MIDR and ACTLR needs to be handled.
>>Please advise if I missed anything else.
>
>Have you check the register emulation?

Checked midr. Have not checked others.
I think I missed some registers in ctxt_switch_to.

>
>>
>>>
>>>>
>>>>Thinking an SoC with 4 A53(cpu[0-3]) + 2 A72(cpu[4-5]), cpu0 is the first 
>>>>one
>>>>that boots up. When XEN tries to bringup secondary CPUs, add cpu[0-3] to
>>>>cpupool0 and leave cpu[4-5] not in any cpupool. Then when Dom0 boots up,
>>>>`xl cpupool-list -c` will show cpu[0-3] in Pool-0.
>>>>
>>>>Then use the following script to create a new cpupool and add cpu[4-5] to
>>>>the cpupool.
>>>>#xl cpupool-create name=\"Pool-A72\" sched=\"credit2\"
>>>>#xl cpupool-cpu-add Pool-A72 4
>>>>#xl cpupool-cpu-add Pool-A72 5
>>>>#xl create -d /root/xen/domu-test pool=\"Pool-A72\"
>>>
>>>I am a bit confused with these runes. It means that only the first kind of
>>>CPUs have pool assigned. Why don't you directly create all the pools at boot
>>>time?
>>
>>

Re: [Xen-devel] [RFC 0/5] xen/arm: support big.little SoC

[Peng Fan]

Hello Julien,

On Mon, Sep 19, 2016 at 10:09:06AM +0200, Julien Grall wrote:
>Hello Peng,
>
>On 19/09/2016 04:08, van.free...@gmail.com wrote:
>>From: Peng Fan <peng@nxp.com>
>>
>>This patchset is to support XEN run on big.little SoC.
>>The idea of the patch is from
>>"https://lists.xenproject.org/archives/html/xen-devel/2016-05/msg00465.html;
>>
>>There are some changes to cpupool and add x86 stub functions to avoid build
>>break. Sending The RFC patchset out is to request for comments to see whether
>>this implementation is acceptable or not. Patchset have been tested based on
>>xen-4.8 unstable on NXP i.MX8.
>>
>>I use Big/Little CPU and cpupool to explain the idea.
>>A pool contains Big CPUs is called Big Pool.
>>A pool contains Little CPUs is called Little Pool.
>>If a pool does not contains any physical cpus, Little CPUs or Big CPUs
>>can be added to the cpupool. But the cpupool can not contain both Little
>>and Big CPUs. The CPUs in a cpupool must have the same cpu type(midr value 
>>for ARM).
>>CPUs can not be added to the cpupool which contains cpus that have different 
>>cpu type.
>>Little CPUs can not be moved to Big Pool if there are Big CPUs in Big Pool,
>>and versa. Domain in Big Pool can not be migrated to Little Pool, and versa.
>>When XEN tries to bringup all the CPUs, only add CPUs with the same cpu 
>>type(same midr value)
>>into cpupool0.
>
>As mentioned in the mail you pointed above, this series is not enough to make
>big.LITTLE working on then. Xen is always using the boot CPU to detect the
>list of features. With big.LITTLE features may not be the same.
>
>And I would prefer to see Xen supporting big.LITTLE correctly before
>beginning to think to expose big.LITTLE to the userspace (via cpupool)
>automatically.

Do you mean vcpus be scheduled between big and little cpus freely?

This patchset is to use cpupool to block the vcpu be scheduled between big and
little cpus.

>
>See for instance v->arch.actlr = READ_SYSREG32(ACTLR_EL1).

Thanks for this. I only expose cpuid to guest, missed actlr. I'll check
the A53 and A72 TRM about AArch64 implementationd defined registers.
This actlr can be added to the cpupool_arch_info as midr.

Reading "vcpu_initialise", seems only MIDR and ACTLR needs to be handled.
Please advise if I missed anything else.

>
>>
>>Thinking an SoC with 4 A53(cpu[0-3]) + 2 A72(cpu[4-5]), cpu0 is the first one
>>that boots up. When XEN tries to bringup secondary CPUs, add cpu[0-3] to
>>cpupool0 and leave cpu[4-5] not in any cpupool. Then when Dom0 boots up,
>>`xl cpupool-list -c` will show cpu[0-3] in Pool-0.
>>
>>Then use the following script to create a new cpupool and add cpu[4-5] to
>>the cpupool.
>> #xl cpupool-create name=\"Pool-A72\" sched=\"credit2\"
>> #xl cpupool-cpu-add Pool-A72 4
>> #xl cpupool-cpu-add Pool-A72 5
>> #xl create -d /root/xen/domu-test pool=\"Pool-A72\"
>
>I am a bit confused with these runes. It means that only the first kind of
>CPUs have pool assigned. Why don't you directly create all the pools at boot
>time?

If need to create all the pools, need to decided how many pools need to be 
created.
I thought about this, but I do not come out a good idea.

The cpupool0 is defined in xen/common/cpupool.c, if need to create many pools,
need to alloc cpupools dynamically when booting. I would not like to change a
lot to common code.

The implementation in this patchset I think is an easy way to let Big and Little
CPUs all run.

>
>Also, in which pool a domain will be created if none is specified?
>
>>Now `xl cpupool-list -c` shows:
>>NameCPU list
>>Pool-0  0,1,2,3
>>Pool-A724,5
>>
>>`xl cpupool-list` shows:
>>Name   CPUs   Sched Active   Domain count
>>Pool-0   4credit   y  1
>>Pool-A72 2   credit2   y  1
>>
>>`xl cpupool-cpu-remove Pool-A72 4`, then `xl cpupool-cpu-add Pool-0 4`
>>not success, because Pool-0 contains A53 CPUs, but CPU4 is an A72 CPU.
>>
>>`xl cpupool-migrate DomU Pool-0` will also fail, because DomU is created
>>in Pool-A72 with A72 vcpu, while Pool-0 have A53 physical cpus.
>>
>>Patch 1/5:
>>use "cpumask_weight(cpupool0->cpu_valid);" to replace "num_online_cpus()",
>>because num_online_cpus() counts all the online CPUs, but now we only
>>need Big or Little CPUs.
>
>So if I understand correctly, if the boot CPU is a little CPU, DOM0 will
>always be able to only use little ones. Is that right?

Yeah. Dom0 only use the little ones.

Thanks,
Peng.
>
>Regards,
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH V1] xen/arm: domain_build: introduce dom0_lowmem bootargs

[Peng Fan]

Hi Edgar,
On Thu, Sep 15, 2016 at 10:26:46AM +0200, Edgar E. Iglesias wrote:
>On Thu, Sep 15, 2016 at 08:20:33AM +0800, Peng Fan wrote:
>> Hi Edgar,
>> On Wed, Sep 14, 2016 at 04:16:58PM +0200, Edgar E. Iglesias wrote:
>> >On Wed, Sep 14, 2016 at 08:40:09PM +0800, Peng Fan wrote:
>> >> On Wed, Sep 14, 2016 at 01:34:10PM +0100, Julien Grall wrote:
>> >> >
>> >> >
>> >> >On 14/09/16 13:18, Peng Fan wrote:
>> >> >>Hello Julien,
>> >> >>
>> >> >>On Wed, Sep 14, 2016 at 01:06:01PM +0100, Julien Grall wrote:
>> >> >>>
>> >> >>>
>> >> >>>On 14/09/16 13:03, Peng Fan wrote:
>> >> >>>>Hello Julien,
>> >> >>>
>> >> >>>Hello Peng,
>> >> >>>
>> >> >>>>On Wed, Sep 14, 2016 at 11:47:10AM +0100, Julien Grall wrote:
>> >> >>>>>Hello,
>> >> >>>>>
>> >> >>>>>On 14/09/16 08:41, Peng Fan wrote:
>> >> >>>>>>On Wed, Sep 14, 2016 at 08:23:24AM +0100, Julien Grall wrote:
>> >> >>>>>>diff --git a/xen/arch/arm/domain_build.c 
>> >> >>>>>>b/xen/arch/arm/domain_build.c
>> >> >>>>>>index 35ab08d..cc71e6f 100644
>> >> >>>>>>--- a/xen/arch/arm/domain_build.c
>> >> >>>>>>+++ b/xen/arch/arm/domain_build.c
>> >> >>>>>>@@ -28,6 +28,8 @@
>> >> >>>>>>
>> >> >>>>>>static unsigned int __initdata opt_dom0_max_vcpus;
>> >> >>>>>>integer_param("dom0_max_vcpus", opt_dom0_max_vcpus);
>> >> >>>>>>+static bool_t __initdata opt_dom0_use_lowmem;
>> >> >>>>>>+boolean_param("dom0_use_lowmem", opt_dom0_use_lowmem);
>> >> >>>>>>
>> >> >>>>>>int dom0_11_mapping = 1;
>> >> >>>>>>
>> >> >>>>>>@@ -244,7 +246,7 @@ static void allocate_memory(struct domain *d, 
>> >> >>>>>>struct
>> >> >>>>>>kernel_info *kinfo)
>> >> >>>>>>   unsigned int order = 
>> >> >>>>>> get_11_allocation_size(kinfo->unassigned_mem);
>> >> >>>>>>   int i;
>> >> >>>>>>
>> >> >>>>>>-bool_t lowmem = is_32bit_domain(d);
>> >> >>>>>>+bool_t lowmem = is_32bit_domain(d) || opt_dom0_use_lowmem;
>> >> >>>>>>   unsigned int bits;
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>>Pass "dom0_use_lowmem=1" to xen to allocate lowmem as much as 
>> >> >>>>>>possible.
>> >> >>>>>
>> >> >>>>>Again, what is the benefit to have a command line option for that?
>> >> >>>>
>> >> >>>>Then you prefer directly change "bool_t lowmem = is_32bit_domain(d);" 
>> >> >>>>to "bool_t lowmem = true" ?
>> >> >>>>I just want to give user a choice.
>> >> >>>
>> >> >>>We don't add new command line parameter just because they look cool to 
>> >> >>>have.
>> >> >>>So far, you did not explain why it would be good to let the choice to 
>> >> >>>the
>> >> >>>user and how it could be used.
>> >> >>
>> >> >>I have not try, if there is no lowmem.
>> >> >>
>> >> >>I have not look into alloc_domheap_pages.
>> >> >>I am not sure whether there is such a platform or not,
>> >> >>just thinking if there is soc that dram memory starts from 4GB, and 
>> >> >>there is no dram
>> >> >>below 4GB. If we still can get memory when lowmem is true, I am ok to 
>> >> >>change directly assign
>> >> >>lowmem with value true. Anyway I have not look into the internals of 
>> >> >>domheap and
>> >> >>not sure whether there is such a platform that no lowmem (:-
>> >> >
>> >> >We cannot exclude this possibility. However, the only reason that Xen is
>> 

Re: [Xen-devel] [PATCH V1] xen/arm: domain_build: introduce dom0_lowmem bootargs

[Peng Fan]

Hi Edgar,
On Wed, Sep 14, 2016 at 04:16:58PM +0200, Edgar E. Iglesias wrote:
>On Wed, Sep 14, 2016 at 08:40:09PM +0800, Peng Fan wrote:
>> On Wed, Sep 14, 2016 at 01:34:10PM +0100, Julien Grall wrote:
>> >
>> >
>> >On 14/09/16 13:18, Peng Fan wrote:
>> >>Hello Julien,
>> >>
>> >>On Wed, Sep 14, 2016 at 01:06:01PM +0100, Julien Grall wrote:
>> >>>
>> >>>
>> >>>On 14/09/16 13:03, Peng Fan wrote:
>> >>>>Hello Julien,
>> >>>
>> >>>Hello Peng,
>> >>>
>> >>>>On Wed, Sep 14, 2016 at 11:47:10AM +0100, Julien Grall wrote:
>> >>>>>Hello,
>> >>>>>
>> >>>>>On 14/09/16 08:41, Peng Fan wrote:
>> >>>>>>On Wed, Sep 14, 2016 at 08:23:24AM +0100, Julien Grall wrote:
>> >>>>>>diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>> >>>>>>index 35ab08d..cc71e6f 100644
>> >>>>>>--- a/xen/arch/arm/domain_build.c
>> >>>>>>+++ b/xen/arch/arm/domain_build.c
>> >>>>>>@@ -28,6 +28,8 @@
>> >>>>>>
>> >>>>>>static unsigned int __initdata opt_dom0_max_vcpus;
>> >>>>>>integer_param("dom0_max_vcpus", opt_dom0_max_vcpus);
>> >>>>>>+static bool_t __initdata opt_dom0_use_lowmem;
>> >>>>>>+boolean_param("dom0_use_lowmem", opt_dom0_use_lowmem);
>> >>>>>>
>> >>>>>>int dom0_11_mapping = 1;
>> >>>>>>
>> >>>>>>@@ -244,7 +246,7 @@ static void allocate_memory(struct domain *d, 
>> >>>>>>struct
>> >>>>>>kernel_info *kinfo)
>> >>>>>>   unsigned int order = get_11_allocation_size(kinfo->unassigned_mem);
>> >>>>>>   int i;
>> >>>>>>   
>> >>>>>>-bool_t lowmem = is_32bit_domain(d);
>> >>>>>>+bool_t lowmem = is_32bit_domain(d) || opt_dom0_use_lowmem;
>> >>>>>>   unsigned int bits;
>> >>>>>>
>> >>>>>>
>> >>>>>>Pass "dom0_use_lowmem=1" to xen to allocate lowmem as much as possible.
>> >>>>>
>> >>>>>Again, what is the benefit to have a command line option for that?
>> >>>>
>> >>>>Then you prefer directly change "bool_t lowmem = is_32bit_domain(d);" to 
>> >>>>"bool_t lowmem = true" ?
>> >>>>I just want to give user a choice.
>> >>>
>> >>>We don't add new command line parameter just because they look cool to 
>> >>>have.
>> >>>So far, you did not explain why it would be good to let the choice to the
>> >>>user and how it could be used.
>> >>
>> >>I have not try, if there is no lowmem.
>> >>
>> >>I have not look into alloc_domheap_pages.
>> >>I am not sure whether there is such a platform or not,
>> >>just thinking if there is soc that dram memory starts from 4GB, and there 
>> >>is no dram
>> >>below 4GB. If we still can get memory when lowmem is true, I am ok to 
>> >>change directly assign
>> >>lowmem with value true. Anyway I have not look into the internals of 
>> >>domheap and
>> >>not sure whether there is such a platform that no lowmem (:-
>> >
>> >We cannot exclude this possibility. However, the only reason that Xen is
>> >requiring to allocate a bank below 4GB for 32-bit domain is to handle
>> >non-LPAE kernel.
>> 
>> Now also need to handle device that have DMA limitation -:)
>
>Hi Peng,
>
>Doesn't your platform have an IOMMU/SMMU?

We have SMMU. This is not related to SMMU. Dom0 use 1:1 mapping and no SMMU 
involved,
the physical memory assigned to Dom0 maybe higher than 4GB, but Some IPs only
supports 32bits DMA in Dom0. Then assign a 64bits dma address to a device only 
supports 32
bits device in Linux will hang the device or else.

Regards,
Peng.
>
>Cheers,
>Edgar

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH V1] xen/arm: domain_build: introduce dom0_lowmem bootargs

[Peng Fan]

On Wed, Sep 14, 2016 at 01:34:10PM +0100, Julien Grall wrote:
>
>
>On 14/09/16 13:18, Peng Fan wrote:
>>Hello Julien,
>>
>>On Wed, Sep 14, 2016 at 01:06:01PM +0100, Julien Grall wrote:
>>>
>>>
>>>On 14/09/16 13:03, Peng Fan wrote:
>>>>Hello Julien,
>>>
>>>Hello Peng,
>>>
>>>>On Wed, Sep 14, 2016 at 11:47:10AM +0100, Julien Grall wrote:
>>>>>Hello,
>>>>>
>>>>>On 14/09/16 08:41, Peng Fan wrote:
>>>>>>On Wed, Sep 14, 2016 at 08:23:24AM +0100, Julien Grall wrote:
>>>>>>diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>>>>>>index 35ab08d..cc71e6f 100644
>>>>>>--- a/xen/arch/arm/domain_build.c
>>>>>>+++ b/xen/arch/arm/domain_build.c
>>>>>>@@ -28,6 +28,8 @@
>>>>>>
>>>>>>static unsigned int __initdata opt_dom0_max_vcpus;
>>>>>>integer_param("dom0_max_vcpus", opt_dom0_max_vcpus);
>>>>>>+static bool_t __initdata opt_dom0_use_lowmem;
>>>>>>+boolean_param("dom0_use_lowmem", opt_dom0_use_lowmem);
>>>>>>
>>>>>>int dom0_11_mapping = 1;
>>>>>>
>>>>>>@@ -244,7 +246,7 @@ static void allocate_memory(struct domain *d, struct
>>>>>>kernel_info *kinfo)
>>>>>>   unsigned int order = get_11_allocation_size(kinfo->unassigned_mem);
>>>>>>   int i;
>>>>>>  
>>>>>>-bool_t lowmem = is_32bit_domain(d);
>>>>>>+bool_t lowmem = is_32bit_domain(d) || opt_dom0_use_lowmem;
>>>>>>   unsigned int bits;
>>>>>>
>>>>>>
>>>>>>Pass "dom0_use_lowmem=1" to xen to allocate lowmem as much as possible.
>>>>>
>>>>>Again, what is the benefit to have a command line option for that?
>>>>
>>>>Then you prefer directly change "bool_t lowmem = is_32bit_domain(d);" to 
>>>>"bool_t lowmem = true" ?
>>>>I just want to give user a choice.
>>>
>>>We don't add new command line parameter just because they look cool to have.
>>>So far, you did not explain why it would be good to let the choice to the
>>>user and how it could be used.
>>
>>I have not try, if there is no lowmem.
>>
>>I have not look into alloc_domheap_pages.
>>I am not sure whether there is such a platform or not,
>>just thinking if there is soc that dram memory starts from 4GB, and there is 
>>no dram
>>below 4GB. If we still can get memory when lowmem is true, I am ok to change 
>>directly assign
>>lowmem with value true. Anyway I have not look into the internals of domheap 
>>and
>>not sure whether there is such a platform that no lowmem (:-
>
>We cannot exclude this possibility. However, the only reason that Xen is
>requiring to allocate a bank below 4GB for 32-bit domain is to handle
>non-LPAE kernel.

Now also need to handle device that have DMA limitation -:)

>
>I personally don't think this is a hard requirement and instead of panicking
>we should print a warning to say "no bank has been allocated below 4GB" or "a
>bank of N MB has been allocated below 4GB".
>
>So my suggestion is to allocate as much as possible lowmem (i.e below 4GB)
>and if it does not work print a warning then allocate the first bank above
>4GB.
>
>I much prefer to let Xen decide itself what to do when possible rather than
>asking the user to specify himself whether lowmem is required.
>
>This is making life easier for distribution to support multiple platforms
>with Xen without having to modify the command line.

Thanks. I'll follow you suggestion and do some test. Then send out a
new patch.

Thanks,
Peng.

>
>Regards,
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH V1] xen/arm: domain_build: introduce dom0_lowmem bootargs

[Peng Fan]

Hello Julien,

On Wed, Sep 14, 2016 at 01:06:01PM +0100, Julien Grall wrote:
>
>
>On 14/09/16 13:03, Peng Fan wrote:
>>Hello Julien,
>
>Hello Peng,
>
>>On Wed, Sep 14, 2016 at 11:47:10AM +0100, Julien Grall wrote:
>>>Hello,
>>>
>>>On 14/09/16 08:41, Peng Fan wrote:
>>>>On Wed, Sep 14, 2016 at 08:23:24AM +0100, Julien Grall wrote:
>>>>diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>>>>index 35ab08d..cc71e6f 100644
>>>>--- a/xen/arch/arm/domain_build.c
>>>>+++ b/xen/arch/arm/domain_build.c
>>>>@@ -28,6 +28,8 @@
>>>>
>>>>static unsigned int __initdata opt_dom0_max_vcpus;
>>>>integer_param("dom0_max_vcpus", opt_dom0_max_vcpus);
>>>>+static bool_t __initdata opt_dom0_use_lowmem;
>>>>+boolean_param("dom0_use_lowmem", opt_dom0_use_lowmem);
>>>>
>>>>int dom0_11_mapping = 1;
>>>>
>>>>@@ -244,7 +246,7 @@ static void allocate_memory(struct domain *d, struct
>>>>kernel_info *kinfo)
>>>>unsigned int order = get_11_allocation_size(kinfo->unassigned_mem);
>>>>int i;
>>>>
>>>>-bool_t lowmem = is_32bit_domain(d);
>>>>+bool_t lowmem = is_32bit_domain(d) || opt_dom0_use_lowmem;
>>>>unsigned int bits;
>>>>
>>>>
>>>>Pass "dom0_use_lowmem=1" to xen to allocate lowmem as much as possible.
>>>
>>>Again, what is the benefit to have a command line option for that?
>>
>>Then you prefer directly change "bool_t lowmem = is_32bit_domain(d);" to 
>>"bool_t lowmem = true" ?
>>I just want to give user a choice.
>
>We don't add new command line parameter just because they look cool to have.
>So far, you did not explain why it would be good to let the choice to the
>user and how it could be used.

I have not try, if there is no lowmem.

I have not look into alloc_domheap_pages.
I am not sure whether there is such a platform or not,
just thinking if there is soc that dram memory starts from 4GB, and there is no 
dram
below 4GB. If we still can get memory when lowmem is true, I am ok to change 
directly assign
lowmem with value true. Anyway I have not look into the internals of domheap and
not sure whether there is such a platform that no lowmem (:-

while ( order >= min_low_order )
{
for ( bits = order ; bits <= (lowmem ? 32 : PADDR_BITS); bits++ )
{
pg = alloc_domheap_pages(d, order, MEMF_bits(bits));
if ( pg != NULL )
goto got_bank0;
}
order--;
}

panic("Unable to allocate first memory bank");

Thanks,
Peng.
>
>Regards,
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH V1] xen/arm: domain_build: introduce dom0_lowmem bootargs

[Peng Fan]

Hello Julien,
On Wed, Sep 14, 2016 at 11:47:10AM +0100, Julien Grall wrote:
>Hello,
>
>On 14/09/16 08:41, Peng Fan wrote:
>>On Wed, Sep 14, 2016 at 08:23:24AM +0100, Julien Grall wrote:
>>diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>>index 35ab08d..cc71e6f 100644
>>--- a/xen/arch/arm/domain_build.c
>>+++ b/xen/arch/arm/domain_build.c
>>@@ -28,6 +28,8 @@
>>
>>static unsigned int __initdata opt_dom0_max_vcpus;
>>integer_param("dom0_max_vcpus", opt_dom0_max_vcpus);
>>+static bool_t __initdata opt_dom0_use_lowmem;
>>+boolean_param("dom0_use_lowmem", opt_dom0_use_lowmem);
>>
>>int dom0_11_mapping = 1;
>>
>>@@ -244,7 +246,7 @@ static void allocate_memory(struct domain *d, struct
>>kernel_info *kinfo)
>> unsigned int order = get_11_allocation_size(kinfo->unassigned_mem);
>> int i;
>>  
>>-bool_t lowmem = is_32bit_domain(d);
>>+bool_t lowmem = is_32bit_domain(d) || opt_dom0_use_lowmem;
>> unsigned int bits;
>>
>>
>>Pass "dom0_use_lowmem=1" to xen to allocate lowmem as much as possible.
>
>Again, what is the benefit to have a command line option for that?

Then you prefer directly change "bool_t lowmem = is_32bit_domain(d);" to 
"bool_t lowmem = true" ?
I just want to give user a choice.

Thanks,
Peng.

>
>Regards,
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH V1] xen/arm: domain_build: introduce dom0_lowmem bootargs

[Peng Fan]

Hello Julien,
On Wed, Sep 14, 2016 at 08:23:24AM +0100, Julien Grall wrote:
>Hello,
>
>On 14/09/2016 06:12, Peng Fan wrote:
>>On AArch64 SoCs, some IPs may only have the capability to access
>>32bits address space. The physical memory assigned for Dom0 maybe
>>not in 4GB address space, then the IPs will not work properly.
>>
>>Introduce dom0_lowmem bootargs, user could pass "dom0_lowmem=xx"
>>to xen. It means how much memory user would like to be allocated
>>in lower 4GB memory space. If there is not enough memory for
>>dom0_lowmem, higher memory will be allocated.
>>
>>Thinking such a memory layout on an AArch64 SoC:
>>Region 0: 2GB(0x8000 - 0x)
>>Region 1: 4GB(0x88000 - 0x97fff)
>>If user would like to assign 2GB for Dom0 and 1GB of the 2GB memory
>>in Region 0, user could pass "dom0=2048M dom0_lowmem=1024M" to xen.
>
>See my comments on v1, I still don't think this new parameter is useful.

Thanks for comments. I see :)

>
>The commit message does not explain what is the advantage to only allocate a
>bit of low mem and not as much as we can (as we do on for 32-bit dom0).

This patch is just want to resolve the issue in
https://lists.xen.org/archives/html/xen-devel/2016-09/msg00235.html.
I am not keen on the method in this patch, a bit complicated.
The method in this patch is try to allocate the size lowmem indicated in 
bootargs.

Rethinking about DomU, seems allocate as much as possible lowmem for Dom0 is ok.

Then, do you agree to use the following patch?
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 35ab08d..cc71e6f 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -28,6 +28,8 @@
 
static unsigned int __initdata opt_dom0_max_vcpus;
integer_param("dom0_max_vcpus", opt_dom0_max_vcpus);
+static bool_t __initdata opt_dom0_use_lowmem;
+boolean_param("dom0_use_lowmem", opt_dom0_use_lowmem);

int dom0_11_mapping = 1;
  
@@ -244,7 +246,7 @@ static void allocate_memory(struct domain *d, struct 
kernel_info *kinfo)
 unsigned int order = get_11_allocation_size(kinfo->unassigned_mem);
 int i;

-bool_t lowmem = is_32bit_domain(d);
+bool_t lowmem = is_32bit_domain(d) || opt_dom0_use_lowmem;
 unsigned int bits;


Pass "dom0_use_lowmem=1" to xen to allocate lowmem as much as possible.

>
>>
>>Signed-off-by: Peng Fan <peng@nxp.com>
>>Cc: Stefano Stabellini <sstabell...@kernel.org>
>>Cc: Julien Grall <julien.gr...@arm.com>
>>---
>>
>>RFC->V1:
>> This patch is to resolve the issue in 
>> https://lists.xen.org/archives/html/xen-devel/2016-09/msg00235.html
>> No code change since RFC.
>> Tested on xen-4.8 unstable with AArch64. See partial log:
>> "dom0_mem = 2048M  dom0_lowmem=128M"
>> (XEN) Allocated 0x008800-0x009000 (128MB/2048MB, order 15)
>> (XEN) Allocated 0x088000-0x08c000 (1024MB/1920MB, order 18)
>> (XEN) Allocated 0x09c000-0x09e000 (512MB/896MB, order 17)
>> (XEN) Allocated 0x09e000-0x09f000 (256MB/384MB, order 16)
>> (XEN) Allocated 0x08f800-0x09 (128MB/128MB, order 15)
>>
>> "dom0_mem = 2048M  dom0_lowmem=1024M"
>> (XEN) Allocated 0x00a000-0x00c000 (512MB/2048MB, order 17)
>> (XEN) Allocated 0x00c000-0x00e000 (512MB/1536MB, order 17)
>> (XEN) Allocated 0x088000-0x08c000 (1024MB/1024MB, order 18)
>>
>> "dom0_mem = 1024M  dom0_lowmem=1024M"
>> (XEN) Allocated 0x00a000-0x00c000 (512MB/1024MB, order 17)
>> (XEN) Allocated 0x00c000-0x00e000 (512MB/512MB, order 17)
>>
>> xen/arch/arm/domain_build.c | 30 +-
>> 1 file changed, 29 insertions(+), 1 deletion(-)
>>
>>diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>>index 35ab08d..0f53bba 100644
>>--- a/xen/arch/arm/domain_build.c
>>+++ b/xen/arch/arm/domain_build.c
>>@@ -33,6 +33,8 @@ int dom0_11_mapping = 1;
>>
>> #define DOM0_MEM_DEFAULT 0x800 /* 128 MiB */
>> static u64 __initdata dom0_mem = DOM0_MEM_DEFAULT;
>>+/* Only for AArch64 */
>>+static u64 __initdata dom0_lowmem;
>>
>> static void __init parse_dom0_mem(const char *s)
>> {
>>@@ -42,6 +44,12 @@ static void __init parse_dom0_mem(const char *s)
>> }
>> custom_param("dom0_mem", parse_dom0_mem);
>>
>>+static void __init parse_dom0_lowmem(const char *s)
>>+{
>>+dom0_lowmem = parse_size_and_unit(s, );
>>+}
>>+custom_param("dom0_l

[Xen-devel] [PATCH V1] xen/arm: domain_build: introduce dom0_lowmem bootargs

[Peng Fan]

On AArch64 SoCs, some IPs may only have the capability to access
32bits address space. The physical memory assigned for Dom0 maybe
not in 4GB address space, then the IPs will not work properly.

Introduce dom0_lowmem bootargs, user could pass "dom0_lowmem=xx"
to xen. It means how much memory user would like to be allocated
in lower 4GB memory space. If there is not enough memory for
dom0_lowmem, higher memory will be allocated.

Thinking such a memory layout on an AArch64 SoC:
Region 0: 2GB(0x8000 - 0x)
Region 1: 4GB(0x88000 - 0x97fff)
If user would like to assign 2GB for Dom0 and 1GB of the 2GB memory
in Region 0, user could pass "dom0=2048M dom0_lowmem=1024M" to xen.

Signed-off-by: Peng Fan <peng@nxp.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
Cc: Julien Grall <julien.gr...@arm.com>
---

RFC->V1:
 This patch is to resolve the issue in 
https://lists.xen.org/archives/html/xen-devel/2016-09/msg00235.html
 No code change since RFC.
 Tested on xen-4.8 unstable with AArch64. See partial log:
 "dom0_mem = 2048M  dom0_lowmem=128M"
 (XEN) Allocated 0x008800-0x009000 (128MB/2048MB, order 15)
 (XEN) Allocated 0x088000-0x08c000 (1024MB/1920MB, order 18)
 (XEN) Allocated 0x09c000-0x09e000 (512MB/896MB, order 17)
 (XEN) Allocated 0x09e000-0x09f000 (256MB/384MB, order 16)
 (XEN) Allocated 0x08f800-0x09 (128MB/128MB, order 15)

 "dom0_mem = 2048M  dom0_lowmem=1024M"
 (XEN) Allocated 0x00a000-0x00c000 (512MB/2048MB, order 17)
 (XEN) Allocated 0x00c000-0x00e000 (512MB/1536MB, order 17)
 (XEN) Allocated 0x088000-0x08c000 (1024MB/1024MB, order 18)

 "dom0_mem = 1024M  dom0_lowmem=1024M"
 (XEN) Allocated 0x00a000-0x00c000 (512MB/1024MB, order 17)
 (XEN) Allocated 0x00c000-0x00e000 (512MB/512MB, order 17)

 xen/arch/arm/domain_build.c | 30 +-
 1 file changed, 29 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 35ab08d..0f53bba 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -33,6 +33,8 @@ int dom0_11_mapping = 1;
 
 #define DOM0_MEM_DEFAULT 0x800 /* 128 MiB */
 static u64 __initdata dom0_mem = DOM0_MEM_DEFAULT;
+/* Only for AArch64 */
+static u64 __initdata dom0_lowmem;
 
 static void __init parse_dom0_mem(const char *s)
 {
@@ -42,6 +44,12 @@ static void __init parse_dom0_mem(const char *s)
 }
 custom_param("dom0_mem", parse_dom0_mem);
 
+static void __init parse_dom0_lowmem(const char *s)
+{
+dom0_lowmem = parse_size_and_unit(s, );
+}
+custom_param("dom0_lowmem", parse_dom0_lowmem);
+
 //#define DEBUG_11_ALLOCATION
 #ifdef DEBUG_11_ALLOCATION
 # define D11PRINT(fmt, args...) printk(XENLOG_DEBUG fmt, ##args)
@@ -244,7 +252,7 @@ static void allocate_memory(struct domain *d, struct 
kernel_info *kinfo)
 unsigned int order = get_11_allocation_size(kinfo->unassigned_mem);
 int i;
 
-bool_t lowmem = is_32bit_domain(d);
+bool_t lowmem = is_32bit_domain(d) || !!dom0_lowmem;
 unsigned int bits;
 
 /*
@@ -263,6 +271,9 @@ static void allocate_memory(struct domain *d, struct 
kernel_info *kinfo)
  * First try and allocate the largest thing we can as low as
  * possible to be bank 0.
  */
+if ( dom0_lowmem )
+order = get_order_from_bytes(dom0_lowmem);
+
 while ( order >= min_low_order )
 {
 for ( bits = order ; bits <= (lowmem ? 32 : PADDR_BITS); bits++ )
@@ -278,6 +289,11 @@ static void allocate_memory(struct domain *d, struct 
kernel_info *kinfo)
 
  got_bank0:
 
+if ( dom0_lowmem ) {
+dom0_lowmem -= pfn_to_paddr((1 << order));
+lowmem = is_32bit_domain(d) || !!dom0_lowmem;
+}
+
 if ( !insert_11_bank(d, kinfo, pg, order) )
 BUG(); /* Cannot fail for first bank */
 
@@ -325,6 +341,16 @@ static void allocate_memory(struct domain *d, struct 
kernel_info *kinfo)
 }
 }
 
+if ( dom0_lowmem && lowmem )
+{
+dom0_lowmem -= pfn_to_paddr((1 << order));
+lowmem = is_32bit_domain(d) || !!dom0_lowmem;
+}
+else
+{
+lowmem = false;
+}
+
 /*
  * Success, next time around try again to get the largest order
  * allocation possible.
@@ -2098,6 +2124,8 @@ int construct_dom0(struct domain *d)
 
 d->max_pages = ~0U;
 
+BUG_ON(dom0_mem < dom0_lowmem);
+
 kinfo.unassigned_mem = dom0_mem;
 
 rc = kernel_probe();
-- 
2.6.6


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC] xen/arm: domain_build: introduce dom0_lowmem bootargs

[Peng Fan]

On Tue, Sep 13, 2016 at 02:24:31PM +0100, Julien Grall wrote:
>
>
>On 13/09/16 14:12, Peng Fan wrote:
>>Hi Julien,
>>On Tue, Sep 13, 2016 at 01:59:01PM +0100, Julien Grall wrote:
>>>Hello Peng,
>>>
>>>On 13/09/16 13:55, Peng Fan wrote:
>>>>On AArch64 SoCs, some IPs may only have the capability to access
>>>>32bits address space. The physical memory assigned for Dom0 maybe
>>>>not in 4GB address space, then the IPs will not work properly.
>>>>
>>>>Introduce dom0_lowmem bootargs, user could pass "dom0_lowmem=xx"
>>>>to xen. It means how much memory user would like to be allocated
>>>>in lower 4GB memory space. If there is not enough memory for
>>>>dom0_lowmem, higher memory will be allocated.
>>>>
>>>>Thinking such a memory layout on an AArch64 SoC:
>>>>Region 0: 2GB(0x8000 - 0x)
>>>>Region 1: 4GB(0x88000 - 0x97fff)
>>>>If user would like to assign 2GB for Dom0 and 1GB of the 2GB memory
>>>>in Region 0, user could pass "dom0=2048M dom0_lowmem=1024M" to xen.
>>>>
>>>>Signed-off-by: Peng Fan <peng@nxp.com>
>>>>Cc: Stefano Stabellini <sstabell...@kernel.org>
>>>>Cc: Julien Grall <julien.gr...@arm.com>
>>>>---
>>>>
>>>>This patch is to resolve the issue mentioned in
>>>>https://lists.xen.org/archives/html/xen-devel/2016-09/msg00235.html
>>>>This patch not tested on latest 4.8-unstable, I only tested similar
>>>>patch on xen 4.7 on AArch64 platform.
>>>
>>>Please test any patch send upstream on 4.8-unstable. The code may have
>>>changed.
>>
>>I have rebase this patch based on 4.8-unstable.
>>latest commit:
>>"
>>commit a3fe74e4345e66ddb7aa514395260a5e5f8b0cdc
>>Author: Tamas K Lengyel <tamas.leng...@zentific.com>
>>Date:   Mon Aug 1 11:59:14 2016 -0600
>>
>>arm/vm_event: get/set registers
>>"
>>
>>Since I have not rebased my platform patches to 4.8-unstable, so have not 
>>tested.
>>
>>Please kindly comments whether introudcing "dom0_lowmem" is acceptable or not
>>to resolve the issue I met in 
>>https://lists.xen.org/archives/html/xen-devel/2016-09/msg00235.html.
>>
>>I'll rebase my platform patches and do some test.
>>
>>>
>>>>
>>>>The idea of patch is that user could specify the lowmem that user would
>>>>like to use. I rethought the ideas in 
>>>>https://lists.xen.org/archives/html/xen-devel/2016-09/msg00487.html,
>>>>but that is not good. lowmem is precise, it maybe used for some IPs that 
>>>>maybe
>>>>passthrough to DomU, so we only allocate the needed memory for Dom0.
>>>
>>>Why? IPs passthrough to DomU will have to be protected by an SMMU so it does
>>>not matter whether Dom0 is using all the lowmem or not.
>>
>>I just think there maybe some cases that some physical memory need to be
>>reserved for DomU usage.
>>SMMU is not a must when we passthrough a non DMA capable device to DomU.
>>So I think an DRAM area can be encoded in dts, and passthrough to DomU.
>
>How do you make sure that DomU will program the device with the correct
>address? A malicious DomU could decide to use a physical address belonging to
>another DomU or even Xen and would be able to read/write on it.

When I debug DomU, I use a uart port as an early console for DomU.
I just marked the uart with xen,passthrough and status disabled in Dom0 dts,
then in DomU dts, I create a uart node, and in xl cfg, I mapped the address
space. I did not use SMMU here. Just mapped the uart physical address
to DomU uart guest physical address. And DomU can access the uart for my
debug usage.

>
>So if a device needs to access the physical memory it either needs to be
>protected by an SMMU or Xen/DOM0 is programming the device on behalf of the
>guest. All other solutions are IHMO unsafe.

My understanding about reserve lowmem for DomU maybe not that correct.
I'll test this patch based on 4.8 and send out V2 later.

Thanks,
Peng.

>
>Regards,
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC] xen/arm: domain_build: introduce dom0_lowmem bootargs

[Peng Fan]

Hi Julien,
On Tue, Sep 13, 2016 at 01:59:01PM +0100, Julien Grall wrote:
>Hello Peng,
>
>On 13/09/16 13:55, Peng Fan wrote:
>>On AArch64 SoCs, some IPs may only have the capability to access
>>32bits address space. The physical memory assigned for Dom0 maybe
>>not in 4GB address space, then the IPs will not work properly.
>>
>>Introduce dom0_lowmem bootargs, user could pass "dom0_lowmem=xx"
>>to xen. It means how much memory user would like to be allocated
>>in lower 4GB memory space. If there is not enough memory for
>>dom0_lowmem, higher memory will be allocated.
>>
>>Thinking such a memory layout on an AArch64 SoC:
>>Region 0: 2GB(0x8000 - 0x)
>>Region 1: 4GB(0x88000 - 0x97fff)
>>If user would like to assign 2GB for Dom0 and 1GB of the 2GB memory
>>in Region 0, user could pass "dom0=2048M dom0_lowmem=1024M" to xen.
>>
>>Signed-off-by: Peng Fan <peng@nxp.com>
>>Cc: Stefano Stabellini <sstabell...@kernel.org>
>>Cc: Julien Grall <julien.gr...@arm.com>
>>---
>>
>>This patch is to resolve the issue mentioned in
>>https://lists.xen.org/archives/html/xen-devel/2016-09/msg00235.html
>>This patch not tested on latest 4.8-unstable, I only tested similar
>>patch on xen 4.7 on AArch64 platform.
>
>Please test any patch send upstream on 4.8-unstable. The code may have
>changed.

I have rebase this patch based on 4.8-unstable.
latest commit:
"
commit a3fe74e4345e66ddb7aa514395260a5e5f8b0cdc
Author: Tamas K Lengyel <tamas.leng...@zentific.com>
Date:   Mon Aug 1 11:59:14 2016 -0600

arm/vm_event: get/set registers
"

Since I have not rebased my platform patches to 4.8-unstable, so have not 
tested.

Please kindly comments whether introudcing "dom0_lowmem" is acceptable or not
to resolve the issue I met in 
https://lists.xen.org/archives/html/xen-devel/2016-09/msg00235.html.

I'll rebase my platform patches and do some test.

>
>>
>>The idea of patch is that user could specify the lowmem that user would
>>like to use. I rethought the ideas in 
>>https://lists.xen.org/archives/html/xen-devel/2016-09/msg00487.html,
>>but that is not good. lowmem is precise, it maybe used for some IPs that maybe
>>passthrough to DomU, so we only allocate the needed memory for Dom0.
>
>Why? IPs passthrough to DomU will have to be protected by an SMMU so it does
>not matter whether Dom0 is using all the lowmem or not.

I just think there maybe some cases that some physical memory need to be
reserved for DomU usage.
SMMU is not a must when we passthrough a non DMA capable device to DomU.
So I think an DRAM area can be encoded in dts, and passthrough to DomU.

Thanks,
Peng.
>
>Regards,
>
>>
>> xen/arch/arm/domain_build.c | 30 +-
>> 1 file changed, 29 insertions(+), 1 deletion(-)
>>
>>diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>>index 35ab08d..0f53bba 100644
>>--- a/xen/arch/arm/domain_build.c
>>+++ b/xen/arch/arm/domain_build.c
>>@@ -33,6 +33,8 @@ int dom0_11_mapping = 1;
>>
>> #define DOM0_MEM_DEFAULT 0x800 /* 128 MiB */
>> static u64 __initdata dom0_mem = DOM0_MEM_DEFAULT;
>>+/* Only for AArch64 */
>>+static u64 __initdata dom0_lowmem;
>>
>> static void __init parse_dom0_mem(const char *s)
>> {
>>@@ -42,6 +44,12 @@ static void __init parse_dom0_mem(const char *s)
>> }
>> custom_param("dom0_mem", parse_dom0_mem);
>>
>>+static void __init parse_dom0_lowmem(const char *s)
>>+{
>>+dom0_lowmem = parse_size_and_unit(s, );
>>+}
>>+custom_param("dom0_lowmem", parse_dom0_lowmem);
>>+
>> //#define DEBUG_11_ALLOCATION
>> #ifdef DEBUG_11_ALLOCATION
>> # define D11PRINT(fmt, args...) printk(XENLOG_DEBUG fmt, ##args)
>>@@ -244,7 +252,7 @@ static void allocate_memory(struct domain *d, struct 
>>kernel_info *kinfo)
>> unsigned int order = get_11_allocation_size(kinfo->unassigned_mem);
>> int i;
>>
>>-bool_t lowmem = is_32bit_domain(d);
>>+bool_t lowmem = is_32bit_domain(d) || !!dom0_lowmem;
>> unsigned int bits;
>>
>> /*
>>@@ -263,6 +271,9 @@ static void allocate_memory(struct domain *d, struct 
>>kernel_info *kinfo)
>>  * First try and allocate the largest thing we can as low as
>>  * possible to be bank 0.
>>  */
>>+if ( dom0_lowmem )
>>+order = get_order_from_bytes(dom0_lowmem);
>>+
>> while ( order >= min_low_order )
>> {
>> for ( bits = order ; bits <= (lowmem ? 32 : PADDR_BITS); bits++ 

[Xen-devel] [RFC] xen/arm: domain_build: introduce dom0_lowmem bootargs

[Peng Fan]

On AArch64 SoCs, some IPs may only have the capability to access
32bits address space. The physical memory assigned for Dom0 maybe
not in 4GB address space, then the IPs will not work properly.

Introduce dom0_lowmem bootargs, user could pass "dom0_lowmem=xx"
to xen. It means how much memory user would like to be allocated
in lower 4GB memory space. If there is not enough memory for
dom0_lowmem, higher memory will be allocated.

Thinking such a memory layout on an AArch64 SoC:
Region 0: 2GB(0x8000 - 0x)
Region 1: 4GB(0x88000 - 0x97fff)
If user would like to assign 2GB for Dom0 and 1GB of the 2GB memory
in Region 0, user could pass "dom0=2048M dom0_lowmem=1024M" to xen.

Signed-off-by: Peng Fan <peng@nxp.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
Cc: Julien Grall <julien.gr...@arm.com>
---

This patch is to resolve the issue mentioned in
https://lists.xen.org/archives/html/xen-devel/2016-09/msg00235.html
This patch not tested on latest 4.8-unstable, I only tested similar
patch on xen 4.7 on AArch64 platform.

The idea of patch is that user could specify the lowmem that user would
like to use. I rethought the ideas in 
https://lists.xen.org/archives/html/xen-devel/2016-09/msg00487.html,
but that is not good. lowmem is precise, it maybe used for some IPs that maybe
passthrough to DomU, so we only allocate the needed memory for Dom0.

 xen/arch/arm/domain_build.c | 30 +-
 1 file changed, 29 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 35ab08d..0f53bba 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -33,6 +33,8 @@ int dom0_11_mapping = 1;
 
 #define DOM0_MEM_DEFAULT 0x800 /* 128 MiB */
 static u64 __initdata dom0_mem = DOM0_MEM_DEFAULT;
+/* Only for AArch64 */
+static u64 __initdata dom0_lowmem;
 
 static void __init parse_dom0_mem(const char *s)
 {
@@ -42,6 +44,12 @@ static void __init parse_dom0_mem(const char *s)
 }
 custom_param("dom0_mem", parse_dom0_mem);
 
+static void __init parse_dom0_lowmem(const char *s)
+{
+dom0_lowmem = parse_size_and_unit(s, );
+}
+custom_param("dom0_lowmem", parse_dom0_lowmem);
+
 //#define DEBUG_11_ALLOCATION
 #ifdef DEBUG_11_ALLOCATION
 # define D11PRINT(fmt, args...) printk(XENLOG_DEBUG fmt, ##args)
@@ -244,7 +252,7 @@ static void allocate_memory(struct domain *d, struct 
kernel_info *kinfo)
 unsigned int order = get_11_allocation_size(kinfo->unassigned_mem);
 int i;
 
-bool_t lowmem = is_32bit_domain(d);
+bool_t lowmem = is_32bit_domain(d) || !!dom0_lowmem;
 unsigned int bits;
 
 /*
@@ -263,6 +271,9 @@ static void allocate_memory(struct domain *d, struct 
kernel_info *kinfo)
  * First try and allocate the largest thing we can as low as
  * possible to be bank 0.
  */
+if ( dom0_lowmem )
+order = get_order_from_bytes(dom0_lowmem);
+
 while ( order >= min_low_order )
 {
 for ( bits = order ; bits <= (lowmem ? 32 : PADDR_BITS); bits++ )
@@ -278,6 +289,11 @@ static void allocate_memory(struct domain *d, struct 
kernel_info *kinfo)
 
  got_bank0:
 
+if ( dom0_lowmem ) {
+dom0_lowmem -= pfn_to_paddr((1 << order));
+lowmem = is_32bit_domain(d) || !!dom0_lowmem;
+}
+
 if ( !insert_11_bank(d, kinfo, pg, order) )
 BUG(); /* Cannot fail for first bank */
 
@@ -325,6 +341,16 @@ static void allocate_memory(struct domain *d, struct 
kernel_info *kinfo)
 }
 }
 
+if ( dom0_lowmem && lowmem )
+{
+dom0_lowmem -= pfn_to_paddr((1 << order));
+lowmem = is_32bit_domain(d) || !!dom0_lowmem;
+}
+   else
+{
+lowmem = false;
+}
+
 /*
  * Success, next time around try again to get the largest order
  * allocation possible.
@@ -2098,6 +2124,8 @@ int construct_dom0(struct domain *d)
 
 d->max_pages = ~0U;
 
+BUG_ON(dom0_mem < dom0_lowmem);
+
 kinfo.unassigned_mem = dom0_mem;
 
 rc = kernel_probe();
-- 
2.6.6


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [PATCH V3] xen/arm: arm64: Update the Image header

[Peng Fan]

According to Linux Kernel, Documentation/arm64/booting.txt
"
When image_size is zero, a bootloader should attempt to keep as much
memory as possible free for use by the kernel immediately after the
end of the kernel image. The amount of space required will vary
depending on selected features, and is effectively unbound.
"
This will consumes some memory and time, for example,
When booting xen from U-Boot, U-Boot will use the image size
info. Because this information is lacked in XEN image,U-Boot
assume the image size is 16MB to memmove, which will cost lots
time on simulation platform.

The flags field is also filled with value 0xA,
Bit3(physical placement):   1
Bit2-1(Page size):  1
Bit0(endianness):   0

Signed-off-by: Peng Fan <peng@nxp.com>
Cc: Julien Grall <julien.gr...@arm.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
---

V3:
 Drop the image.h macros.h from Linux, included in V2.
 Only update image size and flags entry. offset was kept 0 as before.
 Only little endian supported.

V2:
 According to Linux Kernel, a2c1d73b94ed49 "arm64: Update the Image header",
 included unneccessary stuff.

 xen/arch/arm/arm64/head.S | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
index 91e2817..2cd3699 100644
--- a/xen/arch/arm/arm64/head.S
+++ b/xen/arch/arm/arm64/head.S
@@ -115,8 +115,8 @@ efi_head:
 add x13, x18, #0x16
 b   real_start   /* branch to kernel start */
 .quad   0/* Image load offset from start of RAM */
-.quad   0/* reserved */
-.quad   0/* reserved */
+   .quad   _end - start /* Effective size of kernel image, 
little-endian */
+   .quad   0xa  /* Informative flags(Physical placement 1, 
4KB, LE), little-endian */
 .quad   0/* reserved */
 .quad   0/* reserved */
 .quad   0/* reserved */
-- 
2.6.6


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH V2] xen/arm: arm64: Update the Image header

[Peng Fan]

Hi Julien,

On Fri, Sep 09, 2016 at 02:19:33PM +0100, Julien Grall wrote:
>Hello Peng,
>
>On 01/09/16 02:38, Peng Fan wrote:
>>This patch is mainly modified from Linux kernel:
>>[1] commit a2c1d73b94ed: arm64: Update the Image header
>>[2] commit 6ad1fe5d9077: arm64: avoid R_AARCH64_ABS64 relocations for Image 
>>header fields
>>
>>From [1]:
>>"
>>This patch adds an effective image size to the kernel header which
>>describes the amount of memory from the start of the kernel Image binary
>>which the kernel expects to use before detecting memory and handling any
>>memory reservations. This can be used by bootloaders to choose suitable
>>locations to load the kernel and/or other binaries such that the kernel
>>will not clobber any memory unexpectedly. As before, memory reservations
>>are required to prevent the kernel from clobbering these locations
>>later.
>>
>>Both the image load offset and the effective image size are forced to be
>>little-endian regardless of the native endianness of the kernel to
>>enable bootloaders to load a kernel of arbitrary endianness. Bootloaders
>>which wish to make use of the load offset can inspect the effective
>>image size field for a non-zero value to determine if the offset is of a
>>known endianness. To enable software to determine the endinanness of the
>>kernel as may be required for certain use-cases, a new flags field (also
>>little-endian) is added to the kernel header to export this information.
>>"
>>
>>In this patch, XEN_VIRT_START is used as the text offset. Then, bootloader,
>>such as U-Boot, will load the xen image to "dram_base + text_offset".
>>Not choose 0 as the text offset, because we may have spin table at dram_base.
>>Loading xen to dram_base will override the spin table.
>
>XEN_VIRT_START is *not* the text offset. It is the virtual address mark the
>start of Xen region when paging is enabled.
>
>Furthermore, your description here does not match the behavior of this patch.
>The kernel physical displacement is set to 1, this means the kernel will be
>loaded anywhere in the memory.
>
>It is the job of the bootloader to find an unused place to load Xen into the
>memory.

Agree. I'll keep the text offset 0 as before in V3.

>
>>
>>Introduce image.h and macros.h in this patch, just as Linux kernel.
>>
>>Signed-off-by: Peng Fan <peng@nxp.com>
>>Cc: Stefano Stabellini <sstabell...@kernel.org>
>>Cc: Julien Grall <julien.gr...@arm.com>
>>---
>>
>>V2: Addressing Julien's comments to follow linux kernel patch:
>>a2c1d73b94ed49 "arm64: Update the Image header"
>
>Whilst I suggested to update all the header fields (image load offset,
>effective size, flags...), I don't think we should import a "verbatim" of the
>Linux header image.h. It is really complex for a questionable benefit.

Ok. I'll drop the image.h from Linux Kernel in V3

>
>>
>> xen/arch/arm/arm64/head.S  |  7 +++--
>> xen/arch/arm/xen.lds.S |  5 +++
>> xen/include/asm-arm/arm64/image.h  | 62 
>> ++
>> xen/include/asm-arm/arm64/macros.h | 15 +
>> xen/include/asm-arm/macros.h   |  2 +-
>> 5 files changed, 87 insertions(+), 4 deletions(-)
>> create mode 100644 xen/include/asm-arm/arm64/image.h
>> create mode 100644 xen/include/asm-arm/arm64/macros.h
>>
>>diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
>>index 91e2817..0226463 100644
>>--- a/xen/arch/arm/arm64/head.S
>>+++ b/xen/arch/arm/arm64/head.S
>>@@ -21,6 +21,7 @@
>>  */
>>
>> #include 
>>+#include 
>> #include 
>> #include 
>> #include 
>>@@ -114,9 +115,9 @@ efi_head:
>>  */
>> add x13, x18, #0x16
>> b   real_start   /* branch to kernel start */
>>-.quad   0/* Image load offset from start of RAM 
>>*/
>>-.quad   0/* reserved */
>>-.quad   0/* reserved */
>>+le64sym _xen_offset_le   /* Image load offset from start of RAM, 
>>little-endian */
>>+le64sym _xen_size_le /* Effective size of kernel image, 
>>little-endian */
>>+le64sym _xen_flags_le/* Informative flags, little-endian */
>> .quad   0/* reserved */
>> .quad   0/* reserved */
>> .quad   0/* reserved */
>>diff --git a/xen/arch/arm/xen.lds.S b/xen/arch/arm/xen.lds.S
>>index b24e93b..854c243 100644
>>--- a/xen/arch/arm/xen.lds.S
>>+++ b/xen/arch/arm/xen.lds.S
>>@@ -6,6 +6,7 @@
>> #include 
>> #include 
>> #include 
>>+#include 
>
>Please don't include arm64 specific header in common code.

Fix in V3.

Thanks,
Peng.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] xen arm64 dom0 question

[Peng Fan]

Hi Julien,

On Fri, Sep 02, 2016 at 02:13:07PM +0100, Julien Grall wrote:
>
>
>On 02/09/16 12:27, Peng Fan wrote:
>>Hi Julien, Stefano
>
>Hi Peng,
>
>>
>>On My ARM64 platform, there is 6GB memory.
>>0x8000 - 0xfff: 2GB
>>0x88000 - 0x9: 4GB
>>
>>xen will alloc 1:1 mapping for Dom0 memory, so if I assign dom0_mem with a 
>>bigger
>>value, saying 2048MB or bigger. xen will alloc continus memory from higher 
>>address
>>space in the higer 4GB.
>>
>>But the SD controller in my ARM64 platform has a limitation that it can only
>>access 32bit address space. So if Dom0 memory is at higher 4GB, SD controller
>>can not work as expected, because it only works when the dma address is 32bit
>>address.
>>
>>So, Can we assign a hole in lower 32bit address space for Dom0 guest physical
>>memory, just as DomU? Dynamically find out a hole and size 128MB or bigger?
>>Or do you have any ideas?
>
>Looking at the allocation code (allocate_memory in arch/arm/domain_build.c),
>Xen is trying to allocate as much memory as possible below 4GB for 32-bit
>domain to accommodate non-LPAE kernel.
>
>We haven't though about devices that can only handle 32-bit address at that
>time. I think replacing "lowmen = is_32bit_domain(d)" by "lowmem = true"
>should do the job.
>
>Note that, a proper upstream patch would require to modify the description of
>the function and potentially kill lowmen (or gate it with a command line
>parameter?).

Thanks for reply. I pasted two patches here. Both patch is to solve this 
problem.

In patch 1, allocate_memory will try to allocate memory in lowmem as much
as possible.
My test log for patch 1:
(XEN) Allocated 0x00a000-0x00c000 (512MB/2048MB, order 17)
(XEN) Allocated 0x00c000-0x00e000 (512MB/1536MB, order 17)
(XEN) Allocated 0x009000-0x00a000 (256MB/1024MB, order 16)
(XEN) Allocated 0x00e000-0x00f000 (256MB/768MB, order 16)
(XEN) Allocated 0x008800-0x009000 (128MB/512MB, order 15)
(XEN) Allocated 0x00f000-0x00f800 (128MB/384MB, order 15)
(XEN) Failed at min_low_order, allow high allocations
(XEN) Allocated 0x09e000-0x09f000 (256MB/256MB, order 16)
(XEN) BANK[0] 0x008800-0x00f800 (1792MB)
(XEN) BANK[1] 0x09e000-0x09f000 (256MB)

1792MB allocated in lowmem space.

patch 1:
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 35ab08d..cc71e6f 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -28,6 +28,8 @@
 
 static unsigned int __initdata opt_dom0_max_vcpus;
 integer_param("dom0_max_vcpus", opt_dom0_max_vcpus);
+static unsigned int __initdata opt_dom0_use_lowmem;
+integer_param("dom0_use_lowmem", opt_dom0_use_lowmem);
 
 int dom0_11_mapping = 1;
 
@@ -244,7 +246,7 @@ static void allocate_memory(struct domain *d, struct 
kernel_info *kinfo)
 unsigned int order = get_11_allocation_size(kinfo->unassigned_mem);
 int i;
 
-bool_t lowmem = is_32bit_domain(d);
+bool_t lowmem = is_32bit_domain(d) || !!opt_dom0_use_lowmem;
 unsigned int bits;
 
 /*

In Patch 2, only alloacte lowmem in the first try and allocate memory
for bank0.
My test log:
(XEN) Allocated 0x00a000-0x00c000 (512MB/2048MB, order 17)
(XEN) Allocated 0x098000-0x09c000 (1024MB/1536MB, order 18)
(XEN) Allocated 0x09c000-0x09e000 (512MB/512MB, order 17)
(XEN) BANK[0] 0x00a000-0x00c000 (512MB)
(XEN) BANK[1] 0x098000-0x09e000 (1536MB)

512MB is allocated in lowmem.

patch 2:
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 35ab08d..ad5926a 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -28,6 +28,8 @@
 
 static unsigned int __initdata opt_dom0_max_vcpus;
 integer_param("dom0_max_vcpus", opt_dom0_max_vcpus);
+static unsigned int __initdata opt_dom0_use_lowmem;
+integer_param("dom0_use_lowmem", opt_dom0_use_lowmem);
 
 int dom0_11_mapping = 1;
 
@@ -265,7 +267,9 @@ static void allocate_memory(struct domain *d, struct 
kernel_info *kinfo)
  */
 while ( order >= min_low_order )
 {
-for ( bits = order ; bits <= (lowmem ? 32 : PADDR_BITS); bits++ )
+for ( bits = order ;
+  bits <= ((lowmem || !!opt_dom0_use_lowmem) ? 32 : PADDR_BITS);
+  bits++ )
 {
 pg = alloc_domheap_pages(d, order, MEMF_bits(bits));
 if ( pg != NULL )


I prefer patch 2, because I only need some lowmem for DMA. The method of patch 1
consumes too much lowmem.

What do you think?


Thanks,
Peng.

>
>Regards,
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] xen arm64 dom0 question

[Peng Fan]

Hi Julien, Stefano


On My ARM64 platform, there is 6GB memory.
0x8000 - 0xfff: 2GB
0x88000 - 0x9: 4GB

xen will alloc 1:1 mapping for Dom0 memory, so if I assign dom0_mem with a 
bigger
value, saying 2048MB or bigger. xen will alloc continus memory from higher 
address
space in the higer 4GB.

But the SD controller in my ARM64 platform has a limitation that it can only
access 32bit address space. So if Dom0 memory is at higher 4GB, SD controller
can not work as expected, because it only works when the dma address is 32bit
address.

So, Can we assign a hole in lower 32bit address space for Dom0 guest physical
memory, just as DomU? Dynamically find out a hole and size 128MB or bigger?
Or do you have any ideas?

Thanks,
Peng.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [PATCH] xen/arm: smpboot: drop unneeded code when identifying cpuinfo

[Peng Fan]

The current_cpu_data indicates the cpuinfo for the current cpu.
There is no need to fill the current_cpu_data from boot_cpu_data,
because the following call to identify_cpu will override it.

Signed-off-by: Peng Fan <peng@nxp.com>
Cc: Julien Grall <julien.gr...@arm.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
---
 xen/arch/arm/smpboot.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
index d56b91d..90ad1d0 100644
--- a/xen/arch/arm/smpboot.c
+++ b/xen/arch/arm/smpboot.c
@@ -288,7 +288,6 @@ void start_secondary(unsigned long boot_phys_offset,
 
 set_processor_id(cpuid);
 
-current_cpu_data = boot_cpu_data;
 identify_cpu(_cpu_data);
 
 init_traps();
-- 
2.6.2


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [PATCH V2] xen/arm: arm64: Update the Image header

[Peng Fan]

This patch is mainly modified from Linux kernel:
[1] commit a2c1d73b94ed: arm64: Update the Image header
[2] commit 6ad1fe5d9077: arm64: avoid R_AARCH64_ABS64 relocations for Image 
header fields

From [1]:
"
This patch adds an effective image size to the kernel header which
describes the amount of memory from the start of the kernel Image binary
which the kernel expects to use before detecting memory and handling any
memory reservations. This can be used by bootloaders to choose suitable
locations to load the kernel and/or other binaries such that the kernel
will not clobber any memory unexpectedly. As before, memory reservations
are required to prevent the kernel from clobbering these locations
later.

Both the image load offset and the effective image size are forced to be
little-endian regardless of the native endianness of the kernel to
enable bootloaders to load a kernel of arbitrary endianness. Bootloaders
which wish to make use of the load offset can inspect the effective
image size field for a non-zero value to determine if the offset is of a
known endianness. To enable software to determine the endinanness of the
kernel as may be required for certain use-cases, a new flags field (also
little-endian) is added to the kernel header to export this information.
"

In this patch, XEN_VIRT_START is used as the text offset. Then, bootloader,
such as U-Boot, will load the xen image to "dram_base + text_offset".
Not choose 0 as the text offset, because we may have spin table at dram_base.
Loading xen to dram_base will override the spin table.

Introduce image.h and macros.h in this patch, just as Linux kernel.

Signed-off-by: Peng Fan <peng@nxp.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
Cc: Julien Grall <julien.gr...@arm.com>
---

V2: Addressing Julien's comments to follow linux kernel patch:
a2c1d73b94ed49 "arm64: Update the Image header"

 xen/arch/arm/arm64/head.S  |  7 +++--
 xen/arch/arm/xen.lds.S |  5 +++
 xen/include/asm-arm/arm64/image.h  | 62 ++
 xen/include/asm-arm/arm64/macros.h | 15 +
 xen/include/asm-arm/macros.h   |  2 +-
 5 files changed, 87 insertions(+), 4 deletions(-)
 create mode 100644 xen/include/asm-arm/arm64/image.h
 create mode 100644 xen/include/asm-arm/arm64/macros.h

diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
index 91e2817..0226463 100644
--- a/xen/arch/arm/arm64/head.S
+++ b/xen/arch/arm/arm64/head.S
@@ -21,6 +21,7 @@
  */
 
 #include 
+#include 
 #include 
 #include 
 #include 
@@ -114,9 +115,9 @@ efi_head:
  */
 add x13, x18, #0x16
 b   real_start   /* branch to kernel start */
-.quad   0/* Image load offset from start of RAM */
-.quad   0/* reserved */
-.quad   0/* reserved */
+le64sym _xen_offset_le   /* Image load offset from start of RAM, 
little-endian */
+le64sym _xen_size_le /* Effective size of kernel image, 
little-endian */
+le64sym _xen_flags_le/* Informative flags, little-endian */
 .quad   0/* reserved */
 .quad   0/* reserved */
 .quad   0/* reserved */
diff --git a/xen/arch/arm/xen.lds.S b/xen/arch/arm/xen.lds.S
index b24e93b..854c243 100644
--- a/xen/arch/arm/xen.lds.S
+++ b/xen/arch/arm/xen.lds.S
@@ -6,6 +6,7 @@
 #include 
 #include 
 #include 
+#include 
 #undef ENTRY
 #undef ALIGN
 
@@ -196,6 +197,10 @@ SECTIONS
   .dtb : { *(.dtb) } :text
 #endif
 
+#if defined(__aarch64__)
+  HEAD_SYMBOLS
+#endif
+
   /* Sections to be discarded */
   /DISCARD/ : {
*(.exit.text)
diff --git a/xen/include/asm-arm/arm64/image.h 
b/xen/include/asm-arm/arm64/image.h
new file mode 100644
index 000..54751ca
--- /dev/null
+++ b/xen/include/asm-arm/arm64/image.h
@@ -0,0 +1,62 @@
+/*
+ * Copied and modified from Linux
+ * "commit 29b4817d4018df78086157ea3a55c1d9424a7cfc"
+ *
+ * Linker script macros to generate Image header fields.
+ *
+ * Copyright (C) 2014 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#ifndef __ASM_IMAGE_H
+#define __ASM_IMAGE_H
+
+#ifdef CONFIG_CPU_BIG_ENDIAN
+#define DATA_LE32(data)\
+   data) & 0x00ff) << 24) |  

Re: [Xen-devel] [PATCH] arm64: head: Fill image size

[Peng Fan]

Hi Julien,

On Thu, Aug 18, 2016 at 05:02:01PM +0100, Julien Grall wrote:
>Hello Peng,
>
>On 16/08/16 03:58, Peng Fan wrote:
>>When booting xen from U-Boot, U-Boot will use the image size
>>info. Because this information is lacked in XEN image,U-Boot
>>assume the image size is 16MB to memmove, which will cost lots
>>time on simulation platform.
>
>The patch looks good to me, however I would prefer if you update all the
>header rather than updating only the field you care.
>
>You can give a look to commit a2c1d73b94ed49f5fac12e95052d7b140783f800
>"arm64: Update the Image header" in Linux for more details.

Thanks for comments. Will do this in V2.

Thanks,
Peng.

>
>Regards,
>
>>
>>Signed-off-by: Peng Fan <van.free...@gmail.com>
>>Cc: Stefano Stabellini <sstabell...@kernel.org>
>>Cc: Julien Grall <julien.gr...@arm.com>
>>---
>> xen/arch/arm/arm64/head.S | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>>diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
>>index 91e2817..9fade07 100644
>>--- a/xen/arch/arm/arm64/head.S
>>+++ b/xen/arch/arm/arm64/head.S
>>@@ -115,7 +115,7 @@ efi_head:
>> add x13, x18, #0x16
>> b   real_start   /* branch to kernel start */
>> .quad   0/* Image load offset from start of RAM 
>> */
>>-.quad   0/* reserved */
>>+.quad   _end - start /* Effective size of Image, 
>>little-endian */
>> .quad   0/* reserved */
>> .quad   0/* reserved */
>> .quad   0/* reserved */
>>
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] [PATCH] arm64: head: Fill image size

[Peng Fan]

When booting xen from U-Boot, U-Boot will use the image size
info. Because this information is lacked in XEN image,U-Boot
assume the image size is 16MB to memmove, which will cost lots
time on simulation platform.

Signed-off-by: Peng Fan <van.free...@gmail.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
Cc: Julien Grall <julien.gr...@arm.com>
---
 xen/arch/arm/arm64/head.S | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
index 91e2817..9fade07 100644
--- a/xen/arch/arm/arm64/head.S
+++ b/xen/arch/arm/arm64/head.S
@@ -115,7 +115,7 @@ efi_head:
 add x13, x18, #0x16
 b   real_start   /* branch to kernel start */
 .quad   0/* Image load offset from start of RAM */
-.quad   0/* reserved */
+.quad   _end - start /* Effective size of Image, little-endian 
*/
 .quad   0/* reserved */
 .quad   0/* reserved */
 .quad   0/* reserved */
-- 
2.6.2


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to add disk on ARM64

[Peng Fan]

On Fri, Aug 12, 2016 at 04:57:06PM +0200, Roger Pau Monné wrote:
>On Fri, Aug 12, 2016 at 03:00:34PM +0200, Julien Grall wrote:
>> On 12/08/2016 14:24, Peng Fan wrote:
>> > Hi,
>> 
>> Hello Peng,
>> 
>> I have CCed Roger who is more familiar than me with the hotplug scripts.
>> 
>> > I am using xen master branch on i.MX8 ARM64.
>> > 
>> > My xl configuration:
>> > 
>> > kernel = "/root/xen/Image"
>> > memory = "128"
>> > name = "DomU"
>> > vcpus = 1
>> > serial="pty"
>> > disk = [ 'phy:/dev/loop0,xvda,w' ]
>> > extra = "console=hvc0 root=/dev/xvda debug=/bin/sh"
>> > 
>> > 
>> > And I "losetup /dev/loop0 /root/DomU-rootfs" in Dom0 Linux.
>> > 
>> > But I met the following error:
>> > libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to 
>> > execute: /etc/xen/scripts/block add ^M^M
>> > Device /dev/loop0 is mounted in the privileged domain,^M^M
>> > and so cannot be mounted by a guest.^M^M
>> > libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: 
>> > /etc/xen/scripts/block add [800] exited with error status 1^M^M
>> > libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch 
>> > w=0xfee100: deregister unregistered^M^M
>> > libxl: error: libxl_devi
>> > ce.c:1202:device_hotplug_child_death_cb: script: Device /dev/loop0 is 
>> > mounted in the privileged domain,^M^M
>> > and so cannot be mounted by a guest.^M^M
>> 
>> From my understanding, you have mounted /dev/loop0 in Dom0, is that correct?
>> However, we don't support sharing the same mounting point by default (Roger
>> can you confirm).

Fixed. In tiny rootfs, there is no `stat`. I recompiled busybox with more 
applets,
and all is ok now.

Regards,
Peng.

>
>It seems like the hotplug script has detected that you have the device 
>already attached to Dom0, can you paste the output of `xenstore-ls -fp` 
>when this happens?
> 
>It could be that there's some garbage in xenstore from a device that hasn't 
>been properly detached.
>
>Roger.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to add disk on ARM64

[Peng Fan]

Hi Julien, Roger

On Fri, Aug 12, 2016 at 04:57:06PM +0200, Roger Pau Monné wrote:
>On Fri, Aug 12, 2016 at 03:00:34PM +0200, Julien Grall wrote:
>> On 12/08/2016 14:24, Peng Fan wrote:
>> > Hi,
>> 
>> Hello Peng,
>> 
>> I have CCed Roger who is more familiar than me with the hotplug scripts.
>> 
>> > I am using xen master branch on i.MX8 ARM64.
>> > 
>> > My xl configuration:
>> > 
>> > kernel = "/root/xen/Image"
>> > memory = "128"
>> > name = "DomU"
>> > vcpus = 1
>> > serial="pty"
>> > disk = [ 'phy:/dev/loop0,xvda,w' ]
>> > extra = "console=hvc0 root=/dev/xvda debug=/bin/sh"
>> > 
>> > 
>> > And I "losetup /dev/loop0 /root/DomU-rootfs" in Dom0 Linux.
>> > 
>> > But I met the following error:
>> > libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to 
>> > execute: /etc/xen/scripts/block add ^M^M
>> > Device /dev/loop0 is mounted in the privileged domain,^M^M
>> > and so cannot be mounted by a guest.^M^M
>> > libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: 
>> > /etc/xen/scripts/block add [800] exited with error status 1^M^M
>> > libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch 
>> > w=0xfee100: deregister unregistered^M^M
>> > libxl: error: libxl_devi
>> > ce.c:1202:device_hotplug_child_death_cb: script: Device /dev/loop0 is 
>> > mounted in the privileged domain,^M^M
>> > and so cannot be mounted by a guest.^M^M
>> 
>> From my understanding, you have mounted /dev/loop0 in Dom0, is that correct?
>> However, we don't support sharing the same mounting point by default (Roger
>> can you confirm).

No I only do "losetup /dev/loop0 DomU-rootfs". And I not mount it.

>
>It seems like the hotplug script has detected that you have the device 
>already attached to Dom0, can you paste the output of `xenstore-ls -fp` 
>when this happens?

I add xenstore-ls -fp in /etc/xen/scripts/block.
"
test -b "$dev" || fatal "$dev is not a block device."

xenstore-ls -fp  ---> Here
claim_lock "block"
check_device_sharing "$dev" "$mode" ---> seems fail in this function
"

I use busybox, not sure whether it supports the xen scripts well or not.
such as "losetup -a" is not supported by busybox.

Log:

device_hotplug: calling hotplug script: /etc/xen/scripts/block add
libxl: debug: libxl_device.c:1135:device_hotplug: extra args:
libxl: debug: libxl_device.c:1143:device_hotplug: env:
libxl: debug: libxl_device.c:1150:device_hotplug:   script: 
/etc/xen/scripts/block
libxl: debug: libxl_device.c:1150:device_hotplug:   XENBUS_TYPE: vbd
libxl: debug: libxl_device.c:1150:device_hotplug:   XENBUS_PATH: 
backend/vbd/1/51712
libxl: debug: libxl_device.c:1150:device_hotplug:   XENBUS_BASE_PATH: 
backend
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: 
/etc/xen/scripts/block add 
/tool = ""   (n0)
/tool/xenstored = ""   (n0)
/local = ""   (n0)
/local/domain = ""   (n0)
/local/domain/0 = ""   (n0)
/local/domain/0/domid = "0"   (n0)
/local/domain/0/name = "Domain-0"   (n0)
/local/domain/0/backend = ""   (n0)
/local/domain/0/backend/vbd = ""   (n0)
/local/domain/0/backend/vbd/1 = ""   (n0)
/local/domain/0/backend/vbd/1/51712 = ""   (n0,r1)
/local/domain/0/backend/vbd/1/51712/frontend = 
"/local/domain/1/device/vbd/51712"   (n0,r1)
/local/domain/0/backend/vbd/1/51712/params = "/dev/loop0"   (n0,r1)
/local/domain/0/backend/vbd/1/51712/script = "/etc/xen/scripts/block"   (n0,r1)
/local/domain/0/backend/vbd/1/51712/frontend-id = "1"   (n0,r1)
/local/domain/0/backend/vbd/1/51712/online = "1"   (n0,r1)
/local/domain/0/backend/vbd/1/51712/removable = "0"   (n0,r1)
/local/domain/0/backend/vbd/1/51712/bootable = "1"   (n0,r1)
/local/domain/0/backend/vbd/1/51712/state = "2"   (n0,r1)
/local/domain/0/backend/vbd/1/51712/dev = "xvda"   (n0,r1)
/local/domain/0/backend/vbd/1/51712/type = "phy"   (n0,r1)
/local/domain/0/backend/vbd/1/51712/mode = "w"   (n0,r1)
/local/domain/0/backend/vbd/1/51712/device-type = "disk"   (n0,r1)
/local/domain/0/backend/vbd/1/51712/discard-enable = "1"   (n0,r1)
/local/domain/1 = ""   (n0,r1)
/local/domain/1/vm = "/vm
/63997893-f587-4bdc-9bf3-dd5c7614bb51"   (n0,r1)
/local/domain/1/name = "DomU"   (n0,r1)
/local/domain/1/cpu = ""   (n0,r1)
/

[Xen-devel] Unable to add disk on ARM64

[Peng Fan]

Hi,

I am using xen master branch on i.MX8 ARM64.

My xl configuration:

kernel = "/root/xen/Image"
memory = "128"
name = "DomU"
vcpus = 1
serial="pty"
disk = [ 'phy:/dev/loop0,xvda,w' ]
extra = "console=hvc0 root=/dev/xvda debug=/bin/sh"


And I "losetup /dev/loop0 /root/DomU-rootfs" in Dom0 Linux.

But I met the following error:
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: 
/etc/xen/scripts/block add ^M^M
Device /dev/loop0 is mounted in the privileged domain,^M^M
and so cannot be mounted by a guest.^M^M
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: 
/etc/xen/scripts/block add [800] exited with error status 1^M^M
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0xfee100: 
deregister unregistered^M^M
libxl: error: libxl_devi
ce.c:1202:device_hotplug_child_death_cb: script: Device /dev/loop0 is mounted 
in the privileged domain,^M^M
and so cannot be mounted by a guest.^M^M
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0xfee100: 
deregister unregistered^M^M
libxl: error: libxl_create.c:1244:domcreate_launch_dm: unable to add disk 
devices

Then I change disk to "disk = [ 'format=raw, vdev=xvda, access=rw, 
target=/root/DomU-rootfs' ]"
But met other errors:
"
/etc/xen/scripts/block failed; error detected.
"

Any ideas?

Thanks,
Peng.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Build problems with xen 4.7

[Peng Fan]

Hi,

On Fri, May 13, 2016 at 11:23:29AM -0400, Konrad Rzeszutek Wilk wrote:
>On Fri, May 13, 2016 at 03:25:52PM +0100, M A Young wrote:
>> On Fri, 13 May 2016, Jan Beulich wrote:
>> 
>> > >>> On 13.05.16 at 15:49,  wrote:
>> > > ...
>> > > 
>> > > Still an issue - with 4.7.0-rc1.
>> > 
>> > And I don't recall anyone having contributed a fix/workaround.
>> > 
>> > > If I do:
>> > > 
>> > > $export CFLAGS=" "'
>> > > $make
>> > > 
>> > > I end up with:
>> > > gcc -E -O1 -fno-omit-frame-pointer -m64 -DBUILD_ID -g 
>> > > -fno-strict-aliasing -std=gnu99 
>> > >[...]
>> > > :0:0: error: "__OBJECT_FILE__" redefined [-Werror]
>> > > :0:0: note: this is the location of the previous definition
>> > > :0:0: error: "__OBJECT_LABEL__" redefined [-Werror]
>> > > :0:0: note: this is the location of the previous definition
>> > > cc1: all warnings being treated as errors
>> > > Makefile:62: recipe for target 'compat/callback.i' failed

I met similar issue when cross compile xen for ARM64 on X86 PC using yocto poky 
4.9.3
toolchain.

aarch64-poky-linux-gcc -O2 -pipe -g -feliminate-unused-debug-types -O1 
-fno-omit-frame-pointer -DBUILD_ID -g -fno-strict-aliasing -std=gnu99 -Wall 
-Wstrict-prototypes -Wdeclaration-after-statement -Wno-unused-but-set-variable 
-Wno-unused-local-typedefs -O1 -fno-omit-frame-pointer -DBUILD_ID -g 
-fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes 
-Wdeclaration-after-statement -Wno-unused-but-set-variable 
-Wno-unused-local-typedefs -O1 -fno-omit-frame-pointer -DBUILD_ID -g 
-fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes 
-Wdeclaration-after-statement -Wno-unused-but-set-variable 
-Wno-unused-local-typedefs -nostdinc -fno-builtin -fno-common -Werror 
-Wredundant-decls -Wno-pointer-arith -pipe -g -D__XEN__ -include 
/home/Freenix/work/sw-stash/imx8/8qm/xen/xen/xen/include/xen/config.h 
'-D__OBJECT_FILE__="/home/Freenix/work/sw-stash/imx8/8qm/xen/xen/xen/xen"' 
-Wa,--strip-local-absolute -fno-optimize-sibling-calls -fno-omit-frame-pointer 
-MMD -MF /home/Freenix/work/sw-stash/imx8/8qm/xen/xen/xen/.xen.d -mcpu=generic 
-mgeneral-regs-only -DCONFIG_EARLY_PRINTK 
-DEARLY_PRINTK_INC=\"debug-imx8qm.inc\" -DEARLY_PRINTK_BAUD= 
-DEARLY_UART_BASE_ADDRESS=0x5a06 -DEARLY_UART_REG_SHIFT= 
-fno-optimize-sibling-calls 
-I/home/Freenix/work/sw-stash/imx8/8qm/xen/xen/xen/include -fno-stack-protector 
-fno-exceptions -Wnested-externs -DGCC_HAS_VISIBILITY_ATTRIBUTE -O1 
-fno-omit-frame-pointer -DBUILD_ID -g -fno-strict-aliasing -std=gnu99 -Wall 
-Wstrict-prototypes -Wdeclaration-after-statement -Wno-unused-but-set-variable 
-Wno-unused-local-typedefs -O1 -fno-omit-frame-pointer -DBUILD_ID -g 
-fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes 
-Wdeclaration-after-statement -Wno-unused-but-set-variable 
-Wno-unused-local-typedefs -nostdinc -fno-builtin -fno-common -Werror 
-Wredundant-decls -Wno-pointer-arith -pipe -g -D__XEN__ -include 
/home/Freenix/work/sw-stash/imx8/8qm/xen/xen/xen/include/xen/config.h 
'-D__OBJECT_FILE__="asm-offsets.s"' -Wa,--strip-local-absolute 
-fno-optimize-sibling-calls -fno-omit-frame-pointer -MMD -MF ./.asm-offsets.s.d 
-mcpu=generic -mgeneral-regs-only -DCONFIG_EARLY_PRINTK 
-DEARLY_PRINTK_INC=\"debug-imx8qm.inc\" -DEARLY_PRINTK_BAUD= 
-DEARLY_UART_BASE_ADDRESS=0x5a06 -DEARLY_UART_REG_SHIFT= 
-I/home/Freenix/work/sw-stash/imx8/8qm/xen/xen/xen/include -fno-stack-protector 
-fno-exceptions -Wnested-externs -DGCC_HAS_VISIBILITY_ATTRIBUTE -S -o 
asm-offsets.s arm64/asm-offsets.c
:0:0: error: "__OBJECT_FILE__" redefined [-Werror]
:0:0: note: this is the location of the previous definition
cc1: all warnings being treated as errors


>> > 
>> > My previous recommendation stands: Then just don't do this.
>> 
>> I hacked around it for my test builds (eg. see my test build at
>> https://copr.fedorainfracloud.org/coprs/myoung/xentest/build/204111/
>> ) by not setting CFLAGS in the environment, but by instead adding the 
>> recommended Fedora RPM settings into config/StdGNU.mk via a different 
>> environment variable.
>
>ah:
>
>--- xen-4.7.0/config/StdGNU.mk.orig2016-04-15 22:56:52.191227591 +0100
>+++ xen-4.7.0/config/StdGNU.mk 2016-04-15 23:01:40.978829756 +0100
>@@ -37,6 +37,12 @@
> 
> ifneq ($(debug),y)
> CFLAGS += -O2 -fomit-frame-pointer -fno-tree-coalesce-vars
>+ifeq ($(XEN_TARGET_ARCH),x86_64)
>+#might be cross-compiling so strip out possible x86_32 options
>+CFLAGS += $(shell echo $(CFLAGS_EXTRA) | sed -e 's/-m32//g' -e 
>'s/-march=i686//g' -e 's/-mtune=atom//g')
>+else
>+CFLAGS += $(CFLAGS_EXTRA)
>+endif
> else
> # Less than -O1 produces bad code and large stack frames
> CFLAGS += -O1 -fno-omit-frame-pointer
>
>
>And in the spec file:
>export CFLAGS_EXTRA="$RPM_OPT_FLAGS"
>make %{?_smp_mflags} %{?efi_flags} prefix=/usr dist-xen
>

Does this patch work when cross compile for ARM64?

Thanks,
Peng.

>
>> 
>> Another thing you might need to know if you are building xen on Fedora 24 
>> is that you need to add 

[Xen-devel] [PATCH V2] xen/arm64: config: Correctly define VMAP_VIRT_END

[Peng Fan]

The vmap initialization code (vm_init_type) will round down
the end of the region to a page-aligned address.

On ARM64, the default vmap region is located between 1G and 2G.
Based on the initialization code, the end address is excluded
of the region.

Therefore the current definition of VMAP_VIRT_END will lead the
vmap code to not use the last 4K of the region.

Fix it by defining VMAP_VIRT_END as "VMAP_VIRT_START + GB(1)".

Signed-off-by: Peng Fan <van.free...@gmail.com>
Cc: Julien Grall <julien.gr...@arm.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
---

V2:
 Take Julien's better commit message. Thanks.

 xen/include/asm-arm/config.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/xen/include/asm-arm/config.h b/xen/include/asm-arm/config.h
index 2d11b62..f92c0a0 100644
--- a/xen/include/asm-arm/config.h
+++ b/xen/include/asm-arm/config.h
@@ -147,7 +147,7 @@
 #define SLOT0_ENTRY_SIZE  SLOT0(1)
 
 #define VMAP_VIRT_START  GB(1)
-#define VMAP_VIRT_END(VMAP_VIRT_START + GB(1) - 1)
+#define VMAP_VIRT_END(VMAP_VIRT_START + GB(1))
 
 #define FRAMETABLE_VIRT_START  GB(32)
 #define FRAMETABLE_SIZEGB(32)
-- 
2.6.2


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] xen/arm: setup: initialize xenheap mappings after boot pages avaiable

[Peng Fan]

Hi Julien,

On Tue, May 31, 2016 at 06:08:38PM +0100, Julien Grall wrote:
>Hi Peng,
>
>On 31/05/16 10:58, Peng Fan wrote:
>>>
>>>>So, need to make sure boot pages are ready before setup xenheap mappings.
>>>
>>>init_boot_pages is using mfn_to_virt (see bootmem_region_add), which cannot
>>>work until xenheap_mfn_start is initialized. This is done by
>>>setup_xenheap_mappings.
>>
>>My bad. I did not catch this point. Thanks for correcting me.
>>
>>>
>>>I would be happy to give you hint on how to solve this, but I am not sure to
>>>fully understand your issue. Can you give more details?
>>
>>I did not met issue on my platform. I just think the logic of this piece code
>>may cause errors on platform with large memory (>512GB).
>>
>>How about the following patch?
>>First loop all the memory banks and calculate ram_size/ram_start/ram_end,
>>then set xenheap_virt_end/start and xenheap_mfn_end.
>>Now readly for init boot pages and setup_xenheap_mappings.
>
>Have you tested this patch? I would be surprised to see it working.

Not tested (:- Just an idea in my mind.

>
>>
>>diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
>>index dcb23b7..d3a3af3 100644
>>--- a/xen/arch/arm/setup.c
>>+++ b/xen/arch/arm/setup.c
>>@@ -635,13 +635,24 @@ static void __init setup_mm(unsigned long dtb_paddr, 
>>size_t dtb_size)
>>  paddr_t bank_start = bootinfo.mem.bank[bank].start;
>>  paddr_t bank_size = bootinfo.mem.bank[bank].size;
>>  paddr_t bank_end = bank_start + bank_size;
>>-paddr_t s, e;
>>
>>  ram_size = ram_size + bank_size;
>>  ram_start = min(ram_start,bank_start);
>>  ram_end = max(ram_end,bank_end);
>>+}
>>
>>-setup_xenheap_mappings(bank_start>>PAGE_SHIFT, 
>>bank_size>>PAGE_SHIFT);
>>+total_pages += ram_size >> PAGE_SHIFT;
>>+
>>+xenheap_virt_end = XENHEAP_VIRT_START + ram_end - ram_start;
>
>XENHEAP_VIRT_START will be replaced by xenheap_virt_start which is not
>initialized at this time. It is done by setup_xenheap_mappings.

Could we move the piece code out to setup_mm, before init_boot_pages and 
setup_xenheap_mappings?
"
xenheap_virt_start = DIRECTMAP_VIRT_START +
(base_mfn - mfn) * PAGE_SIZE;
"

>
>In any case, even if the variable are correctly setup the underlying page
>table are not present.

Yeah, but setup_mm is only executed on one cpu and this is only for 
intialization.
Does that really matter if underlying page table are not ready?

>
>The easiest way I can think to fix the problem is splitting the bank with
>chunk of 512GB and calling setup_xenheap_mappings, init_boot_pages for each
>chunk.

Do you mean let process_memory_node handle the >512GB memory bank?

>
>It is not the nicest way, so I will happy if you find a better one.

I have no good idea now (:
I can follow your suggestion to split the >512GB into smaller banks when filling
bootinfo.mem.bank[xx].start and bootinfo.mem.bank[xx].size, if you are happy
with this way.

Thanks,
Peng.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] xen/arm64: config: correct VMAP_VIRT_END

[Peng Fan]

Hi Julien,

On Tue, May 31, 2016 at 12:07:58PM +0100, Julien Grall wrote:
>Hi Peng,
>
>On 27/05/16 11:23, Peng Fan wrote:
>>To ARM64, we should use '(VMAP_VIRT_START + GB(1))' as VMAP_VIRT_END,
>
>s/To/For/

Fix in V2.

>
>>but not '(VMAP_VIRT_START + GB(1) - 1)'.
>>
>>Seeing 'vm_end[type] = PFN_DOWN(end - start);' in vm_init_type,
>>if not correct VMAP_VIRT_END, one page is wasted.
>
>I find difficult to parse the commit message. How about:

Sorry (:

>
>"xen/arm64: config: Correctly define VMAP_VIRT_END
>
>The vmap initialization code (vm_init_type) will round down the end of the
>region to a page-aligned address.
>
>On ARM64, the default vmap region is located between 1G and 2G. Based on the
>initialization code, the end address is excluded of the region.
>
>Therefore the current definition of VMAP_VIRT_END will lead the vmap code to
>not use the last 4K of the region.
>
>Fix it by defining VMAP_VIRT_END as "VMAP_VIRT_START + GB(1)".
>"

Thanks for your good description.

Thanks,
Peng.

>
>>
>>Signed-off-by: Peng Fan <van.free...@gmail.com>
>>Cc: Julien Grall <julien.gr...@arm.com>
>>Cc: Stefano Stabellini <sstabell...@kernel.org>
>>---
>>
>>I found X86 use '(VMAP_VIRT_START + GB(64))' and ARM32 use XENHEAP_VIRT_START,
>>both are aligned address. So, I think ARM64 may also need to use aligned
>>address. I am not very sure (:
>
>Correct, this should be aligned to avoid wasting a page.
>
>>
>>  xen/include/asm-arm/config.h | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>>diff --git a/xen/include/asm-arm/config.h b/xen/include/asm-arm/config.h
>>index 2d11b62..f92c0a0 100644
>>--- a/xen/include/asm-arm/config.h
>>+++ b/xen/include/asm-arm/config.h
>>@@ -147,7 +147,7 @@
>>  #define SLOT0_ENTRY_SIZE  SLOT0(1)
>>
>>  #define VMAP_VIRT_START  GB(1)
>>-#define VMAP_VIRT_END(VMAP_VIRT_START + GB(1) - 1)
>>+#define VMAP_VIRT_END(VMAP_VIRT_START + GB(1))
>>
>>  #define FRAMETABLE_VIRT_START  GB(32)
>>  #define FRAMETABLE_SIZEGB(32)
>>
>
>Regards,
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] xen/arm: setup: initialize xenheap mappings after boot pages avaiable

[Peng Fan]

Hi Julien,

On Mon, May 30, 2016 at 10:53:24PM +0100, Julien Grall wrote:
>Hi Peng,
>
>On 27/05/2016 06:31, Peng Fan wrote:
>>To ARM64, setup_xenheap_mappings may call alloc_boot_pages to allocate
>>first level page table, if there is a big chunk memory (ie, >512GB).
>
>Out of interest, have you found the bug by testing Xen on a platform with
>512GB of RAM? :)

Actually not. My board does not have so large memory. When I am reading
the piece code, I think there is possibility that setup_xenheap_mappings
may need boot pages, but boot pages are not ready.

>
>>So, need to make sure boot pages are ready before setup xenheap mappings.
>
>init_boot_pages is using mfn_to_virt (see bootmem_region_add), which cannot
>work until xenheap_mfn_start is initialized. This is done by
>setup_xenheap_mappings.

My bad. I did not catch this point. Thanks for correcting me.

>
>I would be happy to give you hint on how to solve this, but I am not sure to
>fully understand your issue. Can you give more details?

I did not met issue on my platform. I just think the logic of this piece code
may cause errors on platform with large memory (>512GB).

How about the following patch?
First loop all the memory banks and calculate ram_size/ram_start/ram_end,
then set xenheap_virt_end/start and xenheap_mfn_end.
Now readly for init boot pages and setup_xenheap_mappings.

diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
index dcb23b7..d3a3af3 100644
--- a/xen/arch/arm/setup.c
+++ b/xen/arch/arm/setup.c
@@ -635,13 +635,24 @@ static void __init setup_mm(unsigned long dtb_paddr, 
size_t dtb_size)
 paddr_t bank_start = bootinfo.mem.bank[bank].start;
 paddr_t bank_size = bootinfo.mem.bank[bank].size;
 paddr_t bank_end = bank_start + bank_size;
-paddr_t s, e;
 
 ram_size = ram_size + bank_size;
 ram_start = min(ram_start,bank_start);
 ram_end = max(ram_end,bank_end);
+}
 
-setup_xenheap_mappings(bank_start>>PAGE_SHIFT, bank_size>>PAGE_SHIFT);
+total_pages += ram_size >> PAGE_SHIFT;
+
+xenheap_virt_end = XENHEAP_VIRT_START + ram_end - ram_start;
+xenheap_mfn_start = ram_start >> PAGE_SHIFT;
+xenheap_mfn_end = ram_end >> PAGE_SHIFT;
+
+for ( bank = 0 ; bank < bootinfo.mem.nr_banks; bank++ )
+{
+paddr_t bank_start = bootinfo.mem.bank[bank].start;
+paddr_t bank_size = bootinfo.mem.bank[bank].size;
+paddr_t bank_end = bank_start + bank_size;
+paddr_t s, e;
 
 s = bank_start;
 while ( s < bank_end )
@@ -658,18 +669,12 @@ static void __init setup_mm(unsigned long dtb_paddr, 
size_t dtb_size)
 if ( e > bank_end )
 e = bank_end;
 
-xenheap_mfn_end = e;
-
 dt_unreserved_regions(s, e, init_boot_pages, 0);
 s = n;
 }
-}
-
-total_pages += ram_size >> PAGE_SHIFT;
 
-xenheap_virt_end = XENHEAP_VIRT_START + ram_end - ram_start;
-xenheap_mfn_start = ram_start >> PAGE_SHIFT;
-xenheap_mfn_end = ram_end >> PAGE_SHIFT;
+setup_xenheap_mappings(bank_start>>PAGE_SHIFT, bank_size>>PAGE_SHIFT);
+}
 
 /*
  * Need enough mapped pages for copying the DTB.

Thanks,
Peng.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH] xen/arm64: config: correct VMAP_VIRT_END

[Peng Fan]

To ARM64, we should use '(VMAP_VIRT_START + GB(1))' as VMAP_VIRT_END,
but not '(VMAP_VIRT_START + GB(1) - 1)'.

Seeing 'vm_end[type] = PFN_DOWN(end - start);' in vm_init_type,
if not correct VMAP_VIRT_END, one page is wasted.

Signed-off-by: Peng Fan <van.free...@gmail.com>
Cc: Julien Grall <julien.gr...@arm.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
---

I found X86 use '(VMAP_VIRT_START + GB(64))' and ARM32 use XENHEAP_VIRT_START,
both are aligned address. So, I think ARM64 may also need to use aligned
address. I am not very sure (:

 xen/include/asm-arm/config.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/xen/include/asm-arm/config.h b/xen/include/asm-arm/config.h
index 2d11b62..f92c0a0 100644
--- a/xen/include/asm-arm/config.h
+++ b/xen/include/asm-arm/config.h
@@ -147,7 +147,7 @@
 #define SLOT0_ENTRY_SIZE  SLOT0(1)
 
 #define VMAP_VIRT_START  GB(1)
-#define VMAP_VIRT_END(VMAP_VIRT_START + GB(1) - 1)
+#define VMAP_VIRT_END(VMAP_VIRT_START + GB(1))
 
 #define FRAMETABLE_VIRT_START  GB(32)
 #define FRAMETABLE_SIZEGB(32)
-- 
2.6.2


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH] xen/arm: setup: initialize xenheap mappings after boot pages avaiable

[Peng Fan]

To ARM64, setup_xenheap_mappings may call alloc_boot_pages to allocate
first level page table, if there is a big chunk memory (ie, >512GB).

So, need to make sure boot pages are ready before setup xenheap mappings.

Signed-off-by: Peng Fan <van.free...@gmail.com>
Cc: Julien Grall <julien.gr...@arm.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
---
 xen/arch/arm/setup.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
index dcb23b7..24cf9d3 100644
--- a/xen/arch/arm/setup.c
+++ b/xen/arch/arm/setup.c
@@ -641,8 +641,6 @@ static void __init setup_mm(unsigned long dtb_paddr, size_t 
dtb_size)
 ram_start = min(ram_start,bank_start);
 ram_end = max(ram_end,bank_end);
 
-setup_xenheap_mappings(bank_start>>PAGE_SHIFT, bank_size>>PAGE_SHIFT);
-
 s = bank_start;
 while ( s < bank_end )
 {
@@ -663,6 +661,8 @@ static void __init setup_mm(unsigned long dtb_paddr, size_t 
dtb_size)
 dt_unreserved_regions(s, e, init_boot_pages, 0);
 s = n;
 }
+
+setup_xenheap_mappings(bank_start>>PAGE_SHIFT, bank_size>>PAGE_SHIFT);
 }
 
 total_pages += ram_size >> PAGE_SHIFT;
-- 
2.6.2


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH] xen/arm: setup: fix typo

[Peng Fan]

Typo fix: fdt_get_mem_rsc -> fdt_get_mem_rsv

Signed-off-by: Peng Fan <van.free...@gmail.com>
Cc: Julien Grall <julien.gr...@arm.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
---
 xen/arch/arm/setup.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
index 09ff1ea..dcb23b7 100644
--- a/xen/arch/arm/setup.c
+++ b/xen/arch/arm/setup.c
@@ -185,7 +185,7 @@ void dt_unreserved_regions(paddr_t s, paddr_t e,
 /* If we can't read it, pretend it doesn't exist... */
 continue;
 
-r_e += r_s; /* fdt_get_mem_rsc returns length */
+r_e += r_s; /* fdt_get_mem_rsv returns length */
 
 if ( s < r_e && r_s < e )
 {
-- 
2.6.2


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 4/4] xen/arm: arm64: Remove MPIDR multiprocessing extensions check

[Peng Fan]

Hi Wei,

On Wed, May 25, 2016 at 10:10:11AM +0800, Wei Chen wrote:
>In ARM64, the MPIDR multiprocessing extensions bit is reserved to 1.
>So, the value check for this bit is no longer necessary on ARM64.

From ARM DDI0487A.G, I found the U bit for MPIDR_EL1:
"
Indicates a Uniprocessor system, as distinct from PE 0 in a multiprocessor 
system. The possible
values of this bit are:
0 Processor is part of a multiprocessor system.
1 Processor is part of a uniprocessor system.
"

It's not reserved to 1. Which doc are you refering to?

Regards,
Peng.

>
>Signed-off-by: Wei Chen 
>---
> xen/arch/arm/arm64/head.S | 1 -
> 1 file changed, 1 deletion(-)
>
>diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
>index 3090beb..91e2817 100644
>--- a/xen/arch/arm/arm64/head.S
>+++ b/xen/arch/arm/arm64/head.S
>@@ -267,7 +267,6 @@ common_start:
>   * find that multiprocessor extensions 
> are
>   * present and the system is SMP  */
> mrs   x0, mpidr_el1
>-tbz   x0, _MPIDR_SMP, 1f /* Multiprocessor extension not 
>supported? */
> tbnz  x0, _MPIDR_UP, 1f  /* Uniprocessor system? */
> 
> ldr   x13, =(~MPIDR_HWID_MASK)
>-- 
>2.7.4
>
>
>___
>Xen-devel mailing list
>Xen-devel@lists.xen.org
>http://lists.xen.org/xen-devel

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH] xen/arm: smpboot: drop unneeded code in start_secondary

[Peng Fan]

CPU0 boots up secondary CPUs one by one. Before booting
one secondary CPU, CPU0 will assign hwid to smp_up_cpu
and flush cache. After the secondary CPU boots up,
CPU0 will assign MPIDR_INVALID to smp_up_cpu and flush
cache.

There is no need for secondary CPUs to assign MPIDR_INVALID
to smp_up_cpu. So, drop it.

Signed-off-by: Peng Fan <van.free...@gmail.com>
Cc: Julien Grall <julien.gr...@arm.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
---
 xen/arch/arm/smpboot.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
index c5109bf..6b3c157 100644
--- a/xen/arch/arm/smpboot.c
+++ b/xen/arch/arm/smpboot.c
@@ -309,7 +309,6 @@ void start_secondary(unsigned long boot_phys_offset,
 smp_wmb();
 
 /* Now report this CPU is up */
-smp_up_cpu = MPIDR_INVALID;
 cpumask_set_cpu(cpuid, _online_map);
 smp_wmb();
 
-- 
2.6.2


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH V2] xen/arm: mm: fix nr_second calculation in setup_frametable_mappings

[Peng Fan]

On ARM64, "frametable_size >> SECOND_SHIFT" computes the number
of second level entries, not the number of second level pages.

"ROUNDUP(frametable_size, FIRST_SIZE) >> FIRST_SHIFT" which computes
the number of the first level entries (the number of second level pages),
is the correct one that should be used.

Signed-off-by: Peng Fan <van.free...@gmail.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
Cc: Julien Grall <julien.gr...@arm.com>
---

V2:
 Take Julien's suggestion in 
http://lists.xen.org/archives/html/xen-devel/2016-05/msg01145.html.
 Refine commit log.

 xen/arch/arm/mm.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index 0a4f845..5f60aa4 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -767,7 +767,8 @@ void __init setup_frametable_mappings(paddr_t ps, paddr_t 
pe)
 base_mfn = alloc_boot_pages(frametable_size >> PAGE_SHIFT, 32<<(20-12));
 
 #ifdef CONFIG_ARM_64
-nr_second = frametable_size >> SECOND_SHIFT;
+/* Compute the number of second level pages. */
+nr_second = ROUNDUP(frametable_size, FIRST_SIZE) >> FIRST_SHIFT;
 second_base = alloc_boot_pages(nr_second, 1);
 second = mfn_to_virt(second_base);
 for ( i = 0; i < nr_second; i++ )
-- 
2.6.2


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH V3 1/2] xen/arm: mm: remove unnecessary tlb flush in setup_pagetables

[Peng Fan]

CPU0 is using the boot pages table before relocating xen and
xen_second is not part of them. So, no need to flush the TLB
when filling xen_second.

Signed-off-by: Peng Fan <van.free...@gmail.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
Cc: Julien Grall <julien.gr...@arm.com>
Reviewed-by: Julien Grall <julien.gr...@arm.com>
---
V3:
 Add Julien's review tag.

V2:
 Following Julien's comments:
   split the V1 patch into two patches. This patch only remove tlb flush.
   refine commit log

 xen/arch/arm/mm.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index 94ea054..addd699 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -447,7 +447,6 @@ void __init setup_pagetables(unsigned long 
boot_phys_offset, paddr_t xen_paddr)
 dest_va = BOOT_RELOC_VIRT_START;
 pte = mfn_to_xen_entry(xen_paddr >> PAGE_SHIFT, WRITEALLOC);
 write_pte(xen_second + second_table_offset(dest_va), pte);
-flush_xen_data_tlb_range_va_local(dest_va, SECOND_SIZE);
 
 /* Calculate virt-to-phys offset for the new location */
 phys_offset = xen_paddr - (unsigned long) _start;
-- 
2.6.2


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH V3 2/2] xen/arm: mm: clean up code in setup_pagetables

[Peng Fan]

The base of address for the relocated xen needs to be mapped
at the same virtual address (BOOT_RELOC_VIRT_START) in both
the boot and runtime page tables. So we can merge the two pieces
of code into on code block

Also no need to use write_pte when mapping BOOT_RELOC_VIRT_START
in xen_second, because CPU0 is using boot page tables.

Signed-off-by: Peng Fan <van.free...@gmail.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
Cc: Julien Grall <julien.gr...@arm.com>
---

V3:
 Refine the commit log.

V2:
 Follow Julien's comments:
   split the V1 patch into two patches, this patch is the code movement part.

 xen/arch/arm/mm.c | 10 --
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index addd699..0a4f845 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -443,11 +443,6 @@ void __init setup_pagetables(unsigned long 
boot_phys_offset, paddr_t xen_paddr)
 lpae_t pte, *p;
 int i;
 
-/* Map the destination in the boot misc area. */
-dest_va = BOOT_RELOC_VIRT_START;
-pte = mfn_to_xen_entry(xen_paddr >> PAGE_SHIFT, WRITEALLOC);
-write_pte(xen_second + second_table_offset(dest_va), pte);
-
 /* Calculate virt-to-phys offset for the new location */
 phys_offset = xen_paddr - (unsigned long) _start;
 
@@ -494,9 +489,12 @@ void __init setup_pagetables(unsigned long 
boot_phys_offset, paddr_t xen_paddr)
 pte = boot_second[second_table_offset(BOOT_FDT_VIRT_START)];
 xen_second[second_table_offset(BOOT_FDT_VIRT_START)] = pte;
 
-/* Map the destination in the boot misc area. */
+/* ... Boot Misc area for xen relocation */
 dest_va = BOOT_RELOC_VIRT_START;
 pte = mfn_to_xen_entry(xen_paddr >> PAGE_SHIFT, WRITEALLOC);
+/* Map the destination in xen_second. */
+xen_second[second_table_offset(dest_va)] = pte;
+/* Map the destination in boot_second. */
 write_pte(boot_second + second_table_offset(dest_va), pte);
 flush_xen_data_tlb_range_va_local(dest_va, SECOND_SIZE);
 #ifdef CONFIG_ARM_64
-- 
2.6.2


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] xen/arm: mm: fix nr_second calculation in setup_frametable_mappings

[Peng Fan]

Hi Julien,

On Thu, May 12, 2016 at 11:48:30AM +0100, Julien Grall wrote:
>Hi Peng,
>
>On 12/05/16 07:36, Peng Fan wrote:
>>To ARM64, "frametable_size >> SECOND_SHIFT" means the number
>>of second level entries, not the number of second level pages.
>>
>>"DIV_ROUND_UP(frametable_size >> SECOND_SHIFT, LPAE_ENTRIES)"
>>is the correct way to calculate the second level pages needed
>>for frametable mapping.
>
>Good catch!
>
>>Signed-off-by: Peng Fan <van.free...@gmail.com>
>>Cc: Stefano Stabellini <sstabell...@kernel.org>
>>Cc: Julien Grall <julien.gr...@arm.com>
>>---
>>  xen/arch/arm/mm.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>>diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
>>index 0a4f845..7c7f8e9 100644
>>--- a/xen/arch/arm/mm.c
>>+++ b/xen/arch/arm/mm.c
>>@@ -767,7 +767,7 @@ void __init setup_frametable_mappings(paddr_t ps, paddr_t 
>>pe)
>>  base_mfn = alloc_boot_pages(frametable_size >> PAGE_SHIFT, 32<<(20-12));
>>
>>  #ifdef CONFIG_ARM_64
>>-nr_second = frametable_size >> SECOND_SHIFT;
>>+nr_second = DIV_ROUND_UP(frametable_size >> SECOND_SHIFT, LPAE_ENTRIES);
>
>
>I think the following would be clearer:
>
>ROUNDUP(frametable_size, FIRST_SIZE) >> FIRST_SHIFT;

Agree.

>
>This would also benefit a comment to explain that second level of page table
>are populating.

Do you mean I need to add a comment for the upper code?

For the commit log, I would like to change to this:
"
To ARM64, "frametable_size >> SECOND_SHIFT" means the number
of second level entries, not the number of second level pages.

"ROUNDUP(frametable_size, FIRST_SIZE) >> FIRST_SHIFT" which means
the number of the first level entries(the number of second level pages),
is the correct one that should be used.
"
How do you think?

Thanks,
Peng.
>
>>  second_base = alloc_boot_pages(nr_second, 1);
>>  second = mfn_to_virt(second_base);
>>  for ( i = 0; i < nr_second; i++ )
>>
>
>Regards,
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH] xen/arm: mm: fix nr_second calculation in setup_frametable_mappings

[Peng Fan]

To ARM64, "frametable_size >> SECOND_SHIFT" means the number
of second level entries, not the number of second level pages.

"DIV_ROUND_UP(frametable_size >> SECOND_SHIFT, LPAE_ENTRIES)"
is the correct way to calculate the second level pages needed
for frametable mapping.

Signed-off-by: Peng Fan <van.free...@gmail.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
Cc: Julien Grall <julien.gr...@arm.com>
---
 xen/arch/arm/mm.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index 0a4f845..7c7f8e9 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -767,7 +767,7 @@ void __init setup_frametable_mappings(paddr_t ps, paddr_t 
pe)
 base_mfn = alloc_boot_pages(frametable_size >> PAGE_SHIFT, 32<<(20-12));
 
 #ifdef CONFIG_ARM_64
-nr_second = frametable_size >> SECOND_SHIFT;
+nr_second = DIV_ROUND_UP(frametable_size >> SECOND_SHIFT, LPAE_ENTRIES);
 second_base = alloc_boot_pages(nr_second, 1);
 second = mfn_to_virt(second_base);
 for ( i = 0; i < nr_second; i++ )
-- 
2.6.2


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH V2 2/2] xen/arm: mm: clean up code in setup_pagetables

[Peng Fan]

In setup_pagetables, need to map BOOT_RELOC_VIRT_START
in xen_second and boot_second, so we can merge the two
pieces code into one code block.

Also no need to use write_pte when map BOOT_RELOC_VIRT_START
in xen_second, because CPU0 is using boot page tables now.

Signed-off-by: Peng Fan <van.free...@gmail.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
Cc: Julien Grall <julien.gr...@arm.com>
---

V2:
 Follow Julien's comments:
   split the V1 patch into two patches, this patch is the code movement part.

 xen/arch/arm/mm.c | 10 --
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index addd699..0a4f845 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -443,11 +443,6 @@ void __init setup_pagetables(unsigned long 
boot_phys_offset, paddr_t xen_paddr)
 lpae_t pte, *p;
 int i;
 
-/* Map the destination in the boot misc area. */
-dest_va = BOOT_RELOC_VIRT_START;
-pte = mfn_to_xen_entry(xen_paddr >> PAGE_SHIFT, WRITEALLOC);
-write_pte(xen_second + second_table_offset(dest_va), pte);
-
 /* Calculate virt-to-phys offset for the new location */
 phys_offset = xen_paddr - (unsigned long) _start;
 
@@ -494,9 +489,12 @@ void __init setup_pagetables(unsigned long 
boot_phys_offset, paddr_t xen_paddr)
 pte = boot_second[second_table_offset(BOOT_FDT_VIRT_START)];
 xen_second[second_table_offset(BOOT_FDT_VIRT_START)] = pte;
 
-/* Map the destination in the boot misc area. */
+/* ... Boot Misc area for xen relocation */
 dest_va = BOOT_RELOC_VIRT_START;
 pte = mfn_to_xen_entry(xen_paddr >> PAGE_SHIFT, WRITEALLOC);
+/* Map the destination in xen_second. */
+xen_second[second_table_offset(dest_va)] = pte;
+/* Map the destination in boot_second. */
 write_pte(boot_second + second_table_offset(dest_va), pte);
 flush_xen_data_tlb_range_va_local(dest_va, SECOND_SIZE);
 #ifdef CONFIG_ARM_64
-- 
2.6.2


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH V2 1/2] xen/arm: mm: remove unnecessary tlb flush in setup_pagetables

[Peng Fan]

CPU0 is using the boot pages table before relocating xen and
xen_second is not part of them. So, no need to flush the TLB
when filling xen_second.

Signed-off-by: Peng Fan <van.free...@gmail.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
Cc: Julien Grall <julien.gr...@arm.com>
---

V2:
 Following Julien's comments:
   split the V1 patch into two patches. This patch only remove tlb flush.
   refine commit log

 xen/arch/arm/mm.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index 94ea054..addd699 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -447,7 +447,6 @@ void __init setup_pagetables(unsigned long 
boot_phys_offset, paddr_t xen_paddr)
 dest_va = BOOT_RELOC_VIRT_START;
 pte = mfn_to_xen_entry(xen_paddr >> PAGE_SHIFT, WRITEALLOC);
 write_pte(xen_second + second_table_offset(dest_va), pte);
-flush_xen_data_tlb_range_va_local(dest_va, SECOND_SIZE);
 
 /* Calculate virt-to-phys offset for the new location */
 phys_offset = xen_paddr - (unsigned long) _start;
-- 
2.6.2


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] xen/arm: mm: optimize setup_pagetables

[Peng Fan]

On Wed, May 11, 2016 at 11:03:06AM +0100, Julien Grall wrote:
>
>
>On 11/05/2016 10:57, Peng Fan wrote:
>>Hi Julien,
>
>Hi Peng,
>
>>On Wed, May 11, 2016 at 10:31:49AM +0100, Julien Grall wrote:
>>>
>>>[...]
>>>
>>>>diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
>>>>index 94ea054..bebd82f 100644
>>>>--- a/xen/arch/arm/mm.c
>>>>+++ b/xen/arch/arm/mm.c
>>>>@@ -443,12 +443,6 @@ void __init setup_pagetables(unsigned long 
>>>>boot_phys_offset, paddr_t xen_paddr)
>>>>  lpae_t pte, *p;
>>>>  int i;
>>>>
>>>>-/* Map the destination in the boot misc area. */
>>>>-dest_va = BOOT_RELOC_VIRT_START;
>>>>-pte = mfn_to_xen_entry(xen_paddr >> PAGE_SHIFT, WRITEALLOC);
>>>>-write_pte(xen_second + second_table_offset(dest_va), pte);
>>>>-flush_xen_data_tlb_range_va_local(dest_va, SECOND_SIZE);
>>>>-
>>>>  /* Calculate virt-to-phys offset for the new location */
>>>>  phys_offset = xen_paddr - (unsigned long) _start;
>>>>
>>>>@@ -498,6 +492,11 @@ void __init setup_pagetables(unsigned long 
>>>>boot_phys_offset, paddr_t xen_paddr)
>>>>  /* Map the destination in the boot misc area. */
>>>>  dest_va = BOOT_RELOC_VIRT_START;
>>>>  pte = mfn_to_xen_entry(xen_paddr >> PAGE_SHIFT, WRITEALLOC);
>>>>+xen_second[second_table_offset(dest_va)] = pte;
>>>>+
>>>>+/* Map the destination in the boot misc area. */
>>>>+dest_va = BOOT_RELOC_VIRT_START;
>>>>+pte = mfn_to_xen_entry(xen_paddr >> PAGE_SHIFT, WRITEALLOC);
>>>
>>>Why do you need to recompute "pte" and "dest_va"? They will be the same for
>>>"boot_second" and "xen_second".
>>
>>I just change "write_pte(xen_second + second_table_offset(dest_va), pte);" to
>>"xen_second[second_table_offset(dest_va)] = pte;".
>>
>>The pte and dest_va are not changed.
>
>So you could do some like:
>
>/* Comment */
>dest_va = BOOT...
>pte = ...
>/* Comment */
>xen_second[...] = pte
>/* Comment */
>write_pte(boot_second...);

This looks better. Will use this in V2.

Thanks,
Peng.

>
>Regards,
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] xen/arm: mm: optimize setup_pagetables

[Peng Fan]

Hi Julien,

On Wed, May 11, 2016 at 10:31:49AM +0100, Julien Grall wrote:
>Hi Peng,
>
>I would rename the title: "xen/arm: mm: remove unnecessary tlb flush in
>setup_pagetables".

Thanks. Will fix in V2.

>
>On 11/05/2016 08:59, Peng Fan wrote:
>>Before relocating xen to high address, need to build
>>the mapping BOOT_RELOC_VIRT_START to xen_paddr into
>>boot_pgtable and xen_pgtable.
>>
>>In setup_pagetables, relocate_xen will switch the root
>>page table from boot_pgtable to xen_pgtable/cpu0_pgtable.
>>
>>Before relocate_xen:
>>If touching xen_pgtable, no need to flush_xen_data_tlb_range_va_local.
>>We only need to flush tlb when touch boot_pgtable.
>>And no need to use write_pte for xen_pgtable, we can simply use
>>"xen_second[second_table_offset(dest_va)] = pte;"
>>
>>Also move the piece code near DTB/Fixmap/XEN_VIRT_START to
>>be more easier to understand.
>
>The commit message is rather hard to understand. Can you please rework it
>say:
>  - CPU0 is using the boot pages table and xen_second is not part of them =>
>No need to flush the TLB
>  - Clean up the code (i.e the code movement)
>

Sorry for my bad english. Will refine the commit log in V2.

>However, I would prefer to see 2 patches, one for removing the tlb flush, the
>other to move the code.

Will fix in V2.

>
>[...]
>
>>diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
>>index 94ea054..bebd82f 100644
>>--- a/xen/arch/arm/mm.c
>>+++ b/xen/arch/arm/mm.c
>>@@ -443,12 +443,6 @@ void __init setup_pagetables(unsigned long 
>>boot_phys_offset, paddr_t xen_paddr)
>>  lpae_t pte, *p;
>>  int i;
>>
>>-/* Map the destination in the boot misc area. */
>>-dest_va = BOOT_RELOC_VIRT_START;
>>-pte = mfn_to_xen_entry(xen_paddr >> PAGE_SHIFT, WRITEALLOC);
>>-write_pte(xen_second + second_table_offset(dest_va), pte);
>>-flush_xen_data_tlb_range_va_local(dest_va, SECOND_SIZE);
>>-
>>  /* Calculate virt-to-phys offset for the new location */
>>  phys_offset = xen_paddr - (unsigned long) _start;
>>
>>@@ -498,6 +492,11 @@ void __init setup_pagetables(unsigned long 
>>boot_phys_offset, paddr_t xen_paddr)
>>  /* Map the destination in the boot misc area. */
>>  dest_va = BOOT_RELOC_VIRT_START;
>>  pte = mfn_to_xen_entry(xen_paddr >> PAGE_SHIFT, WRITEALLOC);
>>+xen_second[second_table_offset(dest_va)] = pte;
>>+
>>+/* Map the destination in the boot misc area. */
>>+dest_va = BOOT_RELOC_VIRT_START;
>>+pte = mfn_to_xen_entry(xen_paddr >> PAGE_SHIFT, WRITEALLOC);
>
>Why do you need to recompute "pte" and "dest_va"? They will be the same for
>"boot_second" and "xen_second".

I just change "write_pte(xen_second + second_table_offset(dest_va), pte);" to
"xen_second[second_table_offset(dest_va)] = pte;".

The pte and dest_va are not changed.

>
>Also, I think the code is still confusing in the current state because of the
>comments "Map the destination in the boot misc area.". They need to be
>updated to differentiate xen_second vs boot_second.

Yeah. This confused me when I first read this piece code. I can update
the comments in V2.

Thanks,
Peng.

>
>>  write_pte(boot_second + second_table_offset(dest_va), pte);
>>  flush_xen_data_tlb_range_va_local(dest_va, SECOND_SIZE);
>>  #ifdef CONFIG_ARM_64
>>
>
>Regards,
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH] xen/arm: mm: optimize setup_pagetables

[Peng Fan]

Before relocating xen to high address, need to build
the mapping BOOT_RELOC_VIRT_START to xen_paddr into
boot_pgtable and xen_pgtable.

In setup_pagetables, relocate_xen will switch the root
page table from boot_pgtable to xen_pgtable/cpu0_pgtable.

Before relocate_xen:
If touching xen_pgtable, no need to flush_xen_data_tlb_range_va_local.
We only need to flush tlb when touch boot_pgtable.
And no need to use write_pte for xen_pgtable, we can simply use
"xen_second[second_table_offset(dest_va)] = pte;"

Also move the piece code near DTB/Fixmap/XEN_VIRT_START to
be more easier to understand.

Signed-off-by: Peng Fan <van.free...@gmail.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
Cc: Julien Grall <julien.gr...@arm.com>
---

There is no function change in this patch, I just think
there is no need to use flush_xen_data_tlb_range_va_local and write_pte
at that point. And I tested this patch on AArch64.

 xen/arch/arm/mm.c | 11 +--
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c
index 94ea054..bebd82f 100644
--- a/xen/arch/arm/mm.c
+++ b/xen/arch/arm/mm.c
@@ -443,12 +443,6 @@ void __init setup_pagetables(unsigned long 
boot_phys_offset, paddr_t xen_paddr)
 lpae_t pte, *p;
 int i;
 
-/* Map the destination in the boot misc area. */
-dest_va = BOOT_RELOC_VIRT_START;
-pte = mfn_to_xen_entry(xen_paddr >> PAGE_SHIFT, WRITEALLOC);
-write_pte(xen_second + second_table_offset(dest_va), pte);
-flush_xen_data_tlb_range_va_local(dest_va, SECOND_SIZE);
-
 /* Calculate virt-to-phys offset for the new location */
 phys_offset = xen_paddr - (unsigned long) _start;
 
@@ -498,6 +492,11 @@ void __init setup_pagetables(unsigned long 
boot_phys_offset, paddr_t xen_paddr)
 /* Map the destination in the boot misc area. */
 dest_va = BOOT_RELOC_VIRT_START;
 pte = mfn_to_xen_entry(xen_paddr >> PAGE_SHIFT, WRITEALLOC);
+xen_second[second_table_offset(dest_va)] = pte;
+
+/* Map the destination in the boot misc area. */
+dest_va = BOOT_RELOC_VIRT_START;
+pte = mfn_to_xen_entry(xen_paddr >> PAGE_SHIFT, WRITEALLOC);
 write_pte(boot_second + second_table_offset(dest_va), pte);
 flush_xen_data_tlb_range_va_local(dest_va, SECOND_SIZE);
 #ifdef CONFIG_ARM_64
-- 
2.6.2


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Odroid XU3 support

[Peng Fan]

Hi Julien,

On Mon, May 09, 2016 at 10:49:58AM +0100, Julien Grall wrote:
>
>
>On 08/05/2016 12:59, Peng Fan wrote:
>>Hi Julien,
>
>Hello Peng,
>
>>On Thu, Apr 28, 2016 at 10:50:33AM +0100, Julien Grall wrote:
>>>Hello,
>>>
>>>On 27/04/16 23:53, Suriyan Ramasami wrote:
>>>
>>>>How can I check which core is currently active?
>>>>Judging by this link on big.LITTLE architecture:
>>>>http://forum.odroid.com/viewtopic.php?f=65=2580
>>>>
>>>>result of: cat /proc/cpuinfo | grep "CPU part" is
>>>>CPU part: 0xc07
>>>>
>>>>which stands for A7.
>>>>
>>>>If you do this in dom0, it will show all of them to be 0xc07. They are
>>>>vCPUs after all.
>>>
>>>Which is not a good idea. This means that Linux is not able to detect
>>>potential errata for the underlying cores (in this case the cortex-A15). Also
>>>some userspace may do some runtime optimization based on the kind of CPUs
>>>available in the guest.
>>>
>>>Xen is not ready for big.LITTLE, so I would highly recommend you to disable
>>>either all the Cortex-A15 or Cortex-A7.
>>
>>For Cortex A53 + A72 or A53 + A57, xen also not ready?
>
>Xen is not ready at all for big.LITTLE, no matter the combinations of the
>CPUs.
>
>Some of the cores you mentioned (especially A53 and A57) may contain errata
>that requires workaround in the Linux kernel to avoid potential data
>corruption or deadlock. As the vPIDR will always be equal to the boot CPU
>vMIDR, Linux won't apply the necessary workaround to itself.
>
>So using big and little cores at the same time with Xen is not safe at all,
>even if you create CPU pool with current version of Xen.
>
>>
>>If need to let xen support A53 + A72 or A53 + A57, what work may need to be 
>>done?
>
>I gave some insights later this thread. See:
>
>http://lists.xen.org/archives/html/xen-devel/2016-05/msg00466.html

Thanks for this info. Do you have plan to add bit.LITTLE support for xen?
I would like to use this on AArch64 platform-:)

Thanks,
Peng.

>
>Regards,
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Odroid XU3 support

[Peng Fan]

Hi Julien,

On Thu, Apr 28, 2016 at 10:50:33AM +0100, Julien Grall wrote:
>Hello,
>
>On 27/04/16 23:53, Suriyan Ramasami wrote:
>
>>How can I check which core is currently active?
>>Judging by this link on big.LITTLE architecture:
>>http://forum.odroid.com/viewtopic.php?f=65=2580
>>
>>result of: cat /proc/cpuinfo | grep "CPU part" is
>>CPU part: 0xc07
>>
>>which stands for A7.
>>
>>If you do this in dom0, it will show all of them to be 0xc07. They are
>>vCPUs after all.
>
>Which is not a good idea. This means that Linux is not able to detect
>potential errata for the underlying cores (in this case the cortex-A15). Also
>some userspace may do some runtime optimization based on the kind of CPUs
>available in the guest.
>
>Xen is not ready for big.LITTLE, so I would highly recommend you to disable
>either all the Cortex-A15 or Cortex-A7.

For Cortex A53 + A72 or A53 + A57, xen also not ready?

If need to let xen support A53 + A72 or A53 + A57, what work may need to be 
done?

Thanks,
Peng
>
>For your information, I am planning to send a patch to park any CPUs whose
>MIDR is not matching the boot CPU one.
>
>Regards,
>
>-- 
>Julien Grall
>
>___
>Xen-devel mailing list
>Xen-devel@lists.xen.org
>http://lists.xen.org/xen-devel

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] SMMU, Unhandled context fault

[Peng Fan]

Hi Julien,

On Fri, May 06, 2016 at 01:48:54PM +0100, Julien Grall wrote:
>Hello Peng,
>
>On 03/05/16 14:58, Peng Fan wrote:
>>On Tue, May 03, 2016 at 11:58:17AM +0100, Julien Grall wrote:
>>>On 29/04/16 15:28, Peng Fan wrote:
>>>>Hi Julien,
>>>
>>>Hello Peng,
>>>
>>>>On Thu, Apr 28, 2016 at 02:14:58PM +0100, Julien Grall wrote:
>>>>>>Is there any big difference between XEN SMMU driver and linux SMMU driver?
>>>>>>I know that XEN only support Stage 2. But the initliaization flow is 
>>>>>>almost the same.
>>>>>
>>>>>The SMMU driver for Xen is a port from Linux 3.19-rc0. Since then the Linux
>>>>>driver has been reworked and it might be possible that we have missed some
>>>>>bug fix.
>>>>>
>>>>>Aside that, for Xen, the page tables are always shared between the SMMU and
>>>>>the processor.
>>>>
>>>>Thanks. I shared two picture that dumped using TRACE32.
>>>>
>>>>https://drive.google.com/file/d/0B9ruJqJLIGp7cHhhSFNSNC00MHc/view?usp=sharing
>>>>https://drive.google.com/file/d/0B9ruJqJLIGp7dmlqVllXYTIxajQ/view?usp=sharing
>>>>
>>>>Would you please help check?
>>>
>>>I am sorry but I have got no idea what each columns are supposed to contain.
>>>Can you share more details?
>>
>>I add one more picture.
>>https://drive.google.com/file/d/0B9ruJqJLIGp7SUJOSUVPR1dHRFk/view?usp=sharing
>>The first colum is guest physical address, the second is machine address.
>
>Let's take a step back. Does the MMC is working when used in DOM0?
>
>>The FSR 0x4410, EF is 1, TF is 0. This means the mapping is correct, but 
>>there
>>is an external fault during translation?
>>The PTWF shows An external fault occurred while processing a translation 
>>table walk
>>I may need to check what kind external fault, it maybe not a mapping error.
>
>Edgar mentioned that NSATTR is 0, which means that the input transation as a
>secure attribute. Have you dig into it?

I think we are close to find the root cause, we have a specific hw module
to check the memory transactions. After configure the module to disable
memory access check, I do not met SMMU context fault. I am checking the
configuration for the hw module.

This is not about NSATTR.

Thanks,
Peng.
>
>Regards,
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] SMMU, Unhandled context fault

[Peng Fan]

On Tue, May 03, 2016 at 11:58:17AM +0100, Julien Grall wrote:
>On 29/04/16 15:28, Peng Fan wrote:
>>Hi Julien,
>
>Hello Peng,
>
>>On Thu, Apr 28, 2016 at 02:14:58PM +0100, Julien Grall wrote:
>>>>Is there any big difference between XEN SMMU driver and linux SMMU driver?
>>>>I know that XEN only support Stage 2. But the initliaization flow is almost 
>>>>the same.
>>>
>>>The SMMU driver for Xen is a port from Linux 3.19-rc0. Since then the Linux
>>>driver has been reworked and it might be possible that we have missed some
>>>bug fix.
>>>
>>>Aside that, for Xen, the page tables are always shared between the SMMU and
>>>the processor.
>>
>>Thanks. I shared two picture that dumped using TRACE32.
>>
>>https://drive.google.com/file/d/0B9ruJqJLIGp7cHhhSFNSNC00MHc/view?usp=sharing
>>https://drive.google.com/file/d/0B9ruJqJLIGp7dmlqVllXYTIxajQ/view?usp=sharing
>>
>>Would you please help check?
>
>I am sorry but I have got no idea what each columns are supposed to contain.
>Can you share more details?

I add one more picture.
https://drive.google.com/file/d/0B9ruJqJLIGp7SUJOSUVPR1dHRFk/view?usp=sharing
The first colum is guest physical address, the second is machine address.

The FSR 0x4410, EF is 1, TF is 0. This means the mapping is correct, but 
there
is an external fault during translation?
The PTWF shows An external fault occurred while processing a translation table 
walk
I may need to check what kind external fault, it maybe not a mapping error.

Thanks,
Peng.

>
>Regards,
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] SMMU, Unhandled context fault

[Peng Fan]

Hi Julien,

On Thu, Apr 28, 2016 at 02:14:58PM +0100, Julien Grall wrote:
>Hello,
>
>On 28/04/16 13:56, Peng Fan wrote:
>>On Thu, Apr 28, 2016 at 11:27:22AM +0100, Julien Grall wrote:
>>>
>>>
>>>On 28/04/16 07:39, Peng Fan wrote:
>>>>Hi Julien,
>>>
>>>Hello Peng,
>>>
>>>>On Thu, Apr 28, 2016 at 10:37:54AM +0800, Peng Fan wrote:
>>>>>Hi Julien,
>>>>>On Wed, Apr 27, 2016 at 10:58:28AM +0100, Julien Grall wrote:
>>>>>>Hello Peng,
>>>>>>
>>>>>>On 27/04/2016 03:02, Peng Fan wrote:
>>>>>>>On Tue, Apr 26, 2016 at 04:30:03PM +0200, Edgar E. Iglesias wrote:
>>>>>>>>On Tue, Apr 26, 2016 at 09:56:33PM +0800, Peng Fan wrote:
>>>>>>>>>You mean the PNU bit(Privileged Not Unprivileged) is 1?
>>>>>>>>>I did not met Unhandled context fault each time.
>>>>>>>>>Actually during my serveral boot test, I only met two times.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>I meant the NSSTATE and NSATTR bits in FSYNR are set to zero. I get the
>>>>>>>>impression that the TrustZone state for the SD controller may be
>>>>>>>
>>>>>>>oh. The NSATTR bit is 0. I did not find NSSTATE in my Issue D SMMU spec.
>>>>>>>If without xen, only one linux boots up, sd controller can access memory 
>>>>>>>using
>>>>>>>DMA without issue.
>>>>>>
>>>>>>IIRC, by default Linux baremetal does not protect the devices with the 
>>>>>>SMMU.
>>>>>>
>>>>>>I would recommend you to check whether the SMMUs are in-used and 
>>>>>>configured
>>>>>>to generate a fault (disable_bypass = 1).
>>>>>
>>>>>Ok. I'll set S2CRn to generate fault in xen smmu driver to see whether 
>>>>>SMMUs in-used or not
>>>
>>>I meant in Linux.
>>
>>My bad. Do you mean enabling SMMU driver in Linux with KVM support?
>
>Yes.
>
>[...]
>
>>Is there any big difference between XEN SMMU driver and linux SMMU driver?
>>I know that XEN only support Stage 2. But the initliaization flow is almost 
>>the same.
>
>The SMMU driver for Xen is a port from Linux 3.19-rc0. Since then the Linux
>driver has been reworked and it might be possible that we have missed some
>bug fix.
>
>Aside that, for Xen, the page tables are always shared between the SMMU and
>the processor.

Thanks. I shared two picture that dumped using TRACE32.

https://drive.google.com/file/d/0B9ruJqJLIGp7cHhhSFNSNC00MHc/view?usp=sharing
https://drive.google.com/file/d/0B9ruJqJLIGp7dmlqVllXYTIxajQ/view?usp=sharing

Would you please help check?

The block that marked red, seems not correct. I am also adding debug info
in xen memory allocation part to see what happends.

Thanks,
Peng.

>
>Regards,
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] SMMU, Unhandled context fault

[Peng Fan]

On Thu, Apr 28, 2016 at 11:27:22AM +0100, Julien Grall wrote:
>
>
>On 28/04/16 07:39, Peng Fan wrote:
>>Hi Julien,
>
>Hello Peng,
>
>>On Thu, Apr 28, 2016 at 10:37:54AM +0800, Peng Fan wrote:
>>>Hi Julien,
>>>On Wed, Apr 27, 2016 at 10:58:28AM +0100, Julien Grall wrote:
>>>>Hello Peng,
>>>>
>>>>On 27/04/2016 03:02, Peng Fan wrote:
>>>>>On Tue, Apr 26, 2016 at 04:30:03PM +0200, Edgar E. Iglesias wrote:
>>>>>>On Tue, Apr 26, 2016 at 09:56:33PM +0800, Peng Fan wrote:
>>>>>>>You mean the PNU bit(Privileged Not Unprivileged) is 1?
>>>>>>>I did not met Unhandled context fault each time.
>>>>>>>Actually during my serveral boot test, I only met two times.
>>>>>>
>>>>>>
>>>>>>
>>>>>>I meant the NSSTATE and NSATTR bits in FSYNR are set to zero. I get the
>>>>>>impression that the TrustZone state for the SD controller may be
>>>>>
>>>>>oh. The NSATTR bit is 0. I did not find NSSTATE in my Issue D SMMU spec.
>>>>>If without xen, only one linux boots up, sd controller can access memory 
>>>>>using
>>>>>DMA without issue.
>>>>
>>>>IIRC, by default Linux baremetal does not protect the devices with the SMMU.
>>>>
>>>>I would recommend you to check whether the SMMUs are in-used and configured
>>>>to generate a fault (disable_bypass = 1).
>>>
>>>Ok. I'll set S2CRn to generate fault in xen smmu driver to see whether SMMUs 
>>>in-used or not
>
>I meant in Linux.

My bad. Do you mean enabling SMMU driver in Linux with KVM support?

>
>>>
>>>I found a patch for errata of mmu-500,
>>>https://git.kernel.org/cgit/linux/kernel/git/will/linux.git/commit/?h=iommu/devel=7f0cc5124f5ec66b5b26878ac85137adc6537413
>>>Do you know this?
>>>
>>>I suspect the unstable issue on my platform seems related to this errata,
>>>but i do not have details about this errata.
>
>Have you tried to port the patch to Xen and see if it helps?

I tried. But the value of the register does not change. I am checking with our
IC team.

>
>>
>>I do the following change:
>>  //s2cr = S2CR_TYPE_TRANS | S2CR_PRIVCFG_UNPRIV |
>>   s2cr = S2CR_TYPE_FAULT | S2CR_PRIVCFG_UNPRIV |
>>  (smmu_domain->cfg.cbndx << S2CR_CBNDX_SHIFT);
>>
>>Now XEN SMMU driver reports that:
>>(XEN) smmu: /iommu@5c80: Unexpected global fault, this could be serious
>>(XEN) smmu: /iommu@5c80:GFSR 0x0001, GFSYNR0 0x0004, GFSYNR1 
>>0x0011, GFSYNR2 0x
>>
>>So I think SMMU is working.
>
>Well, it does not help to know whether the SMMU has been correctly
>initialized by Xen.

Is there any big difference between XEN SMMU driver and linux SMMU driver?
I know that XEN only support Stage 2. But the initliaization flow is almost the 
same.

Thanks,
Peng.

>
>Regards,
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] SMMU, Unhandled context fault

[Peng Fan]

Hi Julien,

On Thu, Apr 28, 2016 at 10:37:54AM +0800, Peng Fan wrote:
>Hi Julien,
>On Wed, Apr 27, 2016 at 10:58:28AM +0100, Julien Grall wrote:
>>Hello Peng,
>>
>>On 27/04/2016 03:02, Peng Fan wrote:
>>>On Tue, Apr 26, 2016 at 04:30:03PM +0200, Edgar E. Iglesias wrote:
>>>>On Tue, Apr 26, 2016 at 09:56:33PM +0800, Peng Fan wrote:
>>>>>You mean the PNU bit(Privileged Not Unprivileged) is 1?
>>>>>I did not met Unhandled context fault each time.
>>>>>Actually during my serveral boot test, I only met two times.
>>>>
>>>>
>>>>
>>>>I meant the NSSTATE and NSATTR bits in FSYNR are set to zero. I get the
>>>>impression that the TrustZone state for the SD controller may be
>>>
>>>oh. The NSATTR bit is 0. I did not find NSSTATE in my Issue D SMMU spec.
>>>If without xen, only one linux boots up, sd controller can access memory 
>>>using
>>>DMA without issue.
>>
>>IIRC, by default Linux baremetal does not protect the devices with the SMMU.
>>
>>I would recommend you to check whether the SMMUs are in-used and configured
>>to generate a fault (disable_bypass = 1).
>
>Ok. I'll set S2CRn to generate fault in xen smmu driver to see whether SMMUs 
>in-used or not.
>
>I found a patch for errata of mmu-500,
>https://git.kernel.org/cgit/linux/kernel/git/will/linux.git/commit/?h=iommu/devel=7f0cc5124f5ec66b5b26878ac85137adc6537413
>Do you know this?
>
>I suspect the unstable issue on my platform seems related to this errata,
>but i do not have details about this errata.

I do the following change:
//s2cr = S2CR_TYPE_TRANS | S2CR_PRIVCFG_UNPRIV |
 s2cr = S2CR_TYPE_FAULT | S2CR_PRIVCFG_UNPRIV |
(smmu_domain->cfg.cbndx << S2CR_CBNDX_SHIFT);

Now XEN SMMU driver reports that:
(XEN) smmu: /iommu@5c80: Unexpected global fault, this could be serious
(XEN) smmu: /iommu@5c80:GFSR 0x0001, GFSYNR0 0x0004, GFSYNR1 
0x0011, GFSYNR2 0x

So I think SMMU is working.

Thanks,
Peng.

>
>Thanks,
>Peng.
>
>>
>>Regards,
>>
>>-- 
>>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] SMMU, Unhandled context fault

[Peng Fan]

Hi Julien,
On Wed, Apr 27, 2016 at 10:58:28AM +0100, Julien Grall wrote:
>Hello Peng,
>
>On 27/04/2016 03:02, Peng Fan wrote:
>>On Tue, Apr 26, 2016 at 04:30:03PM +0200, Edgar E. Iglesias wrote:
>>>On Tue, Apr 26, 2016 at 09:56:33PM +0800, Peng Fan wrote:
>>>>You mean the PNU bit(Privileged Not Unprivileged) is 1?
>>>>I did not met Unhandled context fault each time.
>>>>Actually during my serveral boot test, I only met two times.
>>>
>>>
>>>
>>>I meant the NSSTATE and NSATTR bits in FSYNR are set to zero. I get the
>>>impression that the TrustZone state for the SD controller may be
>>
>>oh. The NSATTR bit is 0. I did not find NSSTATE in my Issue D SMMU spec.
>>If without xen, only one linux boots up, sd controller can access memory using
>>DMA without issue.
>
>IIRC, by default Linux baremetal does not protect the devices with the SMMU.
>
>I would recommend you to check whether the SMMUs are in-used and configured
>to generate a fault (disable_bypass = 1).

Ok. I'll set S2CRn to generate fault in xen smmu driver to see whether SMMUs 
in-used or not.

I found a patch for errata of mmu-500,
https://git.kernel.org/cgit/linux/kernel/git/will/linux.git/commit/?h=iommu/devel=7f0cc5124f5ec66b5b26878ac85137adc6537413
Do you know this?

I suspect the unstable issue on my platform seems related to this errata,
but i do not have details about this errata.

Thanks,
Peng.

>
>Regards,
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] SMMU, Unhandled context fault

[Peng Fan]

Hi Julien, Stefano

I met an issue when passthrough a device to DomU, and have no clear idea what's 
wrong.
"
(XEN) smmu: /iommu@5c80: Unhandled context fault: iova=0x42188000, 
fsynr=0x433, cb=0
(XEN) smmu: /iommu@5c80: Unhandled context fault: iova=0x42188020, 
fsynr=0x433, cb=0
"
fsynr is 0x433, PTWF is 1, which means
"An external fault occurred while processing a translation table walk"

My DomU cfg is:
"
kernel = "/root/xen/Image"
device_tree = "/root/xen/sdhc.dtb"
dtdev = [ "/usdhc@5a24" ]
iomem = [ "0x5a240,0x10@0x1a240"]
irqs = [ 257 ]
memory = "128"
name = "domU-sdhc-test"
vcpus = 1
serial="pty"
disk = [ 'phy:/dev/loop0,xvda,w' ]
extra = "console=hvc0 root=/dev/xvda debug=/bin/sh"
"

I saw this patch: http://www.gossamer-threads.com/lists/xen/devel/428030
"xen/arm: Force broadcast of TLB and instruction cache maintenance instructions"
So I guess should we have a similar fix for SMMU TLB?
From my failure log, It should be that TCU can not find the machine address for 
iova(guest physical address).
I remember that When create domU for ARM, the memory is allocated from heap, so 
machine pages
should be there allocated. Also SMMU share the mmu table with domU mmu table.
I guess there maybe something wrong with SMMU TLB related handling. But I have 
no
clear idea on this. Please advise.

Thanks in advance,
Peng.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 2/2] xen/arm64: use shift operator

[Peng Fan]

Hi Julien,
On Fri, Apr 22, 2016 at 05:08:26PM +0100, Julien Grall wrote:
>On 21/04/16 02:06, Peng Fan wrote:
>>Hi Julien,
>
>Hello Peng,
>
>>On Wed, Apr 20, 2016 at 03:44:09PM +0100, Julien Grall wrote:
>>>Hello Peng,
>>>
>>>On 20/04/16 14:54, Peng Fan wrote:
>>>>Use shift operator, but not muliplication.
>>>>No function change.
>>>
>>>Why? The compiler will calculate the address at compilation time.
>>
>>Yeah. The compiler will do this. I just think in asm, we rarely use
>>multiplication. In this file, there is "cmp   x1, #(LPAE_ENTRIES<<3)",
>>here use shift operator but not multiplication.
>
>It took me a bit of time to understand that 8 is the number of byte of an
>LPAE entries. IMHO the "<< 3" would be more confusing.
>
>So I would introduce a macro to get the slot based of an index. Something
>like:
>
>LPAE_SLOT(slot) ((slot) * 8)
>
>And add a comment to explain why 8. You could even introduce LPAE_ENTRY_SIZE.

Thanks for comments. I'll try to submit a new patch following your suggestion.

Thanks,
Peng.

>
>Regards,
>
>>Thanks,
>>Peng.
>>
>>>
>>>Regards,
>>>
>>>>Signed-off-by: Peng Fan <van.free...@gmail.com>
>>>>Cc: Stefano Stabellini <sstabell...@kernel.org>
>>>>Cc: Julien Grall <julien.gr...@arm.com>
>>>>---
>>>>  xen/arch/arm/arm64/head.S | 2 +-
>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>>diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
>>>>index 05e3db0..ad6e593 100644
>>>>--- a/xen/arch/arm/arm64/head.S
>>>>+++ b/xen/arch/arm/arm64/head.S
>>>>@@ -524,7 +524,7 @@ paging:
>>>>  lsl   x2, x2, #THIRD_SHIFT   /* 4K aligned paddr of UART */
>>>>  mov   x3, #PT_DEV_L3
>>>>  orr   x2, x2, x3 /* x2 := 4K dev map including UART */
>>>>-str   x2, [x1, #(FIXMAP_CONSOLE*8)] /* Map it in the first 
>>>>fixmap's slot */
>>>>+str   x2, [x1, #(FIXMAP_CONSOLE << 3)] /* Map it in the first 
>>>>fixmap's slot */
>>>>  1:
>>>>
>>>>  /* Map fixmap into boot_second */
>>>>
>>>
>>>--
>>>Julien Grall
>>
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 2/2] xen/arm64: use shift operator

[Peng Fan]

Hi Julien,

On Wed, Apr 20, 2016 at 03:44:09PM +0100, Julien Grall wrote:
>Hello Peng,
>
>On 20/04/16 14:54, Peng Fan wrote:
>>Use shift operator, but not muliplication.
>>No function change.
>
>Why? The compiler will calculate the address at compilation time.

Yeah. The compiler will do this. I just think in asm, we rarely use
multiplication. In this file, there is "cmp   x1, #(LPAE_ENTRIES<<3)",
here use shift operator but not multiplication.

Thanks,
Peng.

>
>Regards,
>
>>Signed-off-by: Peng Fan <van.free...@gmail.com>
>>Cc: Stefano Stabellini <sstabell...@kernel.org>
>>Cc: Julien Grall <julien.gr...@arm.com>
>>---
>>  xen/arch/arm/arm64/head.S | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>>diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
>>index 05e3db0..ad6e593 100644
>>--- a/xen/arch/arm/arm64/head.S
>>+++ b/xen/arch/arm/arm64/head.S
>>@@ -524,7 +524,7 @@ paging:
>>  lsl   x2, x2, #THIRD_SHIFT   /* 4K aligned paddr of UART */
>>  mov   x3, #PT_DEV_L3
>>  orr   x2, x2, x3 /* x2 := 4K dev map including UART */
>>-str   x2, [x1, #(FIXMAP_CONSOLE*8)] /* Map it in the first fixmap's 
>>slot */
>>+str   x2, [x1, #(FIXMAP_CONSOLE << 3)] /* Map it in the first 
>>fixmap's slot */
>>  1:
>>
>>  /* Map fixmap into boot_second */
>>
>
>-- 
>Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH 1/2] xen/arm64: correct comments

[Peng Fan]

The 'Base address for 4K mapping' is '(x19 >> THIRD_SHIFT) << THIRD_SHIFT'.
Also we are building 4K page mapping, not section mapping.

Signed-off-by: Peng Fan <van.free...@gmail.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
Cc: Julien Grall <julien.gr...@arm.com>
---
 xen/arch/arm/arm64/head.S | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
index 946e2c9..05e3db0 100644
--- a/xen/arch/arm/arm64/head.S
+++ b/xen/arch/arm/arm64/head.S
@@ -471,10 +471,10 @@ skip_bss:
 ldr   x4, =boot_third
 add   x4, x4, x20/* x4 := paddr (boot_third) */
 
-lsr   x2, x19, #THIRD_SHIFT  /* Base address for 4K mapping */
-lsl   x2, x2, #THIRD_SHIFT
-mov   x3, #PT_MEM_L3 /* x2 := Section map */
-orr   x2, x2, x3
+lsr   x2, x19, #THIRD_SHIFT
+lsl   x2, x2, #THIRD_SHIFT  /* Base address for 4K mapping */
+mov   x3, #PT_MEM_L3
+orr   x2, x2, x3/* x2 := Page map */
 
 /* ... map of vaddr(start) in boot_third */
 mov   x1, xzr
-- 
2.6.2


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH 2/2] xen/arm64: use shift operator

[Peng Fan]

Use shift operator, but not muliplication.
No function change.

Signed-off-by: Peng Fan <van.free...@gmail.com>
Cc: Stefano Stabellini <sstabell...@kernel.org>
Cc: Julien Grall <julien.gr...@arm.com>
---
 xen/arch/arm/arm64/head.S | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
index 05e3db0..ad6e593 100644
--- a/xen/arch/arm/arm64/head.S
+++ b/xen/arch/arm/arm64/head.S
@@ -524,7 +524,7 @@ paging:
 lsl   x2, x2, #THIRD_SHIFT   /* 4K aligned paddr of UART */
 mov   x3, #PT_DEV_L3
 orr   x2, x2, x3 /* x2 := 4K dev map including UART */
-str   x2, [x1, #(FIXMAP_CONSOLE*8)] /* Map it in the first fixmap's 
slot */
+str   x2, [x1, #(FIXMAP_CONSOLE << 3)] /* Map it in the first fixmap's 
slot */
 1:
 
 /* Map fixmap into boot_second */
-- 
2.6.2


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC V2] xen: interface: introduce pvclk interface

[Peng Fan]

Hi Ian,

On Thu, Jan 21, 2016 at 12:49:24PM +, Ian Campbell wrote:
>On Thu, 2016-01-21 at 20:35 +0800, Peng Fan wrote:
>> On Thu, Jan 21, 2016 at 12:26:04PM +, Ian Campbell wrote:
>> > Would adding a dummy fixed-clock[0] (or several of them) to the guest
>> > passthrough DT satisfy the driver's requirements? They would be hardcoded
>> > to whatever rate dom0 and/or the tools has decided upon (and had set in the
>> > real h/w).
>> 
>> If using this way, we have the assumption that DomU device driver would not
>> change the rate of the clock driving the device. I am not sure whether this 
>> is
>> ok for so many platform devices based ARM core.
>
>Don't (non-buggy) drivers already need to cope with the possibility that
>the clk core may not be able to satisfy their request for a particular rate
>due to constraints from other ip blocks?
>
>I would also expect the user to want to configure things in dom0 such that
>the specific drivers used in domU are satisfied with what they get (which
>is a bit fiddly perhaps, but platform device passthrough already is
>somewhat that way).
>
>> In /sys/kernel/debug/clk/, there are clk tree info, but
>> clk api are not exposed to userspace as far as I know, so
>> if using sysfs interface to set a known fixed rate or enable/disable the 
>> clock,
>> we need to expose the clk info to userspace.
>
>That seems possible to arrange given a use case for it though, a SMOP (and
>convincing the clk maintainer of the need).

Sorry to bother you again on this.

For the fixed clock method, I add such a xl cfg entry: 
devclk=["uart_root_clk,2400", "gpu_root_clk,2"].
After add code for writeing sysfs clk node in libxl, the clk can be set 
rate/prepared/enabled.
But I do not know the resource release path, when need to shutdown/destroy a 
domain. For example:
when domain creation, the clk is prepared, when domain destroied, I want the 
clk unprepared. But I
do not find out the code path to releasing resource.

Also, since need to add fixed-clock nodes in DomU dts, do you agree to let 
libxl or xc to create the nodes
in dts, but not mannually add them in passthrough node?

Thanks,
Peng.

>
>Worst case is a xen-clkback driver or perhaps even vfio will want to do
>something like this, we can't normally use vfio on Xen, but in this case
>perhaps we could.
>
>
>> Jan said using hypercall in the other mail, do you have any ideas about
>> this?
>
>This would need a whole clock infrastructure in Xen, wouldn't it? I
>described why that is not currently available in an earlier mail, and also
>my opinion that doing the whole thing in Xen would involve pulling in far
>too much SoC specific code for each specific platform.
>
>Ian.
>> 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC V2] xen: interface: introduce pvclk interface

[Peng Fan]

Hi Jan,

On Fri, Jan 22, 2016 at 12:36:31AM -0700, Jan Beulich wrote:
 On 22.01.16 at 02:56,  wrote:
>> On Thu, Jan 21, 2016 at 05:52:12AM -0700, Jan Beulich wrote:
>>>At the very least it would need to be avoided by denying the request.
>>>Upon shared use, either all parties agree, or only one may use the
>>>clock. And passing through a (platform) device would therefore imply
>>>validating that the needed clock(s) are available to the target domain.
>>>Doing this in a consistent way with all control in one component's
>>>hands seems doable only if hypervisor and/or tool stack are the
>>>controlling (and arbitrating) entity. In the end this is one of the
>>>reasons why to me a simple PV I/O interface doesn't seem suitable
>>>here.
>> 
>> How about let userspace libxl pvclk code to denying the request?
>
>Userspace would be fine, but

Then you are ok with the pvclk way to handle clock for platform device 
passthrough?

>- How would this fit in your frontend/backend model, where
>  userspace shouldn't be involved at all?

rethought about this. clk is binded to device. we can not passthrough
one device to two guest, so this means we can not let two different
guest access one clk input. Since this is mainly for embedded products,
just as Ian said "experts option", the developer should be aware of
the clk sharing between two device.

If we truly need to let userspace deny the request. If one clk
already assigned to Dom1, then the toolstack need to fail
the creation of Dom2, if Dom2 want to use the same clock.

In xl configuraiton file for Dom1, introduce such an entry,
vclks=["gpu_root_clk,uart2_root_clk,usdhc2_root_clk"]
and toolstack will create the xenstore nodes.
/local/domain/0/backend/vclk/1/0/nrclks-->3
/local/domain/0/backend/vclk/1/0/clk-0/name-->gpu_root_clk
/local/domain/0/backend/vclk/1/0/clk-1/name-->uart2_root_clk
/local/domain/0/backend/vclk/1/0/clk-2/name-->usdhc2_root_clk

/local/domain/1/device/vclk/0/clk-ring-ref
/local/domain/1/device/vclk/0/event-channel

The DomU dts also contains the clk names. Maybe the dts for clk node can 
be created by the toolstack.

If Dom2 also want to use gpu_root_clk, it should be blocked by toolstack.


Thanks,
Peng.
>- Libxl may be a little too high up the stack, libxc would seem a
>  more appropriate place to me (but that's subject to tools
>  maintainers disagreeing with me).
>
>Jan
>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC V2] xen: interface: introduce pvclk interface

[Peng Fan]

Hi Jan,

On Fri, Jan 22, 2016 at 03:25:40AM -0700, Jan Beulich wrote:
 On 22.01.16 at 10:27,  wrote:
>> Hi Jan,
>> 
>> On Fri, Jan 22, 2016 at 12:36:31AM -0700, Jan Beulich wrote:
>> On 22.01.16 at 02:56,  wrote:
 On Thu, Jan 21, 2016 at 05:52:12AM -0700, Jan Beulich wrote:
>At the very least it would need to be avoided by denying the request.
>Upon shared use, either all parties agree, or only one may use the
>clock. And passing through a (platform) device would therefore imply
>validating that the needed clock(s) are available to the target domain.
>Doing this in a consistent way with all control in one component's
>hands seems doable only if hypervisor and/or tool stack are the
>controlling (and arbitrating) entity. In the end this is one of the
>reasons why to me a simple PV I/O interface doesn't seem suitable
>here.
 
 How about let userspace libxl pvclk code to denying the request?
>>>
>>>Userspace would be fine, but
>> 
>> Then you are ok with the pvclk way to handle clock for platform device 
>> passthrough?
>
>No, not really. While I accept that doing clock management in the
>hypervisor is undesirable, we're still not at the point where such
>a frontend/backend pair would look like the only possible route
>out of a dilemma, and I continue to think that this proposed model
>should indeed only be the last resort.

Thanks for following the thread and giving comments.
Alougth this frustrate me, we still need to find a better option for this.

>
>In particular, with the user space exposure of clock control
>discussed in another sub-thread, the next best option would
>seem to be to handle this via emulation in a device model. Yes,
>ARM guests currently have no qemu attached to them, but I
>guess sooner or later this will need to change anyway.

I have not look into qemu for xen.
If using qemu, then we still need to expose the clk interface to userspace?

>
>>>- How would this fit in your frontend/backend model, where
>>>  userspace shouldn't be involved at all?
>> 
>> rethought about this. clk is binded to device. we can not passthrough
>> one device to two guest, so this means we can not let two different
>> guest access one clk input. Since this is mainly for embedded products,
>> just as Ian said "experts option", the developer should be aware of
>> the clk sharing between two device.
>> 
>> If we truly need to let userspace deny the request. If one clk
>> already assigned to Dom1, then the toolstack need to fail
>> the creation of Dom2, if Dom2 want to use the same clock.
>
>I.e. you're now proposing actual assignment of clocks to guests?
>That's at least one step in the (from my pov) right direction...

Based on the pvclk, I am coding the userspace tool part. Alought we have
not find a good solution for this, I first need it work on my platform.

Later I'll also try the fixed clock way.

Thanks,
Peng.
>
>Jan
>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC V2] xen: interface: introduce pvclk interface

[Peng Fan]

Hi Ian,

On Thu, Jan 21, 2016 at 10:19:32AM +, Ian Campbell wrote:
>On Thu, 2016-01-21 at 16:59 +0800, Peng Fan wrote:
>> 
>> To platform device of ARM, hypervisor is responsible for the mapping
>> between machine address and guest physical address, also responsible
>> for the irq mapping.
>> 
>> But to embedded ARM SoC, the hardware clk IP is handled in Dom0.
>
>Arguably Xen ought to be the one to do this, but we have determined
>(rightly, I think) that doing so for the entirely clk tree of an SoC would
>involve importing an unmanageable amount of code into Xen[0].
>
>In the meantime we defer this to Dom0.
>
>I wonder though if it would be possible to manage the clocks for a
>passthrough device from the toolstack, i.e. is there a sysfs node where one
>can say "keep the clock for this device enabled (at xMhz) even though you
>think the device is unused"?

I am afraid not. The linux device driver without xen can work well, because
there is clk tree and "clk_get,clk_prepare,clk_set_rate" work well in the 
driver.
I do not want to remove the clk apis usage from the device driver for xen, 
because the driver
also serves when no xen support. The pvclk interface is mainly to let the clk 
api can work as no xen.

Introduce pvclk may introduce more stuff, I have no more idea for now, 
if want to let the device driver unchanged and work well for passed through 
device. (:

>
>If so (or if it can be easily added) then the toolstack would just need to
>manage that value as part of the passthrough process rather than having the
>frontend do it via a PV protocol.
>
>Ian.
>
>[0] I'd like at some point for Xen to gain sufficient knowledge of clock IP
>to minimally control things like the CPU and DRAM clocks etc, but that
>needn't imply full clock tree support and would hopefully be a manageable
>amount of code, which would (hopefully) mostly be about trapping and
>emulating writes to one or two clock controllers per platform and
>arbitrating a small set of bits (while allowing dom0 to control the
>others). This is at least a medium if not long term idea though, and for
>all I know it might turn out to be unworkable in practice.

I am happy to see that if one day, xen takes part of the clock management to 
minimize power usage.
These few days, I also look into the power management of xen. seems it mainly 
serves for x86.
Let xen hypervisor handle cpu and dram clocks may be better than let dom0.

Thanks,
Peng.

>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC V2] xen: interface: introduce pvclk interface

[Peng Fan]

Hi Jan,

On Thu, Jan 21, 2016 at 03:21:38AM -0700, Jan Beulich wrote:
 On 21.01.16 at 09:59,  wrote:
>> uart2 needs clock IMX7D_UART2_ROOT_CLK from the ccm.
>> passthrough uart2, hypervisor handles the reg and interrupts, that is 
>> because
>> hypervisor handles the memory map and the interrupt controller(GIC). But 
>> here
>> CCM is not handled by hypervisor, it is handled by Dom0.
>
>This, I take it, describes the current state, which doesn't make clear
>whether this is intentionally that way (and can't be changed), or
>just happens to be that way because so far it didn't matter.
>
>> For ARMV8 server products, I am not sure whether hypercall is better; but to 
>> my case, it's not feasible to use hypercall.
>
>Because of ...?

I guess you mean DomU issues hypercall and Xen forwards the request to Dom0,
then Dom0 reply the response?

If you experts think pvclk is not a good way to handle the clock for passed 
through
device, I can try hypercall way.

>
>> Dom0 handles all the clocks, DomU just send request to Dom0 and ask Dom0 to
>> enable/disable/set rate for a clock for the device. So I think it's okay
>> for multipile parties, the clk subsystem in Dom0 can handle mutiple requests 
>> even if the clock is shared.
>
>I.e. if one party sets one rate, and later another party sets
>a different rate, things are going to work fine? If so, why would
>the different parties need control over the rate in the first place?

oh. thanks for teaching me. If two IPs share one clock, and IP1 for Dom0, IP2 
for DomU,
Dom0 needs clock work at 20Hz, but DomU want clock work at 40Hz. pvclk can not 
avoid this.

If not using pvclk, we develop a new method to handle clock. I guest we can 
also not avoid this?

Thanks,
Peng.

>
>Jan
>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC V2] xen: interface: introduce pvclk interface

[Peng Fan]

Hi Ian,

On Thu, Jan 21, 2016 at 12:26:04PM +, Ian Campbell wrote:
>On Thu, 2016-01-21 at 19:55 +0800, Peng Fan wrote:
>> Hi Ian,
>> 
>> On Thu, Jan 21, 2016 at 10:19:32AM +, Ian Campbell wrote:
>> > On Thu, 2016-01-21 at 16:59 +0800, Peng Fan wrote:
>> > >  
>> > > To platform device of ARM, hypervisor is responsible for the mapping
>> > > between machine address and guest physical address, also responsible
>> > > for the irq mapping.
>> > > 
>> > > But to embedded ARM SoC, the hardware clk IP is handled in Dom0.
>> > 
>> > Arguably Xen ought to be the one to do this, but we have determined
>> > (rightly, I think) that doing so for the entirely clk tree of an SoC
>> > would
>> > involve importing an unmanageable amount of code into Xen[0].
>> > 
>> > In the meantime we defer this to Dom0.
>> > 
>> > I wonder though if it would be possible to manage the clocks for a
>> > passthrough device from the toolstack, i.e. is there a sysfs node where
>> > one
>> > can say "keep the clock for this device enabled (at xMhz) even though
>> > you
>> > think the device is unused"?
>> 
>> I am afraid not. The linux device driver without xen can work well,
>> because
>> there is clk tree and "clk_get,clk_prepare,clk_set_rate" work well in the
>> driver.
>> I do not want to remove the clk apis usage from the device driver for
>> xen, because the driver
>> also serves when no xen support. The pvclk interface is mainly to let the
>> clk api can work as no xen.
>
>Would adding a dummy fixed-clock[0] (or several of them) to the guest
>passthrough DT satisfy the driver's requirements? They would be hardcoded
>to whatever rate dom0 and/or the tools has decided upon (and had set in the
>real h/w).

If using this way, we have the assumption that DomU device driver would not
change the rate of the clock driving the device. I am not sure whether this is
ok for so many platform devices based ARM core.

In /sys/kernel/debug/clk/, there are clk tree info, but
clk api are not exposed to userspace as far as I know, so
if using sysfs interface to set a known fixed rate or enable/disable the clock,
we need to expose the clk info to userspace.

Jan said using hypercall in the other mail, do you have any ideas about this?

Thanks,
Peng.

>
>Ian.
>
>[0] Documentation/devicetree/bindings/clock/fixed-clock.txt
>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC V2] xen: interface: introduce pvclk interface

[Peng Fan]

Hi Jan,

On Thu, Jan 21, 2016 at 12:53:01AM -0700, Jan Beulich wrote:
 On 21.01.16 at 02:29,  wrote:
>> The platform device passthrough part for arm is to mapping the machine io 
>> address
>> to the guest physical io address. Then guest can map the phsical io address 
>> to its
>> own virtual address, then by accessing virtual address, guest can access 
>> machine io address space.
>> So the platform device passthrough does not needs frontend/backend driver to 
>> support, except smmu is handled by xen.
>> 
>> But the platform device needs clk to drive the hardware IP, also may needs 
>> pinmux settings.
>> the driver in guest needs to drive the hardware IP passed through to the 
>> guest, so it needs to operate on the clk.
>> 
>> Just pasted comments from George for V1:
>> 
>> "
>> Just speaking from the perspective of a Xen dev who's not an ARM dev:
>> a few more words on the relationship between pvclk and
>> device-passthrough would be helpful to set the context.  It sounds
>> like:
>> 
>> * On ARM, passing through a device requires a clocksource (at least
>> for many devices)
>> 
>> * dom0 has the hardware clocksource, but at the moment domUs don't
>> have a suitable clocksource
>> 
>> * This patch implements pvclk front/backend suitable for such devices
>> 
>> Is that right?  In which case something like the following would be helpful:
>> 
>> "This patch introduces pvclk, a paravirtualized clock source suitable
>> for devices to use when passing through to domUs on ARM systems."
>> "
>
>That's a possible perspective to take, but not the only one. In
>fact, coming to what I said previously, I wonder whether placing
>the "backend" in Dom0 is the right thing in the first place -
>fundamentally arbitration of hardware use should be done
>(or at least checked/enforced) by the hypervisor. I.e. just like
>while Dom0 may assign a PCI device to a guest, the hypervisor
>is in charge of actually making all the resources needed for this
>to work accessible to the guest, and/or verifying that permissions
>are in place (like e.g. when setting up interrupts). Yet the model
>proposed here completely bypasses the hypervisor afaict.

To platform device of ARM, hypervisor is responsible for the mapping
between machine address and guest physical address, also responsible
for the irq mapping.

But to embedded ARM SoC, the hardware clk IP is handled in Dom0.
On my i.MX platform, the hardware clk IP named
Clock Controller Module will output clks to drive different IPs, such as
uart,gpu,lcd,sd controller,scsi controller,pcie controller.
The hardware clock IP is not same for all ARM SoC vendors. ARM has
spec, such as GIC and SMMU to ask SoC vendors follow the spec, then
it's easy to let hypervisor handle them. But there is no common spec
for the clock IP.

>
>Are there connections between a platform device and its clock(s),
>e.g. in DT? If so, wouldn't it be possible for granting access to a
>platform device to imply granting control of the respective clock?

clock hardware IP is also a device. The following is partial dts for i.MX7Dual.
clks: ccm@3038 {
compatible = "fsl,imx7d-ccm";
reg = <0x3038 0x1>;
interrupts = ,
 ;
#clock-cells = <1>;
clocks = <>, <>;
clock-names = "ckil", "osc";
};

uart2: serial@3089 {
compatible = "fsl,imx7d-uart",
 "fsl,imx6q-uart";
reg = <0x3089 0x1>;
interrupts = ;
clocks = < IMX7D_UART2_ROOT_CLK>,
< IMX7D_UART2_ROOT_CLK>;
clock-names = "ipg", "per";
status = "disabled";
};
uart2 needs clock IMX7D_UART2_ROOT_CLK from the ccm.
passthrough uart2, hypervisor handles the reg and interrupts, that is because
hypervisor handles the memory map and the interrupt controller(GIC). But here
CCM is not handled by hypervisor, it is handled by Dom0.

>In which case clock control might perhaps better become a
>hypercall based mechanism? And further - are all clocks in use for
>at most one platform device (i.e. is there no sharing possible)? If
>not, how do you envision to make multiple parties agree on the
>clock settings and state?

For ARMV8 server products, I am not sure whether hypercall is better; but to my
case, it's not feasible to use hypercall.

Dom0 handles all the clocks, DomU just send request to Dom0 and ask Dom0 to
enable/disable/set rate for a clock for the device. So I think it's okay
for multipile parties, the clk subsystem in Dom0 can handle 

Re: [Xen-devel] [RFC V2] xen: interface: introduce pvclk interface

[Peng Fan]

Hi Jan,

On Thu, Jan 21, 2016 at 05:52:12AM -0700, Jan Beulich wrote:
 On 21.01.16 at 13:06,  wrote:
>> On Thu, Jan 21, 2016 at 03:21:38AM -0700, Jan Beulich wrote:
>> On 21.01.16 at 09:59,  wrote:
 uart2 needs clock IMX7D_UART2_ROOT_CLK from the ccm.
 passthrough uart2, hypervisor handles the reg and interrupts, that is 
 because
 hypervisor handles the memory map and the interrupt controller(GIC). But 
 here
 CCM is not handled by hypervisor, it is handled by Dom0.
>>>
>>>This, I take it, describes the current state, which doesn't make clear
>>>whether this is intentionally that way (and can't be changed), or
>>>just happens to be that way because so far it didn't matter.
>>>
 For ARMV8 server products, I am not sure whether hypercall is better; but 
 to
 my case, it's not feasible to use hypercall.
>>>
>>>Because of ...?
>> 
>> I guess you mean DomU issues hypercall and Xen forwards the request to Dom0,
>> then Dom0 reply the response?
>
>Well, no, that wouldn't be a desirable (or even sane) model.
>
 Dom0 handles all the clocks, DomU just send request to Dom0 and ask Dom0 to
 enable/disable/set rate for a clock for the device. So I think it's okay
 for multipile parties, the clk subsystem in Dom0 can handle mutiple 
 requests 
 even if the clock is shared.
>>>
>>>I.e. if one party sets one rate, and later another party sets
>>>a different rate, things are going to work fine? If so, why would
>>>the different parties need control over the rate in the first place?
>> 
>> oh. thanks for teaching me. If two IPs share one clock, and IP1 for Dom0, 
>> IP2 for DomU,
>> Dom0 needs clock work at 20Hz, but DomU want clock work at 40Hz. pvclk can 
>> not avoid this.
>
>But you mustn't allow a DomU to affect Dom0, nor may two DomU-s
>interact in such a way with one another.
>
>> If not using pvclk, we develop a new method to handle clock. I guest we can 
>> also not avoid this?
>
>At the very least it would need to be avoided by denying the request.
>Upon shared use, either all parties agree, or only one may use the
>clock. And passing through a (platform) device would therefore imply
>validating that the needed clock(s) are available to the target domain.
>Doing this in a consistent way with all control in one component's
>hands seems doable only if hypervisor and/or tool stack are the
>controlling (and arbitrating) entity. In the end this is one of the
>reasons why to me a simple PV I/O interface doesn't seem suitable
>here.

How about let userspace libxl pvclk code to denying the request?

If the pvclk interface is not desirable, I have no more idea on this for now(:

Thanks,
Peng.
>
>Jan
>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC V2] xen: interface: introduce pvclk interface

[Peng Fan]

Hi Ian,

On Thu, Jan 21, 2016 at 12:49:24PM +, Ian Campbell wrote:
>On Thu, 2016-01-21 at 20:35 +0800, Peng Fan wrote:
>> On Thu, Jan 21, 2016 at 12:26:04PM +, Ian Campbell wrote:
>> > Would adding a dummy fixed-clock[0] (or several of them) to the guest
>> > passthrough DT satisfy the driver's requirements? They would be hardcoded
>> > to whatever rate dom0 and/or the tools has decided upon (and had set in the
>> > real h/w).
>> 
>> If using this way, we have the assumption that DomU device driver would not
>> change the rate of the clock driving the device. I am not sure whether this 
>> is
>> ok for so many platform devices based ARM core.
>
>Don't (non-buggy) drivers already need to cope with the possibility that
>the clk core may not be able to satisfy their request for a particular rate
>due to constraints from other ip blocks?

Yeah. the drivers should be ready to cope with this.

>
>I would also expect the user to want to configure things in dom0 such that
>the specific drivers used in domU are satisfied with what they get (which
>is a bit fiddly perhaps, but platform device passthrough already is
>somewhat that way).

Let user to configure such as pinctrl and clk for the platform device in Dom0.
But the clk is set at a fixed rate and let the device in DomU use it.
To embedded products, power usage is sensitive. If I passed through GPU 
controller
to DomU, and fix the clock rate at 400M in Dom0, it will consume more power 
than let DomU
change the rate. Also platform device passthrough is not hotplugable now, this 
means
that the gpu controller will always be alive and clock is always enabled.
I am not sure whether this is acceptable or not.

>
>> In /sys/kernel/debug/clk/, there are clk tree info, but
>> clk api are not exposed to userspace as far as I know, so
>> if using sysfs interface to set a known fixed rate or enable/disable the 
>> clock,
>> we need to expose the clk info to userspace.
>
>That seems possible to arrange given a use case for it though, a SMOP (and
>convincing the clk maintainer of the need).
>
>Worst case is a xen-clkback driver or perhaps even vfio will want to do
>something like this, we can't normally use vfio on Xen, but in this case
>perhaps we could.

vfio is new to me. Just google it. iommu is not a must for platform device 
passthrough.
If the platform device supports DMA, then smmu is a must.

>
>
>> Jan said using hypercall in the other mail, do you have any ideas about
>> this?
>
>This would need a whole clock infrastructure in Xen, wouldn't it? I
>described why that is not currently available in an earlier mail, and also
>my opinion that doing the whole thing in Xen would involve pulling in far
>too much SoC specific code for each specific platform.

Thanks for clarifying.

Thanks,
Peng.

>
>Ian.
>> 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC V2] xen: interface: introduce pvclk interface

[Peng Fan]

Hi Ian and Stefano,

On Thu, Jan 21, 2016 at 04:11:45PM +, Stefano Stabellini wrote:
>On Thu, 21 Jan 2016, Ian Campbell wrote:
>> On Thu, 2016-01-21 at 12:55 +, Stefano Stabellini wrote:
>> > On Thu, 21 Jan 2016, Peng Fan wrote:
>> > > Hi Ian,
>> > > 
>> > > On Thu, Jan 21, 2016 at 12:26:04PM +, Ian Campbell wrote:
>> > > > On Thu, 2016-01-21 at 19:55 +0800, Peng Fan wrote:
>> > > > > Hi Ian,
>> > > > > 
>> > > > > On Thu, Jan 21, 2016 at 10:19:32AM +, Ian Campbell wrote:
>> > > > > > On Thu, 2016-01-21 at 16:59 +0800, Peng Fan wrote:
>> > > > > > >  
>> > > > > > > To platform device of ARM, hypervisor is responsible for the
>> > > > > > > mapping
>> > > > > > > between machine address and guest physical address, also
>> > > > > > > responsible
>> > > > > > > for the irq mapping.
>> > > > > > > 
>> > > > > > > But to embedded ARM SoC, the hardware clk IP is handled in
>> > > > > > > Dom0.
>> > > > > > 
>> > > > > > Arguably Xen ought to be the one to do this, but we have
>> > > > > > determined
>> > > > > > (rightly, I think) that doing so for the entirely clk tree of an
>> > > > > > SoC
>> > > > > > would
>> > > > > > involve importing an unmanageable amount of code into Xen[0].
>> > > > > > 
>> > > > > > In the meantime we defer this to Dom0.
>> > > > > > 
>> > > > > > I wonder though if it would be possible to manage the clocks for
>> > > > > > a
>> > > > > > passthrough device from the toolstack, i.e. is there a sysfs node
>> > > > > > where
>> > > > > > one
>> > > > > > can say "keep the clock for this device enabled (at xMhz) even
>> > > > > > though
>> > > > > > you
>> > > > > > think the device is unused"?
>> > > > > 
>> > > > > I am afraid not. The linux device driver without xen can work well,
>> > > > > because
>> > > > > there is clk tree and "clk_get,clk_prepare,clk_set_rate" work well
>> > > > > in the
>> > > > > driver.
>> > > > > I do not want to remove the clk apis usage from the device driver
>> > > > > for
>> > > > > xen, because the driver
>> > > > > also serves when no xen support. The pvclk interface is mainly to
>> > > > > let the
>> > > > > clk api can work as no xen.
>> > > > 
>> > > > Would adding a dummy fixed-clock[0] (or several of them) to the guest
>> > > > passthrough DT satisfy the driver's requirements? They would be
>> > > > hardcoded
>> > > > to whatever rate dom0 and/or the tools has decided upon (and had set
>> > > > in the
>> > > > real h/w).
>> > > 
>> > > If using this way, we have the assumption that DomU device driver would
>> > > not
>> > > change the rate of the clock driving the device. I am not sure whether
>> > > this is
>> > > ok for so many platform devices based ARM core.
>> > > 
>> > > In /sys/kernel/debug/clk/, there are clk tree info, but
>> > > clk api are not exposed to userspace as far as I know, so
>> > > if using sysfs interface to set a known fixed rate or enable/disable
>> > > the clock,
>> > > we need to expose the clk info to userspace.
>> > 
>> > Keeping in mind that we might now want to let DomU change the actual
>> 
>> s/now/not/?
>
>Sorry, I meant "we might not want to let DomU change" ...

Why not want to let DomU change? I can not get this.

>
>
>> > clock frequency anyway, exposing a dummy clock to keep the DomU driver
>> > happy might be a good option.
>> > 
>> > However whether we set the frequency from the Dom0 kernel or from the
>> > toolstack, how do we find out the right clock rate? From a grep in the
>> > kernel sources, it looks like some drivers still have hardcoded values.
>> > 
>> > Asking the user to provide the clock rate seems a bit too much to me.
>> 

Re: [Xen-devel] [RFC V2] xen: interface: introduce pvclk interface

[Peng Fan]

Hi Jan,

On Wed, Jan 20, 2016 at 07:52:58AM -0700, Jan Beulich wrote:
 On 20.01.16 at 15:37,  wrote:
>> On Wed, Jan 20, 2016 at 07:16:36AM -0700, Jan Beulich wrote:
>> On 20.01.16 at 15:05,  wrote:
 On Wed, Jan 20, 2016 at 05:01:40AM -0700, Jan Beulich wrote:
 On 20.01.16 at 12:40,  wrote:
>> On Wed, Jan 20, 2016 at 03:14:20AM -0700, Jan Beulich wrote:
>> On 20.01.16 at 09:31,  wrote:
>>>And what I'm missing throughout the file is some description of how
>>>clock events (interrupts?) are actually meant to make it into the
>>>guest.
>> 
>> I have a simple implementation v1 patch for linux, 
>> http://lists.xen.org/archives/html/xen-devel/2016-01/msg01860.html.
>> You can see how it works.
>
>No, sorry, that's not the point of the inquiry. It seems to me that
>there are more aspects to this interface, not directly related to
>the request/reply protocol written down here, which need to be
>spelled out event if they don't require any particular definitions
>or type declarations.
 
 I try to follow you about clock events(interrupts?), not sure whether I 
 understand correct or not.
 clock in this file is the clk for a device. In linux side, it managed by 
 clk 
 subsystem, drivers/clk/xx.
 This is not the clock events or clock source in drivers/clocksource/xx.
 For the pvclk interface, there will be no interrupt for the guest.
>>>
>>>Then (also considering the set of commands you propose) what
>>>use is the clock to the guest? It can't get events from it, it can't
>>>read its current value, all it can is get/set its rate, enable/disable,
>>>and prepare/unprepare it. I may be lacking some ARM knowledge
>>>here, but all of this looks pretty odd to me.
>> 
>> I follow this 
>> https://events.linuxfoundation.org/sites/events/files/slides/talk_5.pdf to 
>> do 
>> platform device
>> passthrough on ARM platform. I met the same issue as note in the ppt, at 
>> page 24.
>> 
>> In my test case, the uart driver works well without xen. There is serveral 
>> function calls in the driver, such as
>> clk = clk_get("uart2_root"),rate = clk_get_rate(clk), use rate to set 
>> baudrate for uart.
>> 
>> 
>> There is a clk tree in kernel without XEN or dom0 kernel like the following 
>> (only an example):
>> osc -
>> |-->pll1
>> |-->pll2
>>  |-->pll2_div
>>   |-->uart2_gate
>>   |-->uart2_root  --> clk for uart2
>> 
>> uart2_root is directly handled by Dom0.If I assign uart2 to DomU, DomU does
>> not have the clk tree as above, so DomU can not handle directly handle 
>> uart2_root and the uart2 driver in
>> DomU will run into failure to intialize the uart2 hardware IP.
>> 
>> So I introudce pvclk. Pass the operation for uart2_root in DomU to Dom0 and 
>> Dom0 directly handle the clock management hardware IP.
>> 
>> Hope this is clear.
>
>I'm afraid it's not, but it now looks even more like this is too ARM
>specific for me to comment. I wonder whether a generic (not
>ARM specific) PV I/O protocol is actually warranted here. In my
>(presumably too simplistic) view controlling the clock of a passed
>through platform device should be part of that pass-through,
>not the subject of a PV side band protocol.From an abstract
>pov the passed through device should work without any PV I/O
>protocol; such protocols are generally only to accelerate I/O,
>not to make things work in the first place.

The platform device passthrough part for arm is to mapping the machine io 
address
to the guest physical io address. Then guest can map the phsical io address to 
its
own virtual address, then by accessing virtual address, guest can access 
machine io address space.
So the platform device passthrough does not needs frontend/backend driver to 
support, except smmu is handled by xen.

But the platform device needs clk to drive the hardware IP, also may needs 
pinmux settings.
the driver in guest needs to drive the hardware IP passed through to the guest, 
so it needs to operate on the clk.

Just pasted comments from George for V1:

"
Just speaking from the perspective of a Xen dev who's not an ARM dev:
a few more words on the relationship between pvclk and
device-passthrough would be helpful to set the context.  It sounds
like:

* On ARM, passing through a device requires a clocksource (at least
for many devices)

* dom0 has the hardware clocksource, but at the moment domUs don't
have a suitable clocksource

* This patch implements pvclk front/backend suitable for such devices

Is that right?  In which case something like the following would be helpful:

"This patch introduces pvclk, a paravirtualized clock source suitable
for devices to use when passing through to domUs on ARM systems."
"

Since my use case is for ARM embedded products, X86 may not need this. I try to 
make this interface common,
but 

[Xen-devel] [RFC V2] xen: interface: introduce pvclk interface

[Peng Fan]

Introduce pvclk interface which is useful when doing device passthrough
on ARM platform.

To ARM platform, saying embedded SoCs, clock management hardware IP
is controlled by Dom0, DomU does not have clocksource. So we need a
paravirtualized clock source to let DomU can handle device that are
passed through from Dom0.

Signed-off-by: Peng Fan <van.free...@gmail.com>
Cc: George Dunlap <george.dun...@citrix.com>
Cc: Konrad Rzeszutek Wilk <konrad.w...@oracle.com>
Cc: David Vrabel <david.vra...@citrix.com>
Cc: Julien Grall <julien.gr...@citrix.com>
Cc: Stefano Stabellini <stefano.stabell...@eu.citrix.com>
Cc: Ian Campbell <ian.campb...@citrix.com>
---

V2:
The V1 thread:
http://lists.xen.org/archives/html/xen-devel/2016-01/msg01860.html
The V1 thread binds the interface part to the implementation for linux kernel.
This V2 only contains the pvclk interface part.

In this patch:
Backend,
/local/domain//backend/vclk//nr-clks
/local/domain//backend/vclk///name
Y is frontend domid, Z is clk id, name is the clk name.

Frontend,
/local/domain//device/vclk/clk-ring-ref
/local/domain//device/vclk/event-channel
/local/domain//device/vclk//name
Y is the clk id.

My original idea is to pass clk name which is parsed from dts.
And not expose clk name to xenstore. DomU pass the name to Dom0,
Dom0 use the name to find correct clk and operate on the clk.

Not sure whether I am right or not, please advise.
I have no knowledge of ACPI, just following Ian's
comment on V1, I add the clk id and expose the info to xenstore,
but discard the devid/bus. If devid/bus is needed, I think we
can encoded it into the clk id part in future.
Exposing the info to xenstore means we need to add entry in xl
configuration file like this: vclks =  ["id:clkname", "id:clkname"] -->
vclks = ["1:uart2-root-clk", "3:gpu-root-clk"]. And the libxl pvclk
should parse the vclks entry and fill the contents to xenstore backend nodes.
The frontend xenstore node will be created, by driver probe function 
when DomU booting or by libxl pvclk before DomU boots.
To my use case, Dom0 and DomU both use device tree, I need to build
the mapping table between id and name, since I use name to lookup
the clk in backend, like this:
"clk = __clk_loopkup(clkname); clk_prepare_enable(clk)". Maybe ACPI
is another different case.

 xen/include/public/io/clkif.h | 129 ++
 1 file changed, 129 insertions(+)
 create mode 100644 xen/include/public/io/clkif.h

diff --git a/xen/include/public/io/clkif.h b/xen/include/public/io/clkif.h
new file mode 100644
index 000..f6f9f20
--- /dev/null
+++ b/xen/include/public/io/clkif.h
@@ -0,0 +1,129 @@
+/*
+ * clkif.h
+ *
+ * CLK I/O interface for Xen guest OSes.
+ *
+ * Copyright (C) 2016, Peng Fan <van.free...@gmail.com>
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+ * DEALINGS IN THE SOFTWARE.
+ */
+
+#ifndef __XEN_PUBLIC_IO_CLKIF_H__
+#define __XEN_PUBLIC_IO_CLKIF_H__
+
+#include "ring.h"
+#include "../grant_table.h"
+
+/*
+ * The two halves of an Xen pvclk driver utilize nodes within the
+ * XenStore to communicate and negotiate operating parameters.
+ *
+ * Backend:
+ * /local/domain//backend/vclk//nr-clks
+ * /local/domain//backend/vclk///name
+ *
+ * X - The backend domid
+ * Y - The frontend domid
+ * Z - The clk id
+ * name - The clk name
+ *
+ * Backend example:
+ * /local/domain/0/backend/vclk/1/nr-clks   --> value 2
+ * /local/domain/0/backend/vclk/1/0/name --> uart2-root-clk
+ * /local/domain/0/backend/vclk/1/1/name --> gpu-root-clk
+ *
+ * /local/domain/0/backend/vclk/2/nr-clks   --> value 1
+ * /local/domain/0/backend/vclk/2/0/name --> lcdif-per-clk
+ *
+ * Frontend:
+ * /local/domain//device/vclk/clk-ring-ref
+ * /local/domain//device/vclk/event-channel
+ * /local/domain//device/vclk//name
+ *
+ * Frontend example:
+ * /local/domain

Re: [Xen-devel] [RFC V2] xen: interface: introduce pvclk interface

[Peng Fan]

Hi Juergen,

On Wed, Jan 20, 2016 at 11:40:55AM +0100, Juergen Gross wrote:
>On 20/01/16 10:25, Peng Fan wrote:
>> Hi Juergen,
>> 
>> On Wed, Jan 20, 2016 at 10:05:15AM +0100, Juergen Gross wrote:
>>> On 20/01/16 09:31, Peng Fan wrote:
>>>> Introduce pvclk interface which is useful when doing device passthrough
>>>> on ARM platform.
>>>
>>> ...
>>>
>>>> +/*
>>>> + * Frontend request
>>>> + *
>>>> + * cmd: command for operation on clk, should be XEN_CLK_[xx],
>>>> + *excluding XEN_CLK_END. id is the
>>>> + * id: clk id
>>>> + * rate: clock rate. Used for set rate.
>>>
>>> Which unit? Hz?
>> 
>> Yeah. Hz. I'll add comments in V3.
>> 
>>>
>>>> + */
>>>> +struct xen_clkif_request {
>>>> +  uint32_t cmd;
>>>> +  uint32_t id;
>>>> +  uint64_t rate;
>>>> +};
>>>> +typedef struct xen_clkif_request xen_clkif_request_t;
>>>> +
>>>> +/*
>>>> + * Backend response
>>>> + *
>>>> + * cmd: command for operation on clk, same with the cmd in request.
>>>> + * id: clk id, same with the id in request.
>>>> + * success: indicate failure or success for the cmd.
>>>
>>> Values?
>> 
>> I'd like to let the frontend/backend driver to determine the value.
>> In my implementation for linux, if the backend API supports return value,
>> I'll fill the return value to success entry. If the backend API returns
>> void, I'll just fill 0 to success entry.
>
>So please specify "0" to mean success and add some sensible error
>values. There might be multiple frontend- and/or backend-variants which
>all must agree using the same interface.

Will change this to `int status`, as also observed by Jan.
I'll also define macros such as "#define XEN_CLK_ENABLE_OK 0", "#define 
XEN_CLK_ENABLE_FAILURE -1"

>
>>>> + * rate: clock rate. Used for get rate.
>>>> + *
>>>> + * cmd and id are filled by backend and passed to frontend to
>>>> + * let frontend check whether the response is for the current
>>>> + * request or not.
>>>
>>> I'd rather let the frontend add a request Id to the request which
>>> will be echoed here instead cmd and clock Id.
>> 
>> If using request id, I think we can encode cmd and id into request id?
>
>This should just be an opaque value for the backend. The frontend is
>free how to create this value, it should be unique for every outstanding
>request of a domU, however. It could be built from cmd and id in case
>there can't be multiple requests with the same cmd/id combination
>active in a domU.

Thanks for teaching me on this. So the request id should be globally unique
for backend. So "random value(generated in frontend) + frontend domid" is ok 
for this?

Thanks,
Peng.
>
>
>juergen

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC V2] xen: interface: introduce pvclk interface

[Peng Fan]

Hi Jan,

On Wed, Jan 20, 2016 at 03:14:20AM -0700, Jan Beulich wrote:
 On 20.01.16 at 09:31,  wrote:
>> +/*
>> + * Backend response
>> + *
>> + * cmd: command for operation on clk, same with the cmd in request.
>> + * id: clk id, same with the id in request.
>> + * success: indicate failure or success for the cmd.
>> + * rate: clock rate. Used for get rate.
>> + *
>> + * cmd and id are filled by backend and passed to frontend to
>> + * let frontend check whether the response is for the current
>> + * request or not.
>> + */
>> +struct xen_clkif_response {
>> +uint32_t cmd;
>> +uint32_t id;
>> +uint32_t success;
>> +uint64_t rate;
>> +};
>
>This isn't 32-/64-bit clean. Question is whether echoing cmd is really
>needed.

As Juergen suggested, use a request id. I'll fix this in V3.
32-/64-bit unclean, I can not get you here (:

>
>Also naming a field "success" is pretty odd - is this mean to be a
>boolean? Better name it e.g. status, allowing for different (error)
>indicators.

As you suggested, how about `int status`? And in this clkif.h,
define different status value, such as `#define XEN_CLK_PREPARE_OK 0,
#define XEN_CLK_PREPARE_FAILURE -1`. The frontend and backend should
be aware of the status value.

>
>And what I'm missing throughout the file is some description of how
>clock events (interrupts?) are actually meant to make it into the
>guest.

I have a simple implementation v1 patch for linux, 
http://lists.xen.org/archives/html/xen-devel/2016-01/msg01860.html.
You can see how it works.

Thanks,
Peng.

>
>Jan
>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [RFC V2] xen: interface: introduce pvclk interface

[Peng Fan]

Hi Juergen,

On Wed, Jan 20, 2016 at 10:05:15AM +0100, Juergen Gross wrote:
>On 20/01/16 09:31, Peng Fan wrote:
>> Introduce pvclk interface which is useful when doing device passthrough
>> on ARM platform.
>
>...
>
>> +/*
>> + * Frontend request
>> + *
>> + * cmd: command for operation on clk, should be XEN_CLK_[xx],
>> + *  excluding XEN_CLK_END. id is the
>> + * id: clk id
>> + * rate: clock rate. Used for set rate.
>
>Which unit? Hz?

Yeah. Hz. I'll add comments in V3.

>
>> + */
>> +struct xen_clkif_request {
>> +uint32_t cmd;
>> +uint32_t id;
>> +uint64_t rate;
>> +};
>> +typedef struct xen_clkif_request xen_clkif_request_t;
>> +
>> +/*
>> + * Backend response
>> + *
>> + * cmd: command for operation on clk, same with the cmd in request.
>> + * id: clk id, same with the id in request.
>> + * success: indicate failure or success for the cmd.
>
>Values?

I'd like to let the frontend/backend driver to determine the value.
In my implementation for linux, if the backend API supports return value,
I'll fill the return value to success entry. If the backend API returns
void, I'll just fill 0 to success entry.

>
>> + * rate: clock rate. Used for get rate.
>> + *
>> + * cmd and id are filled by backend and passed to frontend to
>> + * let frontend check whether the response is for the current
>> + * request or not.
>
>I'd rather let the frontend add a request Id to the request which
>will be echoed here instead cmd and clock Id.

If using request id, I think we can encode cmd and id into request id?
To my dts case, clk id is simple. But I am not sure about ACPI part. Maybe
I can not simply encode them into request id.
Wait for more comments on this:)

Thanks,
Peng.
>
>
>Juergen

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel