[XenPPC] Re: xen heap size

2007-02-23 Thread Ryan Harper 
* Hollis Blanchard <[EMAIL PROTECTED]> [2007-02-22 16:30]:
> On Thu, 2007-02-22 at 16:07 -0600, Ryan Harper wrote:
> 
> > IIRC, when dom0 boots with 192MB (the default) I usually see ~19MB of
> > heap available in the boot messages on js20.  Looking at js21, I see:
> > 
> > (XEN) Xen Heap: 135MiB (138548KiB)
> > 
> > RMA different size on js21? 
> 
> That's an unusual size: it's slightly more than the second 64MB RMA
> boundary, which seems to indicate there's a lot of wasted memory before
> dom0 at 192MB. I wonder if this is related to the 4GB of memory in this
> system. A more complete boot log might shed some light on it.

Attached.

> 
> To answer your question, the 970MP (in JS21) supports the same RMA sizes
> as 970 and 970FX (in JS20).

OK.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
[EMAIL PROTECTED]
0 > boot net1 xen || root=/dev/sda3
Trying to load: xen || root=/dev/sda3 from: /ht/[EMAIL PROTECTED]/[EMAIL 
PROTECTED],1 ...

 Bootloader 1.5
  Reading MAC address from device: 00:14:5E:9C:1C:C5
  Requesting IP address via BOOTP: 9.3.189.7
  Requesting file "leaf4" via TFTP
  Receiving data: ##A##-
  TFTP: Received leaf4 (9428 KBytes)
  Successfully loaded
---
OF: Xen/PPC version 3.0-unstable ([EMAIL PROTECTED]) (gcc version 4.1.0 (SUSE 
Linux)) Mon Feb 19 17:24:41 CST 2007
boot_of_init args: 0x0 0x0 0xe11027c 0xe291b0f 0x15
boot msr: 0x10003000
boot_of_init: _start 0040 _end 00c45c90 0xe291b0f
bootargs = xen || root=/dev/sda3
boot_of_module: Dom0 is linked in: 0x479b4c[size 0x758860]
mod0: 177 E L F
boot_of_module: dom0 mod @ 0x00479b4c[0xbd23ac]
boot_of_module: dom0 mod string: root=/dev/sda3
instantiating RTAS at: 0x4000
creating oftree at: 0xc000
pkg_save: saved device tree in 0x57b8 bytes
boot_of_devtree: devtree mod @ 0xc000 - 0x0003c000
OF: timebase-frequency = 14318378 Hz
OF: clock-frequency = 230 KHz
spinning up secondary processor #1: ping = 0x: pong = 0x1
spinning up secondary processor #2: ping = 0x: pong = 0x2
spinning up secondary processor #3: ping = 0x: pong = 0x3
pruning `/ht/[EMAIL PROTECTED]/[EMAIL PROTECTED]' from devtree
pruning `/ht/[EMAIL PROTECTED]/[EMAIL PROTECTED]' from devtree
boot_of_serial: ISA base: 0xf400
boot_of_serial: ISRC=0x44, but forcing poll mode
 __  ___  ___ __ _
 \ \/ /___ _ __   |___ / / _ \_   _ _ __  ___| |_ __ _| |__ | | ___
  \  // _ \ '_ \|_ \| | | |__| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
  /  \  __/ | | |  ___) | |_| |__| |_| | | | \__ \ || (_| | |_) | |  __/
 /_/\_\___|_| |_| |(_)___/\__,_|_| |_|___/\__\__,_|_.__/|_|\___|

 http://www.cl.cam.ac.uk/netos/xen
 University of Cambridge Computer Laboratory

 Xen version 3.0-unstable ([EMAIL PROTECTED]) (gcc version 4.1.0 (SUSE Linux)) 
Mon Feb 19 17:24:41 CST 2007
 Latest ChangeSet: Mon Feb 19 17:16:56 2007 -0600 13949:1c63549d7578

(XEN) Physical RAM map:
(XEN)   : 8000
(XEN)   0001: 8000
(XEN) End of Xen Area: 144MiB (147456KiB)
(XEN) End of RAM: 6144MiB (6291456KiB)
(XEN) boot allocator @ 3c000 - 6d000
(XEN) boot free: 0900 - 8000
(XEN) boot free: 0001 - 00018000
(XEN) total_pages: 0x000f7000
(XEN) NUMA turned off
(XEN) Faking a node at -00018000
(XEN) Domain heap initialised: DMA width 64 bits
(XEN) xenheap: 0006d000 - 0040
(XEN) xenheap: 00c46000 - 0900
(XEN) Xen Heap: 135MiB (138548KiB)
(XEN) Dom Heap: 3880MiB (3973120KiB)
(XEN) CPU[PIR:0 IPI:0 Logical:0] Hello World!
(XEN) xen_mpic_init: start
(XEN) mpic: Setting up MPIC "Xen-U3-MPIC" version 1.2 at f804, max 4 CPUs
(XEN) mpic: ISU size: 124, shift: 7, mask: 7f
(XEN) mpic: Initializing for 124 sources
(XEN) mpic: Setting up HT PICs workarounds for U3/U4
(XEN) mpic:   - HT:07.0 [0xf0] vendor 1022 device 7460 has 24 irqs
(XEN) xen_mpic_init: success
(XEN) requesting IPIs ...
(XEN) IPIs requested...
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Initializing DART 0xf8033000: tbl: 0020[0x200] entries: 
0x8
(XEN) Initializing DART Model U4: ctl: 0x8000 base: 0x200 size: 0x200
(XEN) spinning up at most 16 total processors ...
(XEN) Synchronizing timebase
(XEN) CPU[PIR:1 IPI:1 Logical:1] Hello World!
(XEN) Got ack 
(XEN) score 299, offset 1000
(XEN) score 299, offset 500
(XEN) score 299, offset 250
(XEN) score 299, offset 125
(XEN) score 299, offset 62
(XEN) score 299, offset 31
(XEN) score 299, offset 15
(XEN) score 111, offset 7
(XEN) score -299, offset 3
(XEN) score -299, offset 5
(XEN) score -299, offset 6
(XEN) Min 6 (score -299), Max 7 (score 119)
(XEN) Final offset: 7 (215/300)
(XEN) Synchronizing timebase
(XEN) CPU[PIR:2 IPI:2 Logical:2] Hello World!
(XEN) Got ack 
(XEN)

Re: [XenPPC] Re: xen heap size

2007-02-23 Thread Ryan Harper 
* Jimi Xenidis <[EMAIL PROTECTED]> [2007-02-22 19:30]:
> We don't consider the RMA boundary for the Xen heap at all anymore  
> (not for a while)
> The Xen heap is calculated based on the estimated resources we'll need.
> on example is that we need to get enough HTABs for all the domain, so  
> 1/64th of all of memory is part of the Xen heap size.

Hrm.  One of the items I need to address is determining how much of
dom0's memory allocation runs into the 2-4G IO hole.  One method I was
hoping might work is:


 /* overlap in pages into 2G-4G IO range (if any) */
dom0_overlap = (cpu_default_rma_order_pages() + dom0_nrpages) -
   IO_SIZE_PAGES;

It doesn't look like we can make the assumption that Xen+xenheap will
only occupy the first RMA of the platform.  

The other method I was going to look into was to allocate dom0's rma,
and then calculation would look like:

   dom0_start_mfn = page_to_mfn(d->arch.rma_base);
   dom0_overlap = (dom0_start_mfn + dom0_nrpages - rma_sz) - IO_SIZE_PAGES;
   
Any other good way to figure how much of dom0's allocation will fall
within 2-4G IO hole?

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
[EMAIL PROTECTED]

___
Xen-ppc-devel mailing list
Xen-ppc-devel@lists.xensource.com
http://lists.xensource.com/xen-ppc-devel

Re: [XenPPC] Re: xen heap size

2007-02-23 Thread Hollis Blanchard 
On Fri, 2007-02-23 at 10:21 -0600, Ryan Harper wrote:
> 
> The other method I was going to look into was to allocate dom0's rma,
> and then calculation would look like:
> 
>dom0_start_mfn = page_to_mfn(d->arch.rma_base);
>dom0_overlap = (dom0_start_mfn + dom0_nrpages - rma_sz) -
> IO_SIZE_PAGES;
>
> Any other good way to figure how much of dom0's allocation will fall
> within 2-4G IO hole?

Why are you looking for another approach? What's wrong with this
solution?

-- 
Hollis Blanchard
IBM Linux Technology Center


___
Xen-ppc-devel mailing list
Xen-ppc-devel@lists.xensource.com
http://lists.xensource.com/xen-ppc-devel

Re: [XenPPC] Re: xen heap size

2007-02-23 Thread Ryan Harper 
* Hollis Blanchard <[EMAIL PROTECTED]> [2007-02-23 10:58]:
> On Fri, 2007-02-23 at 10:21 -0600, Ryan Harper wrote:
> > 
> > The other method I was going to look into was to allocate dom0's rma,
> > and then calculation would look like:
> > 
> >dom0_start_mfn = page_to_mfn(d->arch.rma_base);
> >dom0_overlap = (dom0_start_mfn + dom0_nrpages - rma_sz) -
> > IO_SIZE_PAGES;
> >
> > Any other good way to figure how much of dom0's allocation will fall
> > within 2-4G IO hole?
> 
> Why are you looking for another approach? What's wrong with this
> solution?

I was hoping I'd find this:

xen/arch/powerpc/memory.c:145

xenheap_phys_end = xh_pages << PAGE_SHIFT;
printk("End of Xen Area: %luMiB (%luKiB)\n",
   xenheap_phys_end >> 20, xenheap_phys_end >> 10);


which marks the end of the xen area; just what I was looking for.

Now I can do:

   /* overlap in pages into 2G-4G IO range (if any) */
   dom0_overlap = ((xenheap_phys_end >> PAGE_SHIFT) + dom0_nrpages) -
  IO_SIZE_PAGES;


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
[EMAIL PROTECTED]

___
Xen-ppc-devel mailing list
Xen-ppc-devel@lists.xensource.com
http://lists.xensource.com/xen-ppc-devel

Re: [XenPPC] Re: xen heap size

2007-02-23 Thread Hollis Blanchard 
On Fri, 2007-02-23 at 11:12 -0600, Ryan Harper wrote:
> * Hollis Blanchard <[EMAIL PROTECTED]> [2007-02-23 10:58]:
> > On Fri, 2007-02-23 at 10:21 -0600, Ryan Harper wrote:
> > > 
> > > The other method I was going to look into was to allocate dom0's rma,
> > > and then calculation would look like:
> > > 
> > >dom0_start_mfn = page_to_mfn(d->arch.rma_base);
> > >dom0_overlap = (dom0_start_mfn + dom0_nrpages - rma_sz) -
> > > IO_SIZE_PAGES;
> > >
> > > Any other good way to figure how much of dom0's allocation will fall
> > > within 2-4G IO hole?
> > 
> > Why are you looking for another approach? What's wrong with this
> > solution?
> 
> I was hoping I'd find this:
> 
> xen/arch/powerpc/memory.c:145
> 
> xenheap_phys_end = xh_pages << PAGE_SHIFT;
> printk("End of Xen Area: %luMiB (%luKiB)\n",
>xenheap_phys_end >> 20, xenheap_phys_end >> 10);
> 
> 
> which marks the end of the xen area; just what I was looking for.
> 
> Now I can do:
> 
>/* overlap in pages into 2G-4G IO range (if any) */
>dom0_overlap = ((xenheap_phys_end >> PAGE_SHIFT) + dom0_nrpages) -
>   IO_SIZE_PAGES;

You're assuming that the Xen heap ends exactly where dom0 will begin
(i.e. where dom0's RMA is allocated), which is not the case. From your
boot log, we see this:
(XEN) End of Xen Area: 144MiB (147456KiB)
...
(XEN) xenheap: 0006d000 - 0040
(XEN) xenheap: 00c46000 - 0900
(XEN) Xen Heap: 135MiB (138548KiB)

xenheap_phys_end is 144MB (0x900), and when you subtract the space
from the middle (e.g. occupied by Xen itself), you end up with a total
of 135MB.

In this case, dom0's RMA will be allocated at 192MB (0xc00), and the
space from 144MB (0x900) to 192MB will be assigned to the domain
heap.

-- 
Hollis Blanchard
IBM Linux Technology Center


___
Xen-ppc-devel mailing list
Xen-ppc-devel@lists.xensource.com
http://lists.xensource.com/xen-ppc-devel

Re: [XenPPC] Re: xen heap size

2007-02-23 Thread Ryan Harper 
* Hollis Blanchard <[EMAIL PROTECTED]> [2007-02-23 11:43]:
> On Fri, 2007-02-23 at 11:12 -0600, Ryan Harper wrote:
> > * Hollis Blanchard <[EMAIL PROTECTED]> [2007-02-23 10:58]:
> > > On Fri, 2007-02-23 at 10:21 -0600, Ryan Harper wrote:
> > > > 
> > > > The other method I was going to look into was to allocate dom0's rma,
> > > > and then calculation would look like:
> > > > 
> > > >dom0_start_mfn = page_to_mfn(d->arch.rma_base);
> > > >dom0_overlap = (dom0_start_mfn + dom0_nrpages - rma_sz) -
> > > > IO_SIZE_PAGES;
> > > >
> > > > Any other good way to figure how much of dom0's allocation will fall
> > > > within 2-4G IO hole?
> > > 
> > > Why are you looking for another approach? What's wrong with this
> > > solution?
> > 
> > I was hoping I'd find this:
> > 
> > xen/arch/powerpc/memory.c:145
> > 
> > xenheap_phys_end = xh_pages << PAGE_SHIFT;
> > printk("End of Xen Area: %luMiB (%luKiB)\n",
> >xenheap_phys_end >> 20, xenheap_phys_end >> 10);
> > 
> > 
> > which marks the end of the xen area; just what I was looking for.
> > 
> > Now I can do:
> > 
> >/* overlap in pages into 2G-4G IO range (if any) */
> >dom0_overlap = ((xenheap_phys_end >> PAGE_SHIFT) + dom0_nrpages) -
> >   IO_SIZE_PAGES;
> 
> You're assuming that the Xen heap ends exactly where dom0 will begin
> (i.e. where dom0's RMA is allocated), which is not the case. From your
> boot log, we see this:
> (XEN) End of Xen Area: 144MiB (147456KiB)
> ...
> (XEN) xenheap: 0006d000 - 0040
> (XEN) xenheap: 00c46000 - 0900
> (XEN) Xen Heap: 135MiB (138548KiB)
> 
> xenheap_phys_end is 144MB (0x900), and when you subtract the space
> from the middle (e.g. occupied by Xen itself), you end up with a total
> of 135MB.
> 
> In this case, dom0's RMA will be allocated at 192MB (0xc00), and the
> space from 144MB (0x900) to 192MB will be assigned to the domain
> heap.

Ah, I see, RMA alignment issues.  192MB is the next 64MB aligned
chunk past xenheap_phys_end.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
[EMAIL PROTECTED]

___
Xen-ppc-devel mailing list
Xen-ppc-devel@lists.xensource.com
http://lists.xensource.com/xen-ppc-devel

Re: [XenPPC] Re: xen heap size

2007-02-23 Thread Jimi Xenidis 
hmm.. are we always going to include the IO space in dom0->p2m[] or  
only when dom0->maxpage > IO_Base?

-JX
On Feb 23, 2007, at 1:05 PM, Ryan Harper wrote:


* Hollis Blanchard <[EMAIL PROTECTED]> [2007-02-23 11:43]:

On Fri, 2007-02-23 at 11:12 -0600, Ryan Harper wrote:

* Hollis Blanchard <[EMAIL PROTECTED]> [2007-02-23 10:58]:

On Fri, 2007-02-23 at 10:21 -0600, Ryan Harper wrote:


The other method I was going to look into was to allocate  
dom0's rma,

and then calculation would look like:

   dom0_start_mfn = page_to_mfn(d->arch.rma_base);
   dom0_overlap = (dom0_start_mfn + dom0_nrpages - rma_sz) -
IO_SIZE_PAGES;

Any other good way to figure how much of dom0's allocation will  
fall

within 2-4G IO hole?


Why are you looking for another approach? What's wrong with this
solution?


I was hoping I'd find this:

xen/arch/powerpc/memory.c:145

xenheap_phys_end = xh_pages << PAGE_SHIFT;
printk("End of Xen Area: %luMiB (%luKiB)\n",
   xenheap_phys_end >> 20, xenheap_phys_end >> 10);


which marks the end of the xen area; just what I was looking for.

Now I can do:

   /* overlap in pages into 2G-4G IO range (if any) */
   dom0_overlap = ((xenheap_phys_end >> PAGE_SHIFT) +  
dom0_nrpages) -

  IO_SIZE_PAGES;


You're assuming that the Xen heap ends exactly where dom0 will begin
(i.e. where dom0's RMA is allocated), which is not the case. From  
your

boot log, we see this:
(XEN) End of Xen Area: 144MiB (147456KiB)
...
(XEN) xenheap: 0006d000 - 0040
(XEN) xenheap: 00c46000 - 0900
(XEN) Xen Heap: 135MiB (138548KiB)

xenheap_phys_end is 144MB (0x900), and when you subtract the  
space
from the middle (e.g. occupied by Xen itself), you end up with a  
total

of 135MB.

In this case, dom0's RMA will be allocated at 192MB (0xc00),  
and the

space from 144MB (0x900) to 192MB will be assigned to the domain
heap.


Ah, I see, RMA alignment issues.  192MB is the next 64MB aligned
chunk past xenheap_phys_end.


--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
[EMAIL PROTECTED]



___
Xen-ppc-devel mailing list
Xen-ppc-devel@lists.xensource.com
http://lists.xensource.com/xen-ppc-devel

Re: [XenPPC] [PATCH 1 of 6] [PATCH] xen: add arch hook for max_mem hcall

2007-02-23 Thread Jimi Xenidis 


On Feb 21, 2007, at 6:16 PM, Ryan Harper wrote:


2 files changed, 6 insertions(+)
xen/common/domctl.c  |4 
xen/include/xen/shadow.h |2 ++


NIT: I think  mm.h or domain.h is a better home for the macro/proto.
-JX


___
Xen-ppc-devel mailing list
Xen-ppc-devel@lists.xensource.com
http://lists.xensource.com/xen-ppc-devel

Re: [XenPPC] [PATCH 2 of 6] [PATCH] xen: move dom0 memory allocation into construct_dom0()

2007-02-23 Thread Jimi Xenidis 


On Feb 21, 2007, at 6:17 PM, Ryan Harper wrote:


2 files changed, 17 insertions(+), 14 deletions(-)
xen/arch/powerpc/domain_build.c |   24 
xen/arch/powerpc/setup.c|7 +--


# HG changeset patch
# User Ryan Harper <[EMAIL PROTECTED]>
# Date 1172103252 21600
# Node ID 84ec1b4d5cd50cc9d49202eb978a4715c4780e28
# Parent  17815286856eb2b67a64e64f2a0a53a7c5d505e2
[PATCH] xen: move dom0 memory allocation into construct_dom0()
Signed-off-by: Ryan Harper <[EMAIL PROTECTED]>

diff -r 17815286856e -r 84ec1b4d5cd5 xen/arch/powerpc/domain_build.c
--- a/xen/arch/powerpc/domain_build.c   Wed Feb 21 18:14:12 2007 -0600
+++ b/xen/arch/powerpc/domain_build.c   Wed Feb 21 18:14:12 2007 -0600
@@ -112,9 +112,9 @@ int construct_dom0(struct domain *d,
 struct domain_setup_info dsi;
 ulong dst;
 u64 *ofh_tree;
-uint rma_nrpages = 1 << d->arch.rma_order;
-ulong rma_sz = rma_size(d->arch.rma_order);
-ulong rma = page_to_maddr(d->arch.rma_page);
+uint rma_nrpages;
+ulong rma_sz;
+ulong rma;
 start_info_t *si;
 ulong eomem;
 int am64 = 1;
@@ -131,8 +131,6 @@ int construct_dom0(struct domain *d,
 if (image_len == 0)
 panic("No Dom0 image supplied\n");

-cpu_init_vcpu(v);
-
 memset(&dsi, 0, sizeof(struct domain_setup_info));
 dsi.image_addr = image_start;
 dsi.image_len  = image_len;
@@ -154,9 +152,6 @@ int construct_dom0(struct domain *d,

 printk("*** LOADING DOMAIN 0 ***\n");

-/* By default DOM0 is allocated all available memory. */
-d->max_pages = ~0U;
-
 /* default is the max(1/16th of memory, CONFIG_MIN_DOM0_PAGES) */
 if (dom0_nrpages == 0) {
 dom0_nrpages = total_pages >> 4;
@@ -164,6 +159,19 @@ int construct_dom0(struct domain *d,
 if (dom0_nrpages < CONFIG_MIN_DOM0_PAGES)
 dom0_nrpages = CONFIG_MIN_DOM0_PAGES;
 }
+
+/* By default DOM0 is allocated all available memory. */


This comment is not longer correct.


+d->max_pages = dom0_nrpages;


dom0_nrpages has yet to go through all the logic that defines its  
final value, particularly makeing sure it is bigger than the RMA  
specificied, below.



+
+if (0 > allocate_rma(d, cpu_default_rma_order_pages()))
+panic("Error allocating domain 0 RMA\n");
+
+/* init vcpu now that RMA has been allocated */
+cpu_init_vcpu(v);


We can make this part of the vcpu alloc loop that occurs later in  
this function and remove the alloc in setup.c.
NOTE: Linux creates its own stack so there is we do not need the  
following:

/* put stack below everything */
v->arch.ctxt.gprs[1] = dst - STACK_FRAME_OVERHEAD;


+
+rma_nrpages = 1 << d->arch.rma_order;
+rma_sz = rma_size(d->arch.rma_order);
+rma = page_to_maddr(d->arch.rma_page);

 /* make sure we are at least as big as the RMA */
 if (dom0_nrpages > rma_nrpages)
diff -r 17815286856e -r 84ec1b4d5cd5 xen/arch/powerpc/setup.c
--- a/xen/arch/powerpc/setup.c  Wed Feb 21 18:14:12 2007 -0600
+++ b/xen/arch/powerpc/setup.c  Wed Feb 21 18:14:12 2007 -0600
@@ -369,13 +369,8 @@ static void __init __start_xen(multiboot

 /* Create initial domain 0. */
 dom0 = domain_create(0, 0);
-if (dom0 == NULL)
+if ( (dom0 == NULL) || (alloc_vcpu(dom0, 0, 0) == NULL) )
 panic("Error creating domain 0\n");

See the comment above.
BTW: I know that the Xen style is "if ( EXPR )" but Hollis (and I  
agree) insists on "if (EXPR)" in all PPC code.  Just our way of  
sticking it to "the man" :)



-dom0->max_pages = ~0U;
-if (0 > allocate_rma(dom0, cpu_default_rma_order_pages()))
-panic("Error allocating domain 0 RMA\n");
-if (NULL == alloc_vcpu(dom0, 0, 0))
-panic("Error creating domain 0 vcpu 0\n");

 /* The Interrupt Controller will route everything to CPU 0 so we
  * need to make sure Dom0's vVCPU 0 is pinned to the CPU */

___
Xen-ppc-devel mailing list
Xen-ppc-devel@lists.xensource.com
http://lists.xensource.com/xen-ppc-devel



___
Xen-ppc-devel mailing list
Xen-ppc-devel@lists.xensource.com
http://lists.xensource.com/xen-ppc-devel