Re: [Xen-ia64-devel] [Patch] Pass the bare LSAPIC ID to dom0
Hi, Alex and all Alex, thank you for your help. As you said, I make a patch to disable SRAT, SLIT. This patch is short term solution. This patch modify header-signature from SRAT, SLIT to . Because dom0 don't recognized this signature, we can disable SRAT, SLIT. Then dom0 detect non-NUMA box. I tested booting dom0 on our PRIMEQUEST and Tiger4. Signed-off-by: Akio Takebe [EMAIL PROTECTED] Best Regards, Akio Takebe dom0_disable_srat_slit.patch Description: Binary data ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-ia64-devel] [Patch] Pass the bare LSAPIC ID to dom0
Hi, all This patch is RFC. If I get agreement for this patch, I cleanup it, then I'll post xen-devel if necessary. Best Regards, Akio Takebe Hi, Alex and all Alex, thank you for your help. As you said, I make a patch to disable SRAT, SLIT. This patch is short term solution. This patch modify header-signature from SRAT, SLIT to . Because dom0 don't recognized this signature, we can disable SRAT, SLIT. Then dom0 detect non-NUMA box. I tested booting dom0 on our PRIMEQUEST and Tiger4. Signed-off-by: Akio Takebe [EMAIL PROTECTED] Best Regards, Akio Takebe ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-ia64-devel] [Patch] Pass the bare LSAPIC ID to dom0
On Wed, Jul 25, 2007 at 03:02:18PM +0900, Akio Takebe wrote: Content-Description: Mail message body This patch modify header-signature from SRAT, SLIT to . Because dom0 don't recognized this signature, we can disable SRAT, SLIT. Then dom0 detect non-NUMA box. How about OEMx instead of and overwriting OEM ID to something similar to Xen/IA64? -- yamahata ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-ia64-devel] [Patch] Pass the bare LSAPIC ID to dom0
Hi, Isaku
On Wed, Jul 25, 2007 at 03:02:18PM +0900, Akio Takebe wrote:
Content-Description: Mail message body
This patch modify header-signature from SRAT, SLIT to .
Because dom0 don't recognized this signature,
we can disable SRAT, SLIT.
Then dom0 detect non-NUMA box.
How about OEMx instead of and overwriting OEM ID to something
similar to Xen/IA64?
Thank you for your comments.
It's good idea.
Because acpi_table_print() show 6byte oem_id,
how about XenIPF?
78 void acpi_table_print(struct acpi_table_header *header, unsigned long
phys_addr)
79 {
[snip...]
98 printk(KERN_DEBUG PREFIX
99%.4s (v%3.3d %6.6s %8.8s 0x%08x %.4s 0x%08x) @ 0x%p\n,
name,
100header-revision, header-oem_id, header-oem_table_id,
101header-oem_revision, header-asl_compiler_id,
102header-asl_compiler_revision, (void *)phys_addr);
103 }
For example, how about the following sample?
ACPI: RSDP (v002 FUJITS) @ 0x7f924000
ACPI: XSDT (v001 FUJITS PQ 0x0003 FJ 0x0001) @
0x7f92ed68
ACPI: FADT (v003 FUJITS PQ 0x0001 FJ 0x0001) @
0x7f92e6c0
ACPI: MADT (v001 FUJITS PQ 0x0001 FJ 0x0001) @
0x7f92e7b8
ACPI: SPCR (v001 FUJITS PQ 0x0001 FJ 0x0001) @
0x7f92e970
ACPI: OEMx (v001 XenIPF PQ 0x0001 FJ 0x0001) @
0x7f92e9c0 SRAT
ACPI: OEMx (v001 XenIPF PQ 0x0001 FJ 0x0001) @
0x7f92ebb8 SLIT
ACPI: SPMI (v001 FUJITS PQ 0x0001 FJ 0x0001) @
0x7f92ece8
ACPI: WSPT (v001 FUJITS PQ 0x0001 FJ 0x0001) @
0x7f92ed40
ACPI: DSDT (v002 FUJITS PQ 0x0001 MSFT 0x0202) @
0x
Best Regards,
Akio Takebe
___
Xen-ia64-devel mailing list
Xen-ia64-devel@lists.xensource.com
http://lists.xensource.com/xen-ia64-devel
Re: [Xen-ia64-devel] [Patch] Pass the bare LSAPIC ID to dom0
On Wed, Jul 25, 2007 at 04:45:50PM +0900, Akio Takebe wrote: Because acpi_table_print() show 6byte oem_id, how about XenIPF? Sounds good. -- yamahata ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-ia64-devel] [Patch] Pass the bare LSAPIC ID to dom0
Quoting Akio Takebe [EMAIL PROTECTED]: Hi, Alex, Tristan, Yongkang and all On Fri, 2007-07-20 at 03:53 +0200, Tristan Gingold wrote: On Thu, Jul 19, 2007 at 05:20:26PM -0600, Alex Williamson wrote: On Wed, 2007-07-18 at 16:03 +0900, Akio Takebe wrote: Hi, all This patch fix a issue which dom0 cannot boot with dom0_max_vcpus. Currently LSAPIC IDs are create by xen, but ACPI SRAT table is the bare table. So on some boxes node_cpuid[].phys_id are different from cpu_physical_id()s, and we cannot boot dom0. I think xen should pass the bare LSAPIC ID to dom0. Hi Akio, I am not sure this patch is complete. Some code of Xen assumes vcpu_id == id (get_lid, IPI). I think you'd better to patch the SRAT (if it is possible). That may be the way to go, it should be possible. BTW, now that I look at it again, lsapic-id == 0 really looks like an implementation dependent check. Thanks, I'm sorry, and I'll debug more. I'll check vcpu_id and SRAT code. Tristan's idea may be good, I'll try it. Hi, after more thoughts, I think trying NUMA in dom0 is bound to fail or to be useless. Memory mapping is virtualized and vcpu can move... Tristan. ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-ia64-devel] [Patch] Pass the bare LSAPIC ID to dom0
Hi, Alex and Tristan Alex, Thank you for your information, and could you try this patch? I modified SRAT and lsapic of dom0 for disable cpus. This patch work fine on our PRIMEQUEST. after more thoughts, I think trying NUMA in dom0 is bound to fail or to be useless. Memory mapping is virtualized and vcpu can move... Yeah, I think so, we need properly support NUMA. Best Regards, Akio Takebe Quoting Akio Takebe [EMAIL PROTECTED]: Hi, Alex, Tristan, Yongkang and all On Fri, 2007-07-20 at 03:53 +0200, Tristan Gingold wrote: On Thu, Jul 19, 2007 at 05:20:26PM -0600, Alex Williamson wrote: On Wed, 2007-07-18 at 16:03 +0900, Akio Takebe wrote: Hi, all This patch fix a issue which dom0 cannot boot with dom0_max_vcpus. Currently LSAPIC IDs are create by xen, but ACPI SRAT table is the bare table. So on some boxes node_cpuid[].phys_id are different from cpu_physical_id()s, and we cannot boot dom0. I think xen should pass the bare LSAPIC ID to dom0. Hi Akio, I am not sure this patch is complete. Some code of Xen assumes vcpu_id == id (get_lid, IPI). I think you'd better to patch the SRAT (if it is possible). That may be the way to go, it should be possible. BTW, now that I look at it again, lsapic-id == 0 really looks like an implementation dependent check. Thanks, I'm sorry, and I'll debug more. I'll check vcpu_id and SRAT code. Tristan's idea may be good, I'll try it. Hi, after more thoughts, I think trying NUMA in dom0 is bound to fail or to be useless. Memory mapping is virtualized and vcpu can move... Tristan. debug_numa_test.patch Description: Binary data ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-ia64-devel] [Patch] Pass the bare LSAPIC ID to dom0
On Mon, 2007-07-23 at 22:00 +0900, Akio Takebe wrote: Hi, Alex and Tristan Alex, Thank you for your information, and could you try this patch? I modified SRAT and lsapic of dom0 for disable cpus. This patch work fine on our PRIMEQUEST. This patch gets me to the same point as Yongkang, then I take the bugcheck below. I think we at least need vcpu_get_lid() to return the proper value before we can expect this to work. With this patch, MADT/SRAT id/eid != vpu_get_lid(). As Tristan mentioned, this patch is only a small step towards NUMA support. To make the SRAT processor entries meaningful, vcpus need have affinity to run on the node they represent. Does that scheduling exist? Without it, the guest OS may actually make more bad decisions in scheduling that it does now without locality info. To make use of the SRAT memory entries, I think we need to adopt a model similar to what we talked about previously, where each node has a pool of VP memory. Perhaps in the short term, the best way to handle the SRAT is to simply delete it from dom0, or make all the entries disabled. BTW, a minor nit on the patch, it's not an error to not have an SRAT table. It's perfectly legitimate for SMP systems not to provide one. Thanks, Alex kernel BUG at /home/awilliam/xen/20070723/linux-2.6.18-xen.hg/drivers/xen/core/evtchn.c:376! swapper[1]: bugcheck! 0 [1] Modules linked in: Pid: 1, CPU 0, comm: swapper psr : 1010085a6010 ifs : 8692 ip : [a001006d3b90]Not tainted ip is at bind_virq_to_irqhandler+0x110/0x280 unat: pfs : 4692 rsc : 0007 rnat: bsps: pr : 9681 ldrs: ccv : fpsr: 0009804c8a70433f csd : ssd : b0 : a001006d3b90 b6 : a001006da720 b7 : f6 : 0 f7 : 0 f8 : 0 f9 : 0 f10 : 0 f11 : 0 r1 : a001011435f0 r2 : a00100d18000 r3 : a00100f44c30 r8 : 0060 r9 : fff1 r10 : fff04c18 r11 : ee61 r12 : e0007f967d50 r13 : e0007f96 r14 : 0001 r15 : fff1 r16 : r17 : a00100f6c728 r18 : r19 : r20 : 0009804c8a70433f r21 : a0010006d4c0 r22 : r23 : r24 : fff0 r25 : r26 : a00100f6c710 r27 : a00100f562b0 r28 : a00100f6c740 r29 : a00100eb9ab8 r30 : a00100eb9ab8 r31 : a00100f438f0 Call Trace: [a0010001d640] show_stack+0x40/0xa0 sp=e0007f9678e0 bsp=e0007f9612c8 [a0010001e2a0] show_regs+0x840/0x880 sp=e0007f967ab0 bsp=e0007f961270 [a00100042a20] die+0x1c0/0x380 sp=e0007f967ab0 bsp=e0007f961228 [a00100042c30] die_if_kernel+0x50/0x80 sp=e0007f967ad0 bsp=e0007f9611f0 [a00100044360] ia64_bad_break+0x280/0x4a0 sp=e0007f967ad0 bsp=e0007f9611c8 [a001000693a0] xen_leave_kernel+0x0/0x3e0 sp=e0007f967b80 bsp=e0007f9611c8 [a001006d3b90] bind_virq_to_irqhandler+0x110/0x280 sp=e0007f967d50 bsp=e0007f961138 [a00100019670] xen_register_percpu_irq+0x4b0/0x8e0 sp=e0007f967d60 bsp=e0007f9610f0 [a00100019c40] xen_smp_intr_init_early+0x40/0xc0 sp=e0007f967d60 bsp=e0007f9610c8 [a00100019dd0] xen_platform_send_ipi+0x50/0x2e0 sp=e0007f967d60 bsp=e0007f9610a0 [a0010001a100] ia64_send_ipi+0xa0/0x120 sp=e0007f967d60 bsp=e0007f961068 [a00100063be0] __cpu_up+0x3e0/0x980 sp=e0007f967d60 bsp=e0007f961010 [a001000ce970] cpu_up+0x170/0x2c0 sp=e0007f967e00 bsp=e0007f960fd0 [a00100011720] init+0x140/0x840 sp=e0007f967e00 bsp=e0007f960f98 [a0010001bb10] kernel_thread_helper+0x30/0x60 sp=e0007f967e30 bsp=e0007f960f70 [a001000110e0] start_kernel_thread+0x20/0x40 sp=e0007f967e30 bsp=e0007f960f70 0Kernel panic - not syncing: Attempted to kill init! -- Alex Williamson HP Open Source Linux Org. ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-ia64-devel] [Patch] Pass the bare LSAPIC ID to dom0
On Fri, 2007-07-20 at 18:39 +0900, Akio Takebe wrote: I think this issue is occurred by mismatch infomation of SRAT and LSAPIC. Alex, could you give me ACPI SRAT infomation of the super dome? Here's a raw dump of the SRAT: 000: 53 52 41 54 78 01 00 00 01 4f 48 50 00 00 00 00 010: 53 44 33 32 41 00 00 00 00 00 00 00 20 20 48 50 020: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 030: 01 28 10 41 13 0e 2c 03 00 00 00 00 00 00 00 00 040: 00 e0 ff 7f 00 00 00 00 01 00 00 00 01 00 00 00 050: 00 00 00 00 15 01 2c 01 01 28 10 00 00 00 00 00 060: 00 e0 ff 7f 00 00 00 00 00 20 00 00 00 00 00 00 070: 02 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 080: 01 28 10 00 00 00 00 00 00 00 00 00 01 00 00 00 090: 00 00 00 f8 00 00 00 00 01 00 00 00 01 00 00 00 0a0: 00 00 00 00 00 00 00 00 01 28 10 00 00 00 00 00 0b0: 00 00 00 00 02 00 00 00 00 00 00 fc 00 00 00 00 0c0: 01 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 0d0: 01 28 10 43 14 00 2c 00 00 00 00 80 40 00 00 00 0e0: 00 00 00 80 00 00 00 00 01 00 00 00 01 00 00 00 0f0: 47 50 45 41 13 0e 2c 03 00 10 02 00 01 00 00 00 100: 02 03 61 fc 41 07 00 00 00 10 02 04 01 00 00 00 110: 02 03 61 fc 41 07 00 00 00 10 02 08 01 00 00 00 120: 02 03 61 fc 41 07 00 00 00 10 02 0c 01 00 00 00 130: 02 03 61 fc 41 07 00 00 00 10 09 00 01 00 00 00 140: 09 03 61 fc 41 07 00 00 00 10 09 04 01 00 00 00 150: 09 03 61 fc 41 07 00 00 00 10 09 08 01 00 00 00 160: 09 03 61 fc 41 07 00 00 00 10 09 0c 01 00 00 00 170: 09 03 61 fc 41 07 00 00 And from the boot log (with your patch): XEN) ACPI: LSAPIC (acpi_id[0x10] lsapic_id[0x00] lsapic_eid[0x02] enabled) (XEN) ACPI: LSAPIC (acpi_id[0x12] lsapic_id[0x04] lsapic_eid[0x02] enabled) (XEN) ACPI: LSAPIC (acpi_id[0x14] lsapic_id[0x08] lsapic_eid[0x02] enabled) (XEN) ACPI: LSAPIC (acpi_id[0x16] lsapic_id[0x0c] lsapic_eid[0x02] enabled) (XEN) ACPI: LSAPIC (acpi_id[0x48] lsapic_id[0x00] lsapic_eid[0x09] enabled) (XEN) ACPI: LSAPIC (acpi_id[0x4a] lsapic_id[0x04] lsapic_eid[0x09] enabled) (XEN) ACPI: LSAPIC (acpi_id[0x4c] lsapic_id[0x08] lsapic_eid[0x09] enabled) (XEN) ACPI: LSAPIC (acpi_id[0x4e] lsapic_id[0x0c] lsapic_eid[0x09] enabled) (XEN) enable lsapic entry: 0xf741fd2501cc (id:eid=0:0) - modified (XEN) enable lsapic entry: 0xf741fd2501d8 (id:eid=4:2) (XEN) enable lsapic entry: 0xf741fd2501e4 (id:eid=8:2) (XEN) enable lsapic entry: 0xf741fd2501f0 (id:eid=c:2) (XEN) enable lsapic entry: 0xf741fd2501fc (id:eid=4:0) - modified (XEN) enable lsapic entry: 0xf741fd250208 (id:eid=4:9) (XEN) enable lsapic entry: 0xf741fd250214 (id:eid=8:9) (XEN) enable lsapic entry: 0xf741fd250220 (id:eid=c:9) Thanks, Alex -- Alex Williamson HP Open Source Linux Org. ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-ia64-devel] [Patch] Pass the bare LSAPIC ID to dom0
On Fri, 2007-07-20 at 14:58 -0600, Alex Williamson wrote: On Fri, 2007-07-20 at 18:39 +0900, Akio Takebe wrote: I think this issue is occurred by mismatch infomation of SRAT and LSAPIC. Alex, could you give me ACPI SRAT infomation of the super dome? Here's a raw dump of the SRAT: BTW, this might be a little confusing if you don't know the how the HP systems handle proximity domains and NUMA. This system is configured for no node local memory. Therefore all the memory is listed in proximity domain 16. This is a special node used to represent memory interleaved from the other nodes. The processors are on nodes 2 and 9, which you can see is reflected in the lsapic eid value. Decode below. I'm a little concerned that changing ACPI static tables may end up breaking references that exist in ACPI namespace. Is that just my own paranoia? Thanks, Alex 000: 53 52 41 54 78 01 00 00 01 4f 48 50 00 00 00 00 010: 53 44 33 32 41 00 00 00 00 00 00 00 20 20 48 50 020: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 030: 01 28 10 41 13 0e 2c 03 00 00 00 00 00 00 00 00 040: 00 e0 ff 7f 00 00 00 00 01 00 00 00 01 00 00 00 050: 00 00 00 00 15 01 2c 01 memory pxm 16 base 0x size 0x7fffe000 type memory enabled 01 28 10 00 00 00 00 00 060: 00 e0 ff 7f 00 00 00 00 00 20 00 00 00 00 00 00 070: 02 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 memory pxm 16 base 0x7fffe000 size 0x2000 type reserved enabled 080: 01 28 10 00 00 00 00 00 00 00 00 00 01 00 00 00 090: 00 00 00 f8 00 00 00 00 01 00 00 00 01 00 00 00 0a0: 00 00 00 00 00 00 00 00 memory pxm 16 base 0x0001 size 0xf800 type memory enabled 01 28 10 00 00 00 00 00 0b0: 00 00 00 00 02 00 00 00 00 00 00 fc 00 00 00 00 0c0: 01 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 memory pxm 16 base 0x0002 size 0xfc00 type memory enabled 0d0: 01 28 10 43 14 00 2c 00 00 00 00 80 40 00 00 00 0e0: 00 00 00 80 00 00 00 00 01 00 00 00 01 00 00 00 0f0: 47 50 45 41 13 0e 2c 03 memory pxm 16 base 0x00408000 size 0x8000 type memory enabled 00 10 02 00 01 00 00 00 100: 02 03 61 fc 41 07 00 00 processor pxm 2 apic id 0 enabled eid 2 00 10 02 04 01 00 00 00 110: 02 03 61 fc 41 07 00 00 processor pxm 2 apic id 4 enabled eid 2 00 10 02 08 01 00 00 00 120: 02 03 61 fc 41 07 00 00 processor pxm 2 apic id 8 enabled eid 2 00 10 02 0c 01 00 00 00 130: 02 03 61 fc 41 07 00 00 processor pxm 2 apic id c enabled eid 2 00 10 09 00 01 00 00 00 140: 09 03 61 fc 41 07 00 00 processor pxm 9 apic id 0 enabled eid 9 00 10 09 04 01 00 00 00 150: 09 03 61 fc 41 07 00 00 processor pxm 9 apic id 4 enabled eid 9 00 10 09 08 01 00 00 00 160: 09 03 61 fc 41 07 00 00 processor pxm 9 apic id 8 enabled eid 9 00 10 09 0c 01 00 00 00 170: 09 03 61 fc 41 07 00 00 processor pxm 9 apic id c enabled eid 9 -- Alex Williamson HP Open Source Linux Org. ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
RE: [Xen-ia64-devel] [Patch] Pass the bare LSAPIC ID to dom0
Hi, Anthony and all As you said, we will need to disccuss NUMA support. But this issue is definitely a bug. This bug is related to CONFIG_NUMA=y of dom0. By using dom0_max_vcpus = 2 this issue may be occurred on non-NUMA machine, because current dom0's .config is CONFIG_NUMA=y. So I'd like to have this patch applied, please. Maybe we need to generally consider the NUMA support. Yes, I think so. We need to consider NUMA support with x86 and power people. Best Regards, Akio Takebe ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-ia64-devel] [Patch] Pass the bare LSAPIC ID to dom0
On Wed, 2007-07-18 at 16:03 +0900, Akio Takebe wrote: Hi, all This patch fix a issue which dom0 cannot boot with dom0_max_vcpus. Currently LSAPIC IDs are create by xen, but ACPI SRAT table is the bare table. So on some boxes node_cpuid[].phys_id are different from cpu_physical_id()s, and we cannot boot dom0. I think xen should pass the bare LSAPIC ID to dom0. Applied. Thanks, Alex -- Alex Williamson HP Open Source Linux Org. ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-ia64-devel] [Patch] Pass the bare LSAPIC ID to dom0
On Thu, 2007-07-19 at 17:20 -0600, Alex Williamson wrote: On Wed, 2007-07-18 at 16:03 +0900, Akio Takebe wrote: Hi, all This patch fix a issue which dom0 cannot boot with dom0_max_vcpus. Currently LSAPIC IDs are create by xen, but ACPI SRAT table is the bare table. So on some boxes node_cpuid[].phys_id are different from cpu_physical_id()s, and we cannot boot dom0. I think xen should pass the bare LSAPIC ID to dom0. Applied. Thanks, I should have tested this one more, this actually makes things worse for me. Without this patch, I can boot a 2 cell, 8-way super dome partition with dom0_max_cpus=8. With this patch, it crashes in sd_degenerate just as you reported your system did before. Note in the log below that id:eid generated does not match that provided by the system. Thoughts? Thanks, Alex -- Alex Williamson HP Open Source Linux Org. (XEN) Console output is synchronous. (XEN) Xen command line: BOOT_IMAGE=net0:awilliam/ia64/xen.gz dom0_max_vcpus=8 dom0_mem=2G sync_console loglvl=all (XEN) xen image pstart: 0x400, xenheap pend: 0x800 (XEN) Xen patching physical address access by offset: 0x0 (XEN) find_memory: efi_memmap_walk returns max_page=103fff0 (XEN) Before xen_heap_start: f41c6fb0 (XEN) After xen_heap_start: f43d4000 (XEN) warning: skipping physical page 0 (XEN) Init boot pages: 0x4000 - 0x400. (XEN) Init boot pages: 0x800 - 0x7fffc000. (XEN) Init boot pages: 0x1 - 0x1f800. (XEN) Init boot pages: 0x2 - 0x2fc00. (XEN) Init boot pages: 0x408000 - 0x40fc9ed000. (XEN) Init boot pages: 0x40fd96a048 - 0x40fdf8c018. (XEN) Init boot pages: 0x40fdf8c078 - 0x40fdf8ffa1. (XEN) Init boot pages: 0x40fdf8fffd - 0x40fef9a018. (XEN) Init boot pages: 0x40fef9ae88 - 0x40ffdd8000. (XEN) Init boot pages: 0x40ffe8 - 0x40fffc. (XEN) System RAM: 12095MB (12385360kB) (XEN) size of virtual frame_table: 30288kB (XEN) virtual machine to physical table: f6fff7e00080 size: 6160kB (XEN) max_page: 0x103fff0 (XEN) Xen heap: 60MB (61616kB) (XEN) Reserving non-aligned node boundary @ mfn 262144 (XEN) Reserving non-aligned node boundary @ mfn 524288 (XEN) Reserving non-aligned node boundary @ mfn 16908288 (XEN) Domain heap initialised: DMA width 32 bits (XEN) SAL 3.20: HP Orca/IPF version 6.44 (XEN) SAL Platform features: None (XEN) SAL: AP wakeup using external interrupt vector 0xff (XEN) No logical to physical processor mapping available (XEN) avail:0x1180c600, status:0x600,control:0x1180c000, vm?0x0 (XEN) No VT feature supported. (XEN) cpu_init: current=f4114000 (XEN) vhpt_init: vhpt paddr=0x408001, end=0x408001 (XEN) ACPI: Local APIC address f200fee0 (XEN) ACPI: LSAPIC (acpi_id[0x10] lsapic_id[0x00] lsapic_eid[0x02] enabled) (XEN) CPU 0 (0x0002) enabled (BSP) (XEN) ACPI: LSAPIC (acpi_id[0x12] lsapic_id[0x04] lsapic_eid[0x02] enabled) (XEN) CPU 1 (0x0402) enabled (XEN) ACPI: LSAPIC (acpi_id[0x14] lsapic_id[0x08] lsapic_eid[0x02] enabled) (XEN) CPU 2 (0x0802) enabled (XEN) ACPI: LSAPIC (acpi_id[0x16] lsapic_id[0x0c] lsapic_eid[0x02] enabled) (XEN) CPU 3 (0x0c02) enabled (XEN) ACPI: LSAPIC (acpi_id[0x48] lsapic_id[0x00] lsapic_eid[0x09] enabled) (XEN) CPU 4 (0x0009) enabled (XEN) ACPI: LSAPIC (acpi_id[0x4a] lsapic_id[0x04] lsapic_eid[0x09] enabled) (XEN) CPU 5 (0x0409) enabled (XEN) ACPI: LSAPIC (acpi_id[0x4c] lsapic_id[0x08] lsapic_eid[0x09] enabled) (XEN) CPU 6 (0x0809) enabled (XEN) ACPI: LSAPIC (acpi_id[0x4e] lsapic_id[0x0c] lsapic_eid[0x09] enabled) (XEN) CPU 7 (0x0c09) enabled (XEN) ACPI: IOSAPIC (id[0x0] address[0c101000] gsi_base[16]) (XEN) ACPI: IOSAPIC (id[0x1] address[0f821800] gsi_base[24]) (XEN) ACPI: IOSAPIC (id[0x2] address[0f8210002800] gsi_base[35]) (XEN) ACPI: IOSAPIC (id[0x3] address[0f8210004800] gsi_base[46]) (XEN) ACPI: IOSAPIC (id[0x4] address[0f8210006800] gsi_base[57]) (XEN) ACPI: IOSAPIC (id[0x5] address[0f8210008800] gsi_base[68]) (XEN) ACPI: IOSAPIC (id[0x6] address[0f821000c800] gsi_base[79]) (XEN) ACPI: IOSAPIC (id[0x7] address[0f8210010800] gsi_base[90]) (XEN) ACPI: IOSAPIC (id[0x8] address[0f8210012800] gsi_base[101]) (XEN) ACPI: IOSAPIC (id[0x9] address[0f8210014800] gsi_base[112]) (XEN) ACPI: IOSAPIC (id[0xa] address[0f8210016800] gsi_base[123]) (XEN) ACPI: IOSAPIC (id[0xb] address[0f8210018800] gsi_base[134]) (XEN) ACPI: IOSAPIC (id[0xc] address[0f821001c800] gsi_base[145]) (XEN) ACPI: IOSAPIC (id[0xd] address[0c481000] gsi_base[156]) (XEN) ACPI: IOSAPIC (id[0xe] address[0f891800] gsi_base[164]) (XEN) ACPI: IOSAPIC (id[0xf] address[0f8910002800] gsi_base[175]) (XEN) ACPI: IOSAPIC (id[0x10] address[0f8910004800] gsi_base[186]) (XEN) ACPI: IOSAPIC (id[0x11] address[0f8910006800] gsi_base[197]) (XEN) ACPI: IOSAPIC (id[0x12] address[0f8910008800] gsi_base[208]) (XEN) ACPI: IOSAPIC (id[0x13]
Re: [Xen-ia64-devel] [Patch] Pass the bare LSAPIC ID to dom0
On Thu, Jul 19, 2007 at 05:20:26PM -0600, Alex Williamson wrote: On Wed, 2007-07-18 at 16:03 +0900, Akio Takebe wrote: Hi, all This patch fix a issue which dom0 cannot boot with dom0_max_vcpus. Currently LSAPIC IDs are create by xen, but ACPI SRAT table is the bare table. So on some boxes node_cpuid[].phys_id are different from cpu_physical_id()s, and we cannot boot dom0. I think xen should pass the bare LSAPIC ID to dom0. Hi Akio, I am not sure this patch is complete. Some code of Xen assumes vcpu_id == id (get_lid, IPI). I think you'd better to patch the SRAT (if it is possible). Tristan. ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-ia64-devel] [Patch] Pass the bare LSAPIC ID to dom0
On Fri, 2007-07-20 at 03:53 +0200, Tristan Gingold wrote: On Thu, Jul 19, 2007 at 05:20:26PM -0600, Alex Williamson wrote: On Wed, 2007-07-18 at 16:03 +0900, Akio Takebe wrote: Hi, all This patch fix a issue which dom0 cannot boot with dom0_max_vcpus. Currently LSAPIC IDs are create by xen, but ACPI SRAT table is the bare table. So on some boxes node_cpuid[].phys_id are different from cpu_physical_id()s, and we cannot boot dom0. I think xen should pass the bare LSAPIC ID to dom0. Hi Akio, I am not sure this patch is complete. Some code of Xen assumes vcpu_id == id (get_lid, IPI). I think you'd better to patch the SRAT (if it is possible). That may be the way to go, it should be possible. BTW, now that I look at it again, lsapic-id == 0 really looks like an implementation dependent check. Thanks, Alex -- Alex Williamson HP Open Source Linux Org. ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-ia64-devel] [Patch] Pass the bare LSAPIC ID to dom0
Hi, I forgot Signed-off-by line. Signed-off-by: Akio Takebe [EMAIL PROTECTED] Best Regards, Akio Takebe Hi, all This patch fix a issue which dom0 cannot boot with dom0_max_vcpus. Currently LSAPIC IDs are create by xen, but ACPI SRAT table is the bare table. So on some boxes node_cpuid[].phys_id are different from cpu_physical_id()s, and we cannot boot dom0. I think xen should pass the bare LSAPIC ID to dom0. How about you? Detail; I added the debug printks in build_cpu_to_node_map() on dom0, and get the following message. build_cpu_to_node_map: node_cpuid[0].phys_id=0 cpu_physical_id(0)=0 build_cpu_to_node_map: node_cpuid[1].phys_id=1024 cpu_physical_id(1)=256 build_cpu_to_node_map: node_cpuid[2].phys_id=768 cpu_physical_id(2)=512 build_cpu_to_node_map: node_cpuid[3].phys_id=1792 cpu_physical_id(3)=768 build_cpu_to_node_map: node_cpuid[4].phys_id=32768 cpu_physical_id(4)=1024 build_cpu_to_node_map: node_cpuid[5].phys_id=33792 cpu_physical_id(5)=1280 build_cpu_to_node_map: node_cpuid[6].phys_id=33536 cpu_physical_id(6)=1536 build_cpu_to_node_map: node_cpuid[7].phys_id=34560 cpu_physical_id(7)=1792 build_cpu_to_node_map: node_cpuid[8].phys_id=1 cpu_physical_id(8)=-1 build_cpu_to_node_map: node_cpuid[9].phys_id=1025 cpu_physical_id(9)=-1 build_cpu_to_node_map: node_cpuid[10].phys_id=769 cpu_physical_id(10)=-1 build_cpu_to_node_map: node_cpuid[11].phys_id=1793 cpu_physical_id(11)=-1 build_cpu_to_node_map: node_cpuid[12].phys_id=32769 cpu_physical_id(12)=-1 build_cpu_to_node_map: node_cpuid[13].phys_id=33793 cpu_physical_id(13)=-1 build_cpu_to_node_map: node_cpuid[14].phys_id=33537 cpu_physical_id(14)=-1 build_cpu_to_node_map: node_cpuid[15].phys_id=34561 cpu_physical_id(15)=-1 At that time, some cpus cannot get node id like the below. build_cpu_to_node_map: cpu=0 node_cpuid[0].phys_id=0 is , node=0 build_cpu_to_node_map: cpu=1, node=-1 build_cpu_to_node_map: cpu=2, node=-1 build_cpu_to_node_map: cpu=3 node_cpuid[2].phys_id=768 is , node=0 build_cpu_to_node_map: cpu=4 node_cpuid[1].phys_id=1024 is , node=0 build_cpu_to_node_map: cpu=5, node=-1 build_cpu_to_node_map: cpu=6, node=-1 build_cpu_to_node_map: cpu=7 node_cpuid[3].phys_id=1792 is , node=0 build_cpu_to_node_map: cpu=8, node=-1 build_cpu_to_node_map: cpu=9, node=-1 build_cpu_to_node_map: cpu=10, node=-1 build_cpu_to_node_map: cpu=11, node=-1 build_cpu_to_node_map: cpu=12, node=-1 build_cpu_to_node_map: cpu=13, node=-1 build_cpu_to_node_map: cpu=14, node=-1 build_cpu_to_node_map: cpu=15, node=-1 Finally, dom0 get NULL pointer access like the below. ELILO boot: Uncompressing Linux... done Loading file initrd-2.6.18-xen.img-takebe...done Loading file vmlinuz-2.6.18-xen-takebe...done Uncompressing... done __ ___ ___ __ _ \ \/ /___ _ __ |___ / / _ \_ _ _ __ ___| |_ __ _| |__ | | ___ \ // _ \ '_ \|_ \| | | |__| | | | '_ \/ __| __/ _` | '_ \| |/ _ \ / \ __/ | | | ___) | |_| |__| |_| | | | \__ \ || (_| | |_) | | __/ /_/\_\___|_| |_| |(_)___/\__,_|_| |_|___/\__\__,_|_.__/|_|\___| http://www.cl.cam.ac.uk/netos/xen University of Cambridge Computer Laboratory Xen version 3.0-unstable ([EMAIL PROTECTED]) () Thu Jul 12 21:57:50 JST 2007 Latest ChangeSet: Wed Jul 11 11:32:30 2007 -0600 15558:f536eb8576ee (XEN) Xen command line: BOOT_IMAGE=scsi0:EFI\redhat\xen.gz-takebe dom0_mem= 8G dom0_max_vcpus=8 (XEN) xen image pstart: 0x400, xenheap pend: 0x800 (XEN) Xen patching physical address access by offset: 0x0 (XEN) find_memory: efi_memmap_walk returns max_page=540 (XEN) Before xen_heap_start: f41c6fb0 (XEN) After xen_heap_start: f4c4c000 (XEN) Init boot pages: 0x1d8 - 0x400. (XEN) Init boot pages: 0x800 - 0x69cf. (XEN) Init boot pages: 0x6ac6d098 - 0x6b290010. (XEN) Init boot pages: 0x6b290070 - 0x6b293fb0. (XEN) Init boot pages: 0x6b293ff9 - 0x6b296000. (XEN) Init boot pages: 0x6b48d37d - 0x6b49a010. (XEN) Init boot pages: 0x6b49a6a0 - 0x6d00. (XEN) Init boot pages: 0x1 - 0x10. (XEN) Init boot pages: 0x408000 - 0x41. (XEN) Init boot pages: 0x1400400 - 0x150. (XEN) System RAM: 130688MB (133824512kB) (XEN) size of virtual frame_table: 326928kB (XEN) virtual machine to physical table: f6ffd600 size: 65424kB (XEN) max_page: 0x540 (XEN) allocating frame table/mpt table at mfn 0. (XEN) Xen heap: 51MB (52944kB) (XEN) Reserving non-aligned node boundary @ mfn 262144 (XEN) Reserving non-aligned node boundary @ mfn 16908288 (XEN) Domain heap initialised: DMA width 32 bits (XEN) avail:0x31700740, status:0x740,control: 0x3170,
RE: [Xen-ia64-devel] [Patch] Pass the bare LSAPIC ID to dom0
Hi Akio, I guess this patch is part of NUMA support. Have we considered the whole NUMA support? Such as, 1. We should add NUMA support into Xen. 2. Do we need to provide physical NUMA information to dom0? 1. the node topology. 2. the cost of one hop memory access 3. Do we provide physical node topology to Guest domain? 4. Do Xen scheduler need to consider node topolohy? Yes, this patch fixes LSAPIC ID issue. Maybe we need to generally consider the NUMA support. Thanks, Anthony -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Akio Takebe Sent: 2007年7月18日 15:04 To: xen-ia64-devel Subject: [Xen-ia64-devel] [Patch] Pass the bare LSAPIC ID to dom0 Hi, all This patch fix a issue which dom0 cannot boot with dom0_max_vcpus. Currently LSAPIC IDs are create by xen, but ACPI SRAT table is the bare table. So on some boxes node_cpuid[].phys_id are different from cpu_physical_id()s, and we cannot boot dom0. I think xen should pass the bare LSAPIC ID to dom0. How about you? Detail; I added the debug printks in build_cpu_to_node_map() on dom0, and get the following message. build_cpu_to_node_map: node_cpuid[0].phys_id=0 cpu_physical_id(0)=0 build_cpu_to_node_map: node_cpuid[1].phys_id=1024 cpu_physical_id(1)=256 build_cpu_to_node_map: node_cpuid[2].phys_id=768 cpu_physical_id(2)=512 build_cpu_to_node_map: node_cpuid[3].phys_id=1792 cpu_physical_id(3)=768 build_cpu_to_node_map: node_cpuid[4].phys_id=32768 cpu_physical_id(4)=1024 build_cpu_to_node_map: node_cpuid[5].phys_id=33792 cpu_physical_id(5)=1280 build_cpu_to_node_map: node_cpuid[6].phys_id=33536 cpu_physical_id(6)=1536 build_cpu_to_node_map: node_cpuid[7].phys_id=34560 cpu_physical_id(7)=1792 build_cpu_to_node_map: node_cpuid[8].phys_id=1 cpu_physical_id(8)=-1 build_cpu_to_node_map: node_cpuid[9].phys_id=1025 cpu_physical_id(9)=-1 build_cpu_to_node_map: node_cpuid[10].phys_id=769 cpu_physical_id(10)=-1 build_cpu_to_node_map: node_cpuid[11].phys_id=1793 cpu_physical_id(11)=-1 build_cpu_to_node_map: node_cpuid[12].phys_id=32769 cpu_physical_id(12)=-1 build_cpu_to_node_map: node_cpuid[13].phys_id=33793 cpu_physical_id(13)=-1 build_cpu_to_node_map: node_cpuid[14].phys_id=33537 cpu_physical_id(14)=-1 build_cpu_to_node_map: node_cpuid[15].phys_id=34561 cpu_physical_id(15)=-1 At that time, some cpus cannot get node id like the below. build_cpu_to_node_map: cpu=0 node_cpuid[0].phys_id=0 is , node=0 build_cpu_to_node_map: cpu=1, node=-1 build_cpu_to_node_map: cpu=2, node=-1 build_cpu_to_node_map: cpu=3 node_cpuid[2].phys_id=768 is , node=0 build_cpu_to_node_map: cpu=4 node_cpuid[1].phys_id=1024 is , node=0 build_cpu_to_node_map: cpu=5, node=-1 build_cpu_to_node_map: cpu=6, node=-1 build_cpu_to_node_map: cpu=7 node_cpuid[3].phys_id=1792 is , node=0 build_cpu_to_node_map: cpu=8, node=-1 build_cpu_to_node_map: cpu=9, node=-1 build_cpu_to_node_map: cpu=10, node=-1 build_cpu_to_node_map: cpu=11, node=-1 build_cpu_to_node_map: cpu=12, node=-1 build_cpu_to_node_map: cpu=13, node=-1 build_cpu_to_node_map: cpu=14, node=-1 build_cpu_to_node_map: cpu=15, node=-1 Finally, dom0 get NULL pointer access like the below. ELILO boot: Uncompressing Linux... done Loading file initrd-2.6.18-xen.img-takebe...done Loading file vmlinuz-2.6.18-xen-takebe...done Uncompressing... done __ ___ ___ __ _ \ \/ /___ _ __ |___ / / _ \_ _ _ __ ___| |_ __ _| |__ | | ___ \ // _ \ '_ \|_ \| | | |__| | | | '_ \/ __| __/ _` | '_ \| |/ _ \ / \ __/ | | | ___) | |_| |__| |_| | | | \__ \ || (_| | |_) | | __/ /_/\_\___|_| |_| |(_)___/\__,_|_| |_|___/\__\__,_|_.__/|_|\___| http://www.cl.cam.ac.uk/netos/xen University of Cambridge Computer Laboratory Xen version 3.0-unstable ([EMAIL PROTECTED]) () Thu Jul 12 21:57:50 JST 2007 Latest ChangeSet: Wed Jul 11 11:32:30 2007 -0600 15558:f536eb8576ee (XEN) Xen command line: BOOT_IMAGE=scsi0:EFI\redhat\xen.gz-takebe dom0_mem=8G dom0_max_vcpus=8 (XEN) xen image pstart: 0x400, xenheap pend: 0x800 (XEN) Xen patching physical address access by offset: 0x0 (XEN) find_memory: efi_memmap_walk returns max_page=540 (XEN) Before xen_heap_start: f41c6fb0 (XEN) After xen_heap_start: f4c4c000 (XEN) Init boot pages: 0x1d8 - 0x400. (XEN) Init boot pages: 0x800 - 0x69cf. (XEN) Init boot pages: 0x6ac6d098 - 0x6b290010. (XEN) Init boot pages: 0x6b290070 - 0x6b293fb0. (XEN) Init boot pages: 0x6b293ff9 - 0x6b296000. (XEN) Init boot pages: 0x6b48d37d - 0x6b49a010. (XEN) Init boot pages: 0x6b49a6a0 - 0x6d00. (XEN) Init boot pages: 0x1 - 0x10. (XEN) Init boot pages: 0x408000 - 0x41. (XEN) Init boot pages: 0x1400400 - 0x150. (XEN) System RAM: 130688MB (133824512kB) (XEN) size of virtual frame_table: 326928kB (XEN)
