I was unable to, and this does look similar indeed. I tried a variety of pvops kernels and kernel configs and was unable to get past this. I never found resolution and eventually fell back to 3.4.3 w/a xenlinux kernel. Much less sexy but very stable on the same hardware.
I also had related but different problems on IBM 3650 M2s and IBM 3500s with pvops kernels. It seems very prone to crashing at any APIC/ACPI bugs, of which there seem to be quite a bit of in both Dell and IBM. I was toying with the idea of downgrading BIOS's based on the success someone else on xen-devel list reported with that, but I didn't have the time to see that idea through. On Tue, Nov 23, 2010 at 6:51 AM, Ian Campbell <[email protected]> wrote: > Thanks for the report Vincent. > > I've added xen-devel to the CC as well as Cris Daniluk who previously > reported a very similar issue[0] also on an R410 -- Cris did you ever > get a resolution to your issue? > > Vincent's full report is at: > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=603632 > I've also attached the boot log here of which the interesting part looks > to be: > > [ 8.422639] xen: acpi sci 9 > [ 8.434217] Console: colour VGA+ 80x25 > [ 8.441350] console [hvc0] enabled, bootconsole disabled > [ 8.441350] console [hvc0] enabled, bootconsole disabled > [ 8.462694] Xen: using vcpuop timer interface > [ 8.471508] installing Xen timer for CPU 0 > [ 8.479841] BUG: unable to handle kernel paging request at > 0000000000005a08 > [ 8.493868] IP: [<ffffffff810badce>] > __alloc_pages_nodemask+0x8f/0x5f5 > [ 8.507041] PGD 0 > [ 8.511199] Thread overran stack, or stack corrupted > [ 8.521253] Oops: 0000 [#1] SMP > [ 8.527838] last sysfs file: > [ 8.533941] CPU 0 > [ 8.538100] Modules linked in: > [ 8.544342] Pid: 0, comm: swapper Not tainted 2.6.32-5-xen-amd64 > #1 PowerEdge R410 > [ 8.559594] RIP: e030:[<ffffffff810badce>] [<ffffffff810badce>] > __alloc_pages_nodemask+0x8f/0x5f5 > [ 8.577620] RSP: e02b:ffffffff81443c88 EFLAGS: 00010046 > [ 8.588366] RAX: 0000000000000000 RBX: 0000000000005220 RCX: > 0000000000005a00 > [ 8.602752] RDX: 0000000000000000 RSI: 0000000000000002 RDI: > 0000000000005220 > [ 8.617139] RBP: 0000000000004020 R08: 0000000000000002 R09: > ffff88003fc1c010 > [ 8.631525] R10: ffffffff813c2700 R11: 00000000000186a0 R12: > 0000000000005220 > [ 8.645910] R13: 0000000000000002 R14: 0000000000000000 R15: > ffff88000000da28 > [ 8.660300] FS: 0000000000000000(0000) GS:ffff88000349b000(0000) > knlGS:0000000000000000 > [ 8.676591] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 8.688203] CR2: 0000000000005a08 CR3: 0000000001001000 CR4: > 0000000000002660 > [ 8.702589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 8.716975] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > [ 8.731361] Process swapper (pid: 0, threadinfo ffffffff81442000, > task ffffffff814771f0) > [ 8.747654] Stack: > [ 8.751813] ffff88000000da00 00000010813c2765 00000000000212d0 > 00000000000186a0 > [ 8.766199] <0> ffff88000000ac10 ffffffff8100e5b5 > ffffffff8100ec72 00000000000186a0 > [ 8.781625] <0> 00000000000186a0 0000000000000000 > 0000000000005a00 0000000000000000 > [ 8.797572] Call Trace: > [ 8.802603] [<ffffffff8100e5b5>] ? > xen_force_evtchn_callback+0x9/0xa > [ 8.815600] [<ffffffff8100ec72>] ? check_events+0x12/0x20 > [ 8.826695] [<ffffffff810e759d>] ? new_slab+0x42/0x1ca > [ 8.837267] [<ffffffff810e7915>] ? __slab_alloc+0x1f0/0x39b > [ 8.848707] [<ffffffff812f87d8>] ? > irq_to_desc_alloc_node+0x96/0x195 > [ 8.861704] [<ffffffff810e85cb>] ? __kmalloc_node+0xe8/0x146 > [ 8.873317] [<ffffffff812f87d8>] ? > irq_to_desc_alloc_node+0x96/0x195 > [ 8.886316] [<ffffffff812f87d8>] ? > irq_to_desc_alloc_node+0x96/0x195 > [ 8.899317] [<ffffffff811f24df>] ? find_unbound_irq+0x67/0xae > [ 8.911103] [<ffffffff811f259e>] ? bind_virq_to_irq+0x78/0x126 > [ 8.923062] [<ffffffff8100e5b5>] ? > xen_force_evtchn_callback+0x9/0xa > [ 8.936063] [<ffffffff8100e8f6>] ? xen_timer_interrupt+0x0/0x18d > [ 8.948368] [<ffffffff811f29f6>] ? > bind_virq_to_irqhandler+0x19/0x4a > [ 8.961368] [<ffffffff8100e884>] ? xen_setup_timer+0x55/0xaa > [ 8.972982] [<ffffffff81509a5e>] ? xen_time_init+0xaf/0xb5 > [ 8.984247] [<ffffffff8150a491>] ? x86_late_time_init+0xa/0x10 > [ 8.996206] [<ffffffff81506c3d>] ? start_kernel+0x348/0x3e8 > [ 9.007646] [<ffffffff81508c7d>] ? xen_start_kernel+0x57c/0x581 > [ 9.019777] Code: d8 c1 e8 13 83 e0 01 09 44 24 64 41 89 dc 44 23 > 25 28 01 43 00 44 89 e2 83 e2 10 89 54 24 5c 74 05 e8 16 03 25 00 48 8b 4c > 24 50 <48> 83 79 08 00 0f 84 30 05 00 00 83 e3 0f 48 8b 44 24 50 41 bf > [ 9.057561] RIP [<ffffffff810badce>] > __alloc_pages_nodemask+0x8f/0x5f5 > [ 9.070909] RSP <ffffffff81443c88> > [ 9.078015] CR2: 0000000000005a08 > [ 9.084780] ---[ end trace a7919e7f17c0a725 ]--- > [ 9.094136] Kernel panic - not syncing: Attempted to kill the > idle task! > > It's worth noting that the Debian kernels are based on > e73f4955a821f850f5b88c32d12a81714523a95f (less the GPU fixes merged by > bcf16b6b4f34fb40a7aaf637947c7d3bce0be671, which the Debian kernel > maintainer chose to exclude). > > The baseline is slightly old but Debian is now pretty deeply frozen so a > wholesale rebase is not possible, if either of you have run a more > recent kernel the result would be interesting to know. > > The actual crashing RIP corresponds to mm/page_alloc.c:1975 which is in > __alloc_pages_nodemask: > > /* > * Check the zones suitable for the gfp_mask contain at least one > * valid zone. It's possible to have an empty zonelist as a result > * of GFP_THISNODE and a memoryless node > */ > if (unlikely(!zonelist->_zonerefs->zone)) > return NULL; > > zonelist->_zonerefs is an array but looking at the disassembly and the > register dump zonelist itself appears to be 0x5a00 which seems unlikely > to be valid. > > The zonelist ultimately comes from node which is always passed as 0 in > the outer most caller in this stack trace (find_unbound_irq calling > irq_to_desc_alloc_node). > > I'm not sure but looking at the complete bootlog it looks as if the > system may only have node==1 i.e. no 0 node which could plausibly lead > to this sort of issue: > [ 0.000000] Bootmem setup node 1 > 0000000000000000-0000000040000000 > [ 0.000000] NODE_DATA [0000000000008000 - 000000000000ffff] > [ 0.000000] bootmap [0000000000010000 - 0000000000017fff] > pages 8 > [ 0.000000] (8 early reservations) ==> bootmem [0000000000 - > 0040000000] > [ 0.000000] #0 [0000000000 - 0000001000] BIOS data page ==> > [0000000000 - 0000001000] > [ 0.000000] #1 [0003446000 - 0003465000] XEN PAGETABLES ==> > [0003446000 - 0003465000] > [ 0.000000] #2 [0000006000 - 0000008000] TRAMPOLINE ==> > [0000006000 - 0000008000] > [ 0.000000] #3 [0001000000 - 0001694994] TEXT DATA BSS ==> > [0001000000 - 0001694994] > [ 0.000000] #4 [00016b5000 - 0003244e00] RAMDISK ==> > [00016b5000 - 0003244e00] > [ 0.000000] #5 [0003245000 - 0003446000] XEN START INFO ==> > [0003245000 - 0003446000] > [ 0.000000] #6 [0001695000 - 000169532d] BRK ==> > [0001695000 - 000169532d] > [ 0.000000] #7 [0000100000 - 00002e0000] PGTABLE ==> > [0000100000 - 00002e0000] > [ 0.000000] found SMP MP-table at [ffff8800000fe710] fe710 > [ 0.000000] Zone PFN ranges: > [ 0.000000] DMA 0x00000000 -> 0x00001000 > [ 0.000000] DMA32 0x00001000 -> 0x00100000 > [ 0.000000] Normal 0x00100000 -> 0x00100000 > [ 0.000000] Movable zone start PFN for each node > [ 0.000000] early_node_map[2] active PFN ranges > [ 0.000000] 1: 0x00000000 -> 0x000000a0 > [ 0.000000] 1: 0x00000100 -> 0x00040000 > [ 0.000000] On node 1 totalpages: 262048 > [ 0.000000] DMA zone: 56 pages used for memmap > [ 0.000000] DMA zone: 483 pages reserved > [ 0.000000] DMA zone: 3461 pages, LIFO batch:0 > [ 0.000000] DMA32 zone: 3528 pages used for memmap > [ 0.000000] DMA32 zone: 254520 pages, LIFO batch:31 > > Perhaps we should be passing numa_node_id() (e.g. current node) instead > of node 0? There doesn't seem to be another obvious alternative to > passing in an explicit node number to this callchain (some places cope > with -1 but not this path AFAICT). > > It's also not obvious if dom0 should be seeing the tables which describe > the hosts nodes anyway or if we should be clobbering something. Given > that dom0 sees a pseudo-physical address map I'm not convinced seeing > the real SRAT is in any way beneficial. Perhaps we should simply be > clobbering NUMAness until actual PV understanding of NUMA is ready? > > One thing I notice when googling R410 issues is that they apparently > have a "Cores per CPU" BIOS option which might be worth playing with, > since configuring a reduced number of cores might remove node 0 but not > node 1 (odd but not invalid?). Presumably it is also worth making sure > you have the latest BIOS etc. > > It's very much an outside possibility but it is also worth trying the > packages at http://xenbits.xen.org/people/ianc/ which reinstates the > changesets from bcf16b6b4f34fb40a7aaf637947c7d3bce0be671 > > Ian. > > [0] > http://lists.xensource.com/archives/html/xen-devel/2010-06/msg01140.html > > On Tue, 2010-11-16 at 00:32 +0100, Vincent CARON wrote: > > Package: linux-image-2.6.32-5-xen-amd64 > > Version: 2.6.32-27 > > Severity: important > > > > I just tried d-i 6beta1 and booted Squeeeze and its 2.6.32 kernel for > > the first time on my usual server hardware (Dell R410). > > > > I opted for the xen-amd64 kernel, and it boots fine on bare metal. But > > as soon as I tried to boot it as dom0 over Xen hypervisor, it BUG's: > > > > [ 8.479841] BUG: unable to handle kernel paging request at > > 0000000000005a08^M > > [ 8.493868] IP: [<ffffffff810badce>] > > __alloc_pages_nodemask+0x8f/0x5f5^M > > > > Then quickly oopses and panics. I tried various flags: > > - upping dom0_mem from 256M to 1024M (I've been running Lenny/Xen 3.2 > > with 256M happily for several months on the same hw) > > - using Xen 'nommu' > > - using Linux nomodeset > > > > Then I followed instructions on a Xen wiki page to provide verbose > > traces (although they do not look much more verbose than the regular > > boot). > > > > I'm using an IPMI serial-over-lan console which appears as a regular > > UART to Xen. > > > > I'm attaching a boot log to this report. > > > > -- System Information: > > Debian Release: squeeze/sid > > APT prefers testing > > APT policy: (500, 'testing') > > Architecture: amd64 (x86_64) > > > > Kernel: Linux 2.6.32-5-amd64 (SMP w/2 CPU cores) > > Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) > > Shell: /bin/sh linked to /bin/bash > > > > > > > > -- > Ian Campbell > Current Noise: Wolf - Seize The Night > > If you will practice being fictional for a while, you will understand that > fictional characters are sometimes more real than people with bodies and > heartbeats. >
