On 10.09.2025 23:57, Andrew Cooper wrote: > On 10/09/2025 7:58 pm, Jason Andryuk wrote: >> Hi, >> >> We're running Android as a guest and it's running the Compatibility >> Test Suite. During the CTS, the Android domU is rebooted multiple times. >> >> In the middle of the CTS, we've seen reboot fail. xl -vvv shows: >> domainbuilder: detail: Could not allocate memory for HVM guest as we >> cannot claim memory! >> xc: error: panic: xg_dom_boot.c:119: xc_dom_boot_mem_init: can't >> allocate low memory for domain: Out of memory >> libxl: error: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init >> failed: Cannot allocate memory >> domainbuilder: detail: xc_dom_release: called >> >> So the claim failed. The system has enough memory since we're just >> rebooting the same VM. As a work around, I added sleep(1) + retry, >> which works. >> >> The curious part is the memory allocation. For d2 to d5, we have: >> domainbuilder: detail: range: start=0x0 end=0xf0000000 >> domainbuilder: detail: range: start=0x100000000 end=0x1af000000 >> xc: detail: PHYSICAL MEMORY ALLOCATION: >> xc: detail: 4KB PAGES: 0x0000000000000000 >> xc: detail: 2MB PAGES: 0x00000000000006f8 >> xc: detail: 1GB PAGES: 0x0000000000000003 >> >> But when we have to retry the claim for d6, there are no 1GB pages used: >> domainbuilder: detail: range: start=0x0 end=0xf0000000 >> domainbuilder: detail: range: start=0x100000000 end=0x1af000000 >> domainbuilder: detail: HVM claim failed! attempt 0 >> xc: detail: PHYSICAL MEMORY ALLOCATION: >> xc: detail: 4KB PAGES: 0x0000000000002800 >> xc: detail: 2MB PAGES: 0x0000000000000ce4 >> xc: detail: 1GB PAGES: 0x0000000000000000 >> >> But subsequent reboots for d7 and d8 go back to using 1GB pages. >> >> Does the change in memory allocation stick out to anyone? >> >> Unfortunately, I don't have insight into what the failing test is doing. >> >> Xen doesn't seem set up to track the claim across reboot. Retrying >> the claim works in our scenario since we have a controlled configuration. > > This looks to me like a known phenomenon. Ages back, a change was made > in how Xen scrubs memory, from being synchronous in domain_kill(), to > being asynchronous in the idle loop. > > The consequence being that, on an idle system, you can shutdown and > reboot the domain faster, but on a busy system you end up trying to > allocate the new domain while memory from the old domain is still dirty. > > It is a classic example of a false optimisation, which looks great on an > idle system only because the idle CPUs are swallowing the work.
I wouldn't call this a "false optimization", but rather one ... > This impacts the ability to find a 1G aligned block of free memory to > allocate a superpage with, and by the sounds of it, claims (which > predate this behaviour change) aren't aware of the "to be scrubbed" > queue and fail instead. ... which isn't sufficiently integrated with the rest of the allocator. Jan
