Thanks, everyone.
On 2025-09-10 17:57, Andrew Cooper wrote:
On 10/09/2025 7:58 pm, Jason Andryuk wrote:
Hi,
We're running Android as a guest and it's running the Compatibility
Test Suite. During the CTS, the Android domU is rebooted multiple times.
In the middle of the CTS, we've seen reboot fail. xl -vvv shows:
domainbuilder: detail: Could not allocate memory for HVM guest as we
cannot claim memory!
xc: error: panic: xg_dom_boot.c:119: xc_dom_boot_mem_init: can't
allocate low memory for domain: Out of memory
libxl: error: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init
failed: Cannot allocate memory
domainbuilder: detail: xc_dom_release: called
So the claim failed. The system has enough memory since we're just
rebooting the same VM. As a work around, I added sleep(1) + retry,
which works.
The curious part is the memory allocation. For d2 to d5, we have:
domainbuilder: detail: range: start=0x0 end=0xf0000000
domainbuilder: detail: range: start=0x100000000 end=0x1af000000
xc: detail: PHYSICAL MEMORY ALLOCATION:
xc: detail: 4KB PAGES: 0x0000000000000000
xc: detail: 2MB PAGES: 0x00000000000006f8
xc: detail: 1GB PAGES: 0x0000000000000003
But when we have to retry the claim for d6, there are no 1GB pages used:
domainbuilder: detail: range: start=0x0 end=0xf0000000
domainbuilder: detail: range: start=0x100000000 end=0x1af000000
domainbuilder: detail: HVM claim failed! attempt 0
xc: detail: PHYSICAL MEMORY ALLOCATION:
xc: detail: 4KB PAGES: 0x0000000000002800
xc: detail: 2MB PAGES: 0x0000000000000ce4
xc: detail: 1GB PAGES: 0x0000000000000000
But subsequent reboots for d7 and d8 go back to using 1GB pages.
Does the change in memory allocation stick out to anyone?
Unfortunately, I don't have insight into what the failing test is doing.
Xen doesn't seem set up to track the claim across reboot. Retrying
the claim works in our scenario since we have a controlled configuration.
This looks to me like a known phenomenon. Ages back, a change was made
in how Xen scrubs memory, from being synchronous in domain_kill(), to
being asynchronous in the idle loop.
The consequence being that, on an idle system, you can shutdown and
reboot the domain faster, but on a busy system you end up trying to
allocate the new domain while memory from the old domain is still dirty.
It is a classic example of a false optimisation, which looks great on an
idle system only because the idle CPUs are swallowing the work.
This impacts the ability to find a 1G aligned block of free memory to
allocate a superpage with, and by the sounds of it, claims (which
predate this behaviour change) aren't aware of the "to be scrubbed"
queue and fail instead.
Claims check total_avail_pages and outstanding_claims. It looks like
free_heap_pages() sets PGC_need_scrub and then increments
total_avail_pages. But then it's not getting through the accounting far
enough to stake a claim?
Also free_heap_page() looks like it's trying to merge chunks - I thought
that would handle larger allocations. Are they not truly usable until
they've been scrubbed, which leads to the lack of 1GB pages?
Clearly I need to learn more here.
I thought OpenXT had a revert of this. IIRC it was considered a
material regression in being able to know when a domain has gone away.
OpenXT wants to scrub the memory ASAP so there is no remnant data. They
is a patch for that.
Thanks,
Jason
