On 03.12.25 16:32, Grygorii Strashko wrote: > Hi Oleksandr,
Hello Grygorii > > On 03.12.25 13:03, Oleksandr Tyshchenko wrote: >> >> >> On 02.12.25 23:33, Grygorii Strashko wrote: >> >> >> Hello Grygorii >> >>> >>> >>> On 02.12.25 21:32, Oleksandr Tyshchenko wrote: >>>> Creating a guest with a high vCPU count (e.g., >32) fails because >>>> the guest's device tree buffer (DOMU_DTB_SIZE) overflows during >>>> creation. >>>> The FDT nodes for each vCPU quickly exhaust the 4KiB buffer, >>>> causing a guest creation failure. >>>> >>>> Increase the buffer size to 16KiB to support guests up to >>>> the MAX_VIRT_CPUS limit (128). >>>> >>>> Signed-off-by: Oleksandr Tyshchenko <[email protected]> >>>> --- >>>> Noticed when testing the boundary conditions for dom0less guest >>>> creation on Arm64. >>>> >>>> Domain configuration: >>>> fdt mknod /chosen domU0 >>>> fdt set /chosen/domU0 compatible "xen,domain" >>>> fdt set /chosen/domU0 \#address-cells <0x2> >>>> fdt set /chosen/domU0 \#size-cells <0x2> >>>> fdt set /chosen/domU0 memory <0x0 0x10000 > >>>> fdt set /chosen/domU0 cpus <33> >>>> fdt set /chosen/domU0 vpl011 >>>> fdt mknod /chosen/domU0 module@40400000 >>>> fdt set /chosen/domU0/module@40400000 compatible "multiboot,kernel" >>>> "multiboot,module" >>>> fdt set /chosen/domU0/module@40400000 reg <0x0 0x40400000 0x0 0x16000 > >>>> fdt set /chosen/domU0/module@40400000 bootargs "console=ttyAMA0" >>>> >>>> Failure log: >>>> (XEN) Xen dom0less mode detected >>>> (XEN) *** LOADING DOMU cpus=33 memory=0x10000KB *** >>>> (XEN) Loading d1 kernel from boot module @ 0000000040400000 >>>> (XEN) Allocating mappings totalling 64MB for d1: >>>> (XEN) d1 BANK[0] 0x00000040000000-0x00000044000000 (64MB) >>>> (XEN) Device tree generation failed (-22). >>>> (XEN) >>>> (XEN) **************************************** >>>> (XEN) Panic on CPU 0: >>>> (XEN) Could not set up domain domU0 (rc = -22) >>>> (XEN) **************************************** >>>> --- >>>> --- >>>> xen/common/device-tree/dom0less-build.c | 8 +++++--- >>>> 1 file changed, 5 insertions(+), 3 deletions(-) >>>> >>>> diff --git a/xen/common/device-tree/dom0less-build.c b/xen/common/ >>>> device-tree/dom0less-build.c >>>> index 3f5b987ed8..d7d0a47b97 100644 >>>> --- a/xen/common/device-tree/dom0less-build.c >>>> +++ b/xen/common/device-tree/dom0less-build.c >>>> @@ -461,10 +461,12 @@ static int __init >>>> domain_handle_dtb_boot_module(struct domain *d, >>>> /* >>>> * The max size for DT is 2MB. However, the generated DT is small >>>> (not including >>>> - * domU passthrough DT nodes whose size we account separately), 4KB >>>> are enough >>>> - * for now, but we might have to increase it in the future. >>>> + * domU passthrough DT nodes whose size we account separately). The >>>> size is >>>> + * primarily driven by the number of vCPU nodes. The previous 4KiB >>>> buffer was >>>> + * insufficient for guests with high vCPU counts, so it has been >>>> increased >>>> + * to support up to the MAX_VIRT_CPUS limit (128). >>>> */ >>>> -#define DOMU_DTB_SIZE 4096 >>>> +#define DOMU_DTB_SIZE (4096 * 4) >>> >>> May be It wants Kconfig? >>> Or some formula which accounts MAX_VIRT_CPUS? >> >> >> I agree that using a formula that accounts for MAX_VIRT_CPUS is the most >> robust approach. >> >> Here is the empirical data (by testing with the maximum number of device >> tree nodes (e.g., hypervisor and reserved-memory nodes) and enabling all >> optional CPU properties (e.g., clock-frequency)): >> >> cpus=1 >> (XEN) Final compacted FDT size is: 1586 bytes >> >> cpus=2 >> (XEN) Final compacted FDT size is: 1698 bytes >> >> cpus=32 >> (XEN) Final compacted FDT size is: 5058 bytes >> >> cpus=128 >> (XEN) Final compacted FDT size is: 15810 bytes >> >> >> static int __init prepare_dtb_domU(struct domain *d, struct kernel_info >> *kinfo) >> { >> int addrcells, sizecells; >> @@ -569,6 +569,8 @@ static int __init prepare_dtb_domU(struct domain *d, >> struct kernel_info *kinfo) >> if ( ret < 0 ) >> goto err; >> >> + printk("Final compacted FDT size is: %d bytes\n", >> fdt_totalsize(kinfo->fdt)); >> + >> return 0; >> >> err: >> >> This data shows (assuming my testing/calculations are correct): >> >> - A marginal cost of 112 bytes per vCPU in the final, compacted device >> tree. >> - A fixed base size of 1474 bytes for all non-vCPU content. > > Thank for detailed analyses and info. > >> >> Based on that I would propose the following formula with the >> justification: >> >> /* >> * The size is calculated from a fixed baseline plus a scalable >> * portion for each potential vCPU node up to the system limit >> * (MAX_VIRT_CPUS), as the vCPU nodes are the primary consumer >> * of space. >> * >> * The baseline of 2KiB is a safe buffer for all non-vCPU FDT >> * content. The 128 bytes per vCPU is derived from a worst-case >> * analysis of the FDT construction-time size for a single >> * vCPU node. >> */ >> #define DOMU_DTB_SIZE (2048 + (MAX_VIRT_CPUS * 128)) >> >> ********************************************** >> >> Please tell me would you be happy with that? > > It looks ok. Thanks. > One thing I worry about - should it be Xen page aligned? Good question. I could not find any information that the device tree blob (DTB) size itself is required to be page-aligned (at least on Arm64). 1. The Linux Kernel Boot Protocol Documentation on Arm64 says about setting up the device tree": "The device tree blob (dtb) must be placed on an 8-byte boundary and must not exceed 2 megabytes in size." It does not say "the size must be a multiple of..." or "the size must be page-aligned." 2. The official Device Tree Specification says about the "totalsize" field in the header: " - totalsize This field shall contain the total size in bytes of the devicetree data structure." It also does not say "the size must be a multiple of..." or "the size must be page-aligned." My understanding is: no, the size of DTB does not need to be page-aligned. >
