Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
On Wed, 1 Jan 2020, Julien Grall wrote: > On 31/12/2019 22:48, Roman Shaposhnik wrote: > > Hi! > > Hi, > > > > > Good news: one that type was fixed Xen booted just fine and detected > > all the available 2G of memory. > > Thank you for testing. @Stefano are you going to prepare and submit the patch? yes, I will, thank you for spotting a bug in the patch ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
On 31/12/2019 22:48, Roman Shaposhnik wrote: Hi! Hi, Good news: one that type was fixed Xen booted just fine and detected all the available 2G of memory. Thank you for testing. @Stefano are you going to prepare and submit the patch? Cheers, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
Hi Julien,
On Sun, Dec 29, 2019 at 10:01 AM Julien Grall wrote:
>
> Hi,
>
> On 21/12/2019 01:37, Roman Shaposhnik wrote:
> > On Thu, Dec 19, 2019 at 4:01 PM Stefano Stabellini
> > wrote:
> >>
> >> On Thu, 19 Dec 2019, Julien Grall wrote:
> > In fact most of people on Arm are using GRUB rather than EFI directly as
> > this is more friendly to use.
> >
> > Regarding the devicetree, Xen and Linux will completely ignore the
> > memory nodes in Xen if using EFI. This because the EFI memory map will
> > give you an overview of the platform with the EFI regions included.
>
> Aha! So in that sense it is a bug in Xen after all, right? (that's what
> you're
> referring to when you say you now understand what needs to get fixed).
> >>>
> >>> Yes. The EFI memory map is a list of existing memory with a type
> >>> associated to
> >>> it (Conventional, BootServiceCodes, MemoryMappedIO...).
> >>>
> >>> The OS/Hypervisor will have to go through them and check which regions are
> >>> usuable. Compare to Linux, Xen has limited itself to only a few types.
> >>>
> >>> However, I think we can be on a par with Linux here.
> >>
> >> I gave a look at the Linux implementation, the interesting bit is
> >> drivers/firmware/efi/arm-init.c:is_usable_memory as far as I can tell.
> >> I also gave a look at the Xen side, which is
> >> xen/arch/arm/efi/efi-boot.h:efi_process_memory_map_bootinfo. As guessed,
> >> the two are not quite the same.
> >>
> >> One of the main differences is that Linux uses as "System RAM" even
> >> regions that were marked as EFI_BOOT_SERVICES_CODE/DATA and
> >> EFI_LOADER_CODE/DATA because they will get freed anyway. Xen doesn't
> >> do that unless map_bs is set.
> >>
> >> I wrote a quick patch to implement the Linux behavior on Xen, only
> >> lightly tested. I can confirm that I see more memory this way. However,
> >> I am not sure we actually want to import the Linux behavior wholesale.
> >>
> >> Anyway, Roman, could you please let me know if this patch solves the
> >> issue?
> >
> > Tried the attached patch -- but it seems I can't boot at all with this. Xen
> > doesn't print anything on the console either.
>
> Thank you for trying the patch. Do you have earlyprintk enabled for the
> hikey board?
No (since I thought it wasn't possible on ARM :-)) but now that you
mentioned it,
I've found this:
http://xenbits.xenproject.org/docs/4.13-testing/misc/arm/early-printk.txt
and I'd be more than happy to try (hopefully CONFIG_EARLY_PRINTK= hikey960
will do the trick).
> > To Julien's point -- should I reduce the # of types and try again?
>
> From my understanding, the field Attribute is a series of flag telling
> what the region can support.
>
> So it would be possible to have other flags set at the same time as
> EFI_MEMORY_WC. However, the check in the patch below is an == equal and
> would potentially discard a lot of regions (if not all regions).
>
> In other words...
>
> >>
> >>
> >> diff --git a/xen/arch/arm/efi/efi-boot.h b/xen/arch/arm/efi/efi-boot.h
> >> index ca655ff003..ad18ff3669 100644
> >> --- a/xen/arch/arm/efi/efi-boot.h
> >> +++ b/xen/arch/arm/efi/efi-boot.h
> >> @@ -149,10 +149,14 @@ static EFI_STATUS __init
> >> efi_process_memory_map_bootinfo(EFI_MEMORY_DESCRIPTOR *
> >>
> >> for ( Index = 0; Index < (mmap_size / desc_size); Index++ )
> >> {
> >> -if ( desc_ptr->Type == EfiConventionalMemory ||
> >> - (!map_bs &&
> >> - (desc_ptr->Type == EfiBootServicesCode ||
> >> - desc_ptr->Type == EfiBootServicesData)) )
> >> +if ( desc_ptr->Attribute == EFI_MEMORY_WB &&
>
> ... this should be desc_ptr->Attribute & EFI_MEMORY_WB.
>
> Can you give a spin with this change and see how far you can go?
Aha! That makes much more sense -- will give it a try tomorrow
(in conjunction with earlyprintk)
Thanks,
Roman.
> >> + (desc_ptr->Type == EfiConventionalMemory ||
> >> + desc_ptr->Type == EfiLoaderCode ||
> >> + desc_ptr->Type == EfiLoaderData ||
> >> + desc_ptr->Type == EfiACPIReclaimMemory ||
> >> + desc_ptr->Type == EfiPersistentMemory ||
> >> + desc_ptr->Type == EfiBootServicesCode ||
> >> + desc_ptr->Type == EfiBootServicesData) )
> >> {
> >> if ( !meminfo_add_bank(, desc_ptr) )
> >> {
> >> diff --git a/xen/include/efi/efidef.h b/xen/include/efi/efidef.h
> >> index 86a7e111bf..f46207840f 100644
> >> --- a/xen/include/efi/efidef.h
> >> +++ b/xen/include/efi/efidef.h
> >> @@ -147,6 +147,7 @@ typedef enum {
> >> EfiMemoryMappedIO,
> >> EfiMemoryMappedIOPortSpace,
> >> EfiPalCode,
> >> +EfiPersistentMemory,
> >> EfiMaxMemoryType
> >> } EFI_MEMORY_TYPE;
> >>
>
> Cheers,
>
> --
> Julien Grall
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
Hi,
On 21/12/2019 01:37, Roman Shaposhnik wrote:
On Thu, Dec 19, 2019 at 4:01 PM Stefano Stabellini
wrote:
On Thu, 19 Dec 2019, Julien Grall wrote:
In fact most of people on Arm are using GRUB rather than EFI directly as
this is more friendly to use.
Regarding the devicetree, Xen and Linux will completely ignore the
memory nodes in Xen if using EFI. This because the EFI memory map will
give you an overview of the platform with the EFI regions included.
Aha! So in that sense it is a bug in Xen after all, right? (that's what
you're
referring to when you say you now understand what needs to get fixed).
Yes. The EFI memory map is a list of existing memory with a type associated to
it (Conventional, BootServiceCodes, MemoryMappedIO...).
The OS/Hypervisor will have to go through them and check which regions are
usuable. Compare to Linux, Xen has limited itself to only a few types.
However, I think we can be on a par with Linux here.
I gave a look at the Linux implementation, the interesting bit is
drivers/firmware/efi/arm-init.c:is_usable_memory as far as I can tell.
I also gave a look at the Xen side, which is
xen/arch/arm/efi/efi-boot.h:efi_process_memory_map_bootinfo. As guessed,
the two are not quite the same.
One of the main differences is that Linux uses as "System RAM" even
regions that were marked as EFI_BOOT_SERVICES_CODE/DATA and
EFI_LOADER_CODE/DATA because they will get freed anyway. Xen doesn't
do that unless map_bs is set.
I wrote a quick patch to implement the Linux behavior on Xen, only
lightly tested. I can confirm that I see more memory this way. However,
I am not sure we actually want to import the Linux behavior wholesale.
Anyway, Roman, could you please let me know if this patch solves the
issue?
Tried the attached patch -- but it seems I can't boot at all with this. Xen
doesn't print anything on the console either.
Thank you for trying the patch. Do you have earlyprintk enabled for the
hikey board?
To Julien's point -- should I reduce the # of types and try again?
From my understanding, the field Attribute is a series of flag telling
what the region can support.
So it would be possible to have other flags set at the same time as
EFI_MEMORY_WC. However, the check in the patch below is an == equal and
would potentially discard a lot of regions (if not all regions).
In other words...
diff --git a/xen/arch/arm/efi/efi-boot.h b/xen/arch/arm/efi/efi-boot.h
index ca655ff003..ad18ff3669 100644
--- a/xen/arch/arm/efi/efi-boot.h
+++ b/xen/arch/arm/efi/efi-boot.h
@@ -149,10 +149,14 @@ static EFI_STATUS __init
efi_process_memory_map_bootinfo(EFI_MEMORY_DESCRIPTOR *
for ( Index = 0; Index < (mmap_size / desc_size); Index++ )
{
-if ( desc_ptr->Type == EfiConventionalMemory ||
- (!map_bs &&
- (desc_ptr->Type == EfiBootServicesCode ||
- desc_ptr->Type == EfiBootServicesData)) )
+if ( desc_ptr->Attribute == EFI_MEMORY_WB &&
... this should be desc_ptr->Attribute & EFI_MEMORY_WB.
Can you give a spin with this change and see how far you can go?
+ (desc_ptr->Type == EfiConventionalMemory ||
+ desc_ptr->Type == EfiLoaderCode ||
+ desc_ptr->Type == EfiLoaderData ||
+ desc_ptr->Type == EfiACPIReclaimMemory ||
+ desc_ptr->Type == EfiPersistentMemory ||
+ desc_ptr->Type == EfiBootServicesCode ||
+ desc_ptr->Type == EfiBootServicesData) )
{
if ( !meminfo_add_bank(, desc_ptr) )
{
diff --git a/xen/include/efi/efidef.h b/xen/include/efi/efidef.h
index 86a7e111bf..f46207840f 100644
--- a/xen/include/efi/efidef.h
+++ b/xen/include/efi/efidef.h
@@ -147,6 +147,7 @@ typedef enum {
EfiMemoryMappedIO,
EfiMemoryMappedIOPortSpace,
EfiPalCode,
+EfiPersistentMemory,
EfiMaxMemoryType
} EFI_MEMORY_TYPE;
Cheers,
--
Julien Grall
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
On Thu, Dec 19, 2019 at 4:01 PM Stefano Stabellini
wrote:
>
> On Thu, 19 Dec 2019, Julien Grall wrote:
> > > > In fact most of people on Arm are using GRUB rather than EFI directly as
> > > > this is more friendly to use.
> > > >
> > > > Regarding the devicetree, Xen and Linux will completely ignore the
> > > > memory nodes in Xen if using EFI. This because the EFI memory map will
> > > > give you an overview of the platform with the EFI regions included.
> > >
> > > Aha! So in that sense it is a bug in Xen after all, right? (that's what
> > > you're
> > > referring to when you say you now understand what needs to get fixed).
> >
> > Yes. The EFI memory map is a list of existing memory with a type associated
> > to
> > it (Conventional, BootServiceCodes, MemoryMappedIO...).
> >
> > The OS/Hypervisor will have to go through them and check which regions are
> > usuable. Compare to Linux, Xen has limited itself to only a few types.
> >
> > However, I think we can be on a par with Linux here.
>
> I gave a look at the Linux implementation, the interesting bit is
> drivers/firmware/efi/arm-init.c:is_usable_memory as far as I can tell.
> I also gave a look at the Xen side, which is
> xen/arch/arm/efi/efi-boot.h:efi_process_memory_map_bootinfo. As guessed,
> the two are not quite the same.
>
> One of the main differences is that Linux uses as "System RAM" even
> regions that were marked as EFI_BOOT_SERVICES_CODE/DATA and
> EFI_LOADER_CODE/DATA because they will get freed anyway. Xen doesn't
> do that unless map_bs is set.
>
> I wrote a quick patch to implement the Linux behavior on Xen, only
> lightly tested. I can confirm that I see more memory this way. However,
> I am not sure we actually want to import the Linux behavior wholesale.
>
> Anyway, Roman, could you please let me know if this patch solves the
> issue?
Tried the attached patch -- but it seems I can't boot at all with this. Xen
doesn't print anything on the console either.
To Julien's point -- should I reduce the # of types and try again?
Thanks,
Roman.
>
>
>
> diff --git a/xen/arch/arm/efi/efi-boot.h b/xen/arch/arm/efi/efi-boot.h
> index ca655ff003..ad18ff3669 100644
> --- a/xen/arch/arm/efi/efi-boot.h
> +++ b/xen/arch/arm/efi/efi-boot.h
> @@ -149,10 +149,14 @@ static EFI_STATUS __init
> efi_process_memory_map_bootinfo(EFI_MEMORY_DESCRIPTOR *
>
> for ( Index = 0; Index < (mmap_size / desc_size); Index++ )
> {
> -if ( desc_ptr->Type == EfiConventionalMemory ||
> - (!map_bs &&
> - (desc_ptr->Type == EfiBootServicesCode ||
> - desc_ptr->Type == EfiBootServicesData)) )
> +if ( desc_ptr->Attribute == EFI_MEMORY_WB &&
> + (desc_ptr->Type == EfiConventionalMemory ||
> + desc_ptr->Type == EfiLoaderCode ||
> + desc_ptr->Type == EfiLoaderData ||
> + desc_ptr->Type == EfiACPIReclaimMemory ||
> + desc_ptr->Type == EfiPersistentMemory ||
> + desc_ptr->Type == EfiBootServicesCode ||
> + desc_ptr->Type == EfiBootServicesData) )
> {
> if ( !meminfo_add_bank(, desc_ptr) )
> {
> diff --git a/xen/include/efi/efidef.h b/xen/include/efi/efidef.h
> index 86a7e111bf..f46207840f 100644
> --- a/xen/include/efi/efidef.h
> +++ b/xen/include/efi/efidef.h
> @@ -147,6 +147,7 @@ typedef enum {
> EfiMemoryMappedIO,
> EfiMemoryMappedIOPortSpace,
> EfiPalCode,
> +EfiPersistentMemory,
> EfiMaxMemoryType
> } EFI_MEMORY_TYPE;
>
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
Hi,
On 20/12/2019 00:01, Stefano Stabellini wrote:
On Thu, 19 Dec 2019, Julien Grall wrote:
In fact most of people on Arm are using GRUB rather than EFI directly as
this is more friendly to use.
Regarding the devicetree, Xen and Linux will completely ignore the
memory nodes in Xen if using EFI. This because the EFI memory map will
give you an overview of the platform with the EFI regions included.
Aha! So in that sense it is a bug in Xen after all, right? (that's what
you're
referring to when you say you now understand what needs to get fixed).
Yes. The EFI memory map is a list of existing memory with a type associated to
it (Conventional, BootServiceCodes, MemoryMappedIO...).
The OS/Hypervisor will have to go through them and check which regions are
usuable. Compare to Linux, Xen has limited itself to only a few types.
However, I think we can be on a par with Linux here.
I gave a look at the Linux implementation, the interesting bit is
drivers/firmware/efi/arm-init.c:is_usable_memory as far as I can tell.
I also gave a look at the Xen side, which is
xen/arch/arm/efi/efi-boot.h:efi_process_memory_map_bootinfo. As guessed,
the two are not quite the same.
One of the main differences is that Linux uses as "System RAM" even
regions that were marked as EFI_BOOT_SERVICES_CODE/DATA and
EFI_LOADER_CODE/DATA because they will get freed anyway. Xen doesn't
do that unless map_bs is set.
Well, map_bs is used to request to map the boot services into Xen PT. In
other words, you can't consider them as usuable if that option is set.
I wrote a quick patch to implement the Linux behavior on Xen, only
lightly tested. I can confirm that I see more memory this way. However,
I am not sure we actually want to import the Linux behavior wholesale.
This is not what I had in mind, we still need to keep Xen behavior for
boot services and ensure the region are recorded/skipped as expected
(see more below).
Anyway, Roman, could you please let me know if this patch solves the
issue?
Please see a couple of comments below.
diff --git a/xen/arch/arm/efi/efi-boot.h b/xen/arch/arm/efi/efi-boot.h
index ca655ff003..ad18ff3669 100644
--- a/xen/arch/arm/efi/efi-boot.h
+++ b/xen/arch/arm/efi/efi-boot.h
@@ -149,10 +149,14 @@ static EFI_STATUS __init
efi_process_memory_map_bootinfo(EFI_MEMORY_DESCRIPTOR *
for ( Index = 0; Index < (mmap_size / desc_size); Index++ )
{
-if ( desc_ptr->Type == EfiConventionalMemory ||
- (!map_bs &&
- (desc_ptr->Type == EfiBootServicesCode ||
- desc_ptr->Type == EfiBootServicesData)) )
+if ( desc_ptr->Attribute == EFI_MEMORY_WB &&
This is not quite the same as Linux. They use:
desc_ptr->Attribute & EFI_MEMORY_WB
But I don't see why Attribute should be strictly equal to EFI_MEMORY_WB.
+ (desc_ptr->Type == EfiConventionalMemory ||
+ desc_ptr->Type == EfiLoaderCode ||
+ desc_ptr->Type == EfiLoaderData ||
+ desc_ptr->Type == EfiACPIReclaimMemory ||
Linux will explictly reserve the ACPI region (see the caller of
is_usable_memory()). For us this is done in the "else if" part.
+ desc_ptr->Type == EfiPersistentMemory ||
+ desc_ptr->Type == EfiBootServicesCode ||
+ desc_ptr->Type == EfiBootServicesData) )
{
if ( !meminfo_add_bank(, desc_ptr) )
{
diff --git a/xen/include/efi/efidef.h b/xen/include/efi/efidef.h
index 86a7e111bf..f46207840f 100644
--- a/xen/include/efi/efidef.h
+++ b/xen/include/efi/efidef.h
@@ -147,6 +147,7 @@ typedef enum {
EfiMemoryMappedIO,
EfiMemoryMappedIOPortSpace,
EfiPalCode,
+EfiPersistentMemory,
EfiMaxMemoryType
} EFI_MEMORY_TYPE;
Cheers,
--
Julien Grall
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
On Thu, 19 Dec 2019, Julien Grall wrote:
> > > In fact most of people on Arm are using GRUB rather than EFI directly as
> > > this is more friendly to use.
> > >
> > > Regarding the devicetree, Xen and Linux will completely ignore the
> > > memory nodes in Xen if using EFI. This because the EFI memory map will
> > > give you an overview of the platform with the EFI regions included.
> >
> > Aha! So in that sense it is a bug in Xen after all, right? (that's what
> > you're
> > referring to when you say you now understand what needs to get fixed).
>
> Yes. The EFI memory map is a list of existing memory with a type associated to
> it (Conventional, BootServiceCodes, MemoryMappedIO...).
>
> The OS/Hypervisor will have to go through them and check which regions are
> usuable. Compare to Linux, Xen has limited itself to only a few types.
>
> However, I think we can be on a par with Linux here.
I gave a look at the Linux implementation, the interesting bit is
drivers/firmware/efi/arm-init.c:is_usable_memory as far as I can tell.
I also gave a look at the Xen side, which is
xen/arch/arm/efi/efi-boot.h:efi_process_memory_map_bootinfo. As guessed,
the two are not quite the same.
One of the main differences is that Linux uses as "System RAM" even
regions that were marked as EFI_BOOT_SERVICES_CODE/DATA and
EFI_LOADER_CODE/DATA because they will get freed anyway. Xen doesn't
do that unless map_bs is set.
I wrote a quick patch to implement the Linux behavior on Xen, only
lightly tested. I can confirm that I see more memory this way. However,
I am not sure we actually want to import the Linux behavior wholesale.
Anyway, Roman, could you please let me know if this patch solves the
issue?
diff --git a/xen/arch/arm/efi/efi-boot.h b/xen/arch/arm/efi/efi-boot.h
index ca655ff003..ad18ff3669 100644
--- a/xen/arch/arm/efi/efi-boot.h
+++ b/xen/arch/arm/efi/efi-boot.h
@@ -149,10 +149,14 @@ static EFI_STATUS __init
efi_process_memory_map_bootinfo(EFI_MEMORY_DESCRIPTOR *
for ( Index = 0; Index < (mmap_size / desc_size); Index++ )
{
-if ( desc_ptr->Type == EfiConventionalMemory ||
- (!map_bs &&
- (desc_ptr->Type == EfiBootServicesCode ||
- desc_ptr->Type == EfiBootServicesData)) )
+if ( desc_ptr->Attribute == EFI_MEMORY_WB &&
+ (desc_ptr->Type == EfiConventionalMemory ||
+ desc_ptr->Type == EfiLoaderCode ||
+ desc_ptr->Type == EfiLoaderData ||
+ desc_ptr->Type == EfiACPIReclaimMemory ||
+ desc_ptr->Type == EfiPersistentMemory ||
+ desc_ptr->Type == EfiBootServicesCode ||
+ desc_ptr->Type == EfiBootServicesData) )
{
if ( !meminfo_add_bank(, desc_ptr) )
{
diff --git a/xen/include/efi/efidef.h b/xen/include/efi/efidef.h
index 86a7e111bf..f46207840f 100644
--- a/xen/include/efi/efidef.h
+++ b/xen/include/efi/efidef.h
@@ -147,6 +147,7 @@ typedef enum {
EfiMemoryMappedIO,
EfiMemoryMappedIOPortSpace,
EfiPalCode,
+EfiPersistentMemory,
EfiMaxMemoryType
} EFI_MEMORY_TYPE;
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
Hi Roman, On 19/12/2019 00:28, Roman Shaposhnik wrote: On Wed, Dec 18, 2019 at 2:17 PM Julien Grall wrote: Hi Roman, On 18/12/2019 17:03, Roman Shaposhnik wrote: On Wed, Dec 18, 2019 at 3:50 AM Julien Grall wrote: So -- nothing boots directly by UEFI -- everything goes through GRUB. However, my understanding is that GRUB will detect devicetree information provided by UEFI (even though devicetree command is supposed to completely replace that). Hence it is possible that Linux relies on some residuals left in memory by GRUB that Xen doesn't pay attention to (but this is a pretty wild speculation only). While it goes through GRUB, it is a bootloader and will just act as a proxy for EFI. So EFI application such as Xen/Linux can still be loaded and take advantage of runtime servies if present/implemented. Aha! So then it depends on Xen actually using those EFI services. Which leads to my first question: 1. would it be possible to stay completely with just devicetrees information by passing efi=no-rs to Xen? This will only disabled the runtime services (note that they are not supported on Xen on Arm today). What I described above is part of the boot services and can't be disabled. Also, I am not entirely sure GRUB/EFI will update you device-tree to point out the memory that was carved out for things like ATF. Looking at the DTS memory node you provided in another e-mail, it seems the memory map is slightly different. In fact most of people on Arm are using GRUB rather than EFI directly as this is more friendly to use. Regarding the devicetree, Xen and Linux will completely ignore the memory nodes in Xen if using EFI. This because the EFI memory map will give you an overview of the platform with the EFI regions included. Aha! So in that sense it is a bug in Xen after all, right? (that's what you're referring to when you say you now understand what needs to get fixed). Yes. The EFI memory map is a list of existing memory with a type associated to it (Conventional, BootServiceCodes, MemoryMappedIO...). The OS/Hypervisor will have to go through them and check which regions are usuable. Compare to Linux, Xen has limited itself to only a few types. However, I think we can be on a par with Linux here. Cheers, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
Hi Julien! First of all -- thank you so much for detailed explanations -- this is very much appreciated. A few questions still (if you don't mind): On Wed, Dec 18, 2019 at 2:17 PM Julien Grall wrote: > > Hi Roman, > > On 18/12/2019 17:03, Roman Shaposhnik wrote: > > On Wed, Dec 18, 2019 at 3:50 AM Julien Grall wrote: > > So -- nothing boots directly by UEFI -- everything goes through GRUB. > > > > However, my understanding is that GRUB will detect devicetree > > information provided by UEFI (even though devicetree command is > > supposed to completely replace that). Hence it is possible that Linux > > relies on some residuals left in memory by GRUB that Xen doesn't pay > > attention to (but this is a pretty wild speculation only). > > While it goes through GRUB, it is a bootloader and will just act as a > proxy for EFI. So EFI application such as Xen/Linux can still be loaded > and take advantage of runtime servies if present/implemented. Aha! So then it depends on Xen actually using those EFI services. Which leads to my first question: 1. would it be possible to stay completely with just devicetrees information by passing efi=no-rs to Xen? > In fact most of people on Arm are using GRUB rather than EFI directly as > this is more friendly to use. > > Regarding the devicetree, Xen and Linux will completely ignore the > memory nodes in Xen if using EFI. This because the EFI memory map will > give you an overview of the platform with the EFI regions included. Aha! So in that sense it is a bug in Xen after all, right? (that's what you're referring to when you say you now understand what needs to get fixed). Thanks, Roman. ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
Hi,
On 18/12/2019 17:09, Roman Shaposhnik wrote:
Hi,
On Wed, Dec 18, 2019 at 4:56 AM Julien Grall wrote:
So that is, in fact, my first question -- why is Xen not showing
available memory in xl info?
I am not entirely sure what exact information you want.
The output you dumped above contain the available memory for the memory
(see "free_memory").
Are you looking from something different?
Just to be clear: I was giving 2G via devicetrees (the same device
trees that would
make Linux detect 2G of RAM) hence I was expecting xl info to show that. Instead
I only got 1120M shown by xl info.
On 18/12/2019 00:04, Roman Shaposhnik wrote:
memory {
device_type = "memory";
reg = <0x0 0x0 0x0 0x5e0 0x0 0x5f0 0x0 0x1000
0x0 0x5f02000 0x0 0xefd000 0x0 0x6e0 0x0 0x60f000 0x0 0x741
0x0 0x1aaf 0x0 0x21f0 0x0 0x10 0x0 0x2200 0x0
0x1c00>;
};
reserved-memory {
ranges;
#size-cells = <0x2>;
#address-cells = <0x2>;
ramoops@21f0 {
ftrace-size = <0x2>;
console-size = <0x2>;
reg = <0x0 0x21f0 0x0 0x10>;
record-size = <0x2>;
compatible = "ramoops";
};
linux,cma {
linux,cma-default;
reusable;
size = <0x0 0x800>;
compatible = "shared-dma-pool";
};
};
If you look at the REG -- it does now add up to 2Gb, but booting Xen
with it has exactly the
same effect as booting it with: reg = <0x0 0x0 0x0 0x8000>;\
If you boot Xen using EFI, the memory information wil come from EFI and
the DT node will be ignored. So unless UEFI is able to pick up the
modification of the DT memory node, modifying the DT is not going to
affect anything.
That's a good point, but given that I always go through GRUB, I was
expecting devicetree command to completely overshadow whatever
information UEFI may have. Am I wrong?
GRUB will load Xen/Linux as an EFI application. Both of them will ignore
the memory nodes when booting using EFI. For more details, see the
answer I wrote separately.
I am attaching a full log, and I see the following in the logs:
(XEN) Allocating 1:1 mappings totalling 720MB for dom0:
(XEN) BANK[0] 0x000800-0x001c00 (320MB)
(XEN) BANK[1] 0x004000-0x005800 (384MB)
(XEN) BANK[2] 0x007b00-0x007c00 (16MB)
Which sort of makes sense, I guess -- but I still don't understand
where all these ranges
are coming from and how come Xen doesn't see the full 2Gb even with various
devicetrees I tried.
The range aboves describe the memory range given to Dom0. For all the
memory given to Xen,m you want to look at the top of your log:
(XEN) Checking for initrd in /chosen
(XEN) RAM: - 05df
(XEN) RAM: 05f0 - 06dfefff
(XEN) RAM: 06e0 - 0740efff
(XEN) RAM: 0741 - 1db8dfff
(XEN) RAM: 350f - 3dbd2fff
(XEN) RAM: 3dbd3000 - 3dff
(XEN) RAM: 4000 - 5a653fff
(XEN) RAM: 7ada - 7ada3fff
(XEN) RAM: 7aea8000 - 7afa9fff
(XEN) RAM: 7afaa000 - 7ec73fff
(XEN) RAM: 7ec74000 - 7fdddfff
(XEN) RAM: 7fdde000 - 7fea5fff
(XEN) RAM: 7fea6000 - 7ff6dfff
(XEN) RAM: 7000 - 7fff
Looking at the differences with the Linux logs, there is indeed some
memory not detected by Xen.
On Xen, we only consider usuable memory any EFI description with
EfiConventionalMemory, EfiBootServicesCode and EfiBootServicesData.
Linux include more type here, so this may explain why we see a difference.
While Looking at it, I have also noticed that we don't seem to care
about the memory attribute. I suspect this could be another latent issue
in Xen if the attribute does not match.
Anything I can do to help debug this? I can run any kind of debug builds, etc.
if needed.
Thank you for the offer, I think I have a good understanding of the
problem now. So debug should not be necessary.
However, I would appreciate if anyone could help to write a patch for it.
I mean -- at this point it would be really great to get HiKey back to the status
of Xen-on-ARM developer board.
Any ideas here would be greatly apprecaited!
Thanks,
Roman.
P.S. Any guess at what these mean?
(XEN) traps.c:1973:d0v0 HSR=0x93880006 pc=0x008738
gva=0x872f2000 gpa=0x0f
(XEN) traps.c:1973:d0v0 HSR=0x93880006 pc=0x00b734e558
gva=0xb72eb000 gpa=0x0f
(XEN) traps.c:1973:d0v0 HSR=0x93880006 pc=0x008f9d2558
gva=0x8f96f000 gpa=0x0f
It means
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
Hi Roman, On 18/12/2019 17:03, Roman Shaposhnik wrote: On Wed, Dec 18, 2019 at 3:50 AM Julien Grall wrote: So -- nothing boots directly by UEFI -- everything goes through GRUB. However, my understanding is that GRUB will detect devicetree information provided by UEFI (even though devicetree command is supposed to completely replace that). Hence it is possible that Linux relies on some residuals left in memory by GRUB that Xen doesn't pay attention to (but this is a pretty wild speculation only). While it goes through GRUB, it is a bootloader and will just act as a proxy for EFI. So EFI application such as Xen/Linux can still be loaded and take advantage of runtime servies if present/implemented. In fact most of people on Arm are using GRUB rather than EFI directly as this is more friendly to use. Regarding the devicetree, Xen and Linux will completely ignore the memory nodes in Xen if using EFI. This because the EFI memory map will give you an overview of the platform with the EFI regions included. Cheers, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
Hi,
On Wed, Dec 18, 2019 at 4:56 AM Julien Grall wrote:
> > So that is, in fact, my first question -- why is Xen not showing
> > available memory in xl info?
>
> I am not entirely sure what exact information you want.
>
> The output you dumped above contain the available memory for the memory
> (see "free_memory").
>
> Are you looking from something different?
Just to be clear: I was giving 2G via devicetrees (the same device
trees that would
make Linux detect 2G of RAM) hence I was expecting xl info to show that. Instead
I only got 1120M shown by xl info.
> On 18/12/2019 00:04, Roman Shaposhnik wrote:
> > memory {
> > device_type = "memory";
> > reg = <0x0 0x0 0x0 0x5e0 0x0 0x5f0 0x0 0x1000
> > 0x0 0x5f02000 0x0 0xefd000 0x0 0x6e0 0x0 0x60f000 0x0 0x741
> > 0x0 0x1aaf 0x0 0x21f0 0x0 0x10 0x0 0x2200 0x0
> > 0x1c00>;
> > };
> >
> > reserved-memory {
> > ranges;
> > #size-cells = <0x2>;
> > #address-cells = <0x2>;
> >
> > ramoops@21f0 {
> > ftrace-size = <0x2>;
> > console-size = <0x2>;
> > reg = <0x0 0x21f0 0x0 0x10>;
> > record-size = <0x2>;
> > compatible = "ramoops";
> > };
> >
> > linux,cma {
> > linux,cma-default;
> > reusable;
> > size = <0x0 0x800>;
> > compatible = "shared-dma-pool";
> > };
> > };
> >
> > If you look at the REG -- it does now add up to 2Gb, but booting Xen
> > with it has exactly the
> > same effect as booting it with: reg = <0x0 0x0 0x0 0x8000>;\
>
> If you boot Xen using EFI, the memory information wil come from EFI and
> the DT node will be ignored. So unless UEFI is able to pick up the
> modification of the DT memory node, modifying the DT is not going to
> affect anything.
That's a good point, but given that I always go through GRUB, I was
expecting devicetree command to completely overshadow whatever
information UEFI may have. Am I wrong?
> > I am attaching a full log, and I see the following in the logs:
> >
> > (XEN) Allocating 1:1 mappings totalling 720MB for dom0:
> > (XEN) BANK[0] 0x000800-0x001c00 (320MB)
> > (XEN) BANK[1] 0x004000-0x005800 (384MB)
> > (XEN) BANK[2] 0x007b00-0x007c00 (16MB)
> >
> > Which sort of makes sense, I guess -- but I still don't understand
> > where all these ranges
> > are coming from and how come Xen doesn't see the full 2Gb even with various
> > devicetrees I tried.
>
> The range aboves describe the memory range given to Dom0. For all the
> memory given to Xen,m you want to look at the top of your log:
>
> (XEN) Checking for initrd in /chosen
> (XEN) RAM: - 05df
> (XEN) RAM: 05f0 - 06dfefff
> (XEN) RAM: 06e0 - 0740efff
> (XEN) RAM: 0741 - 1db8dfff
> (XEN) RAM: 350f - 3dbd2fff
> (XEN) RAM: 3dbd3000 - 3dff
> (XEN) RAM: 4000 - 5a653fff
> (XEN) RAM: 7ada - 7ada3fff
> (XEN) RAM: 7aea8000 - 7afa9fff
> (XEN) RAM: 7afaa000 - 7ec73fff
> (XEN) RAM: 7ec74000 - 7fdddfff
> (XEN) RAM: 7fdde000 - 7fea5fff
> (XEN) RAM: 7fea6000 - 7ff6dfff
> (XEN) RAM: 7000 - 7fff
>
> Looking at the differences with the Linux logs, there is indeed some
> memory not detected by Xen.
>
> On Xen, we only consider usuable memory any EFI description with
> EfiConventionalMemory, EfiBootServicesCode and EfiBootServicesData.
>
> Linux include more type here, so this may explain why we see a difference.
>
> While Looking at it, I have also noticed that we don't seem to care
> about the memory attribute. I suspect this could be another latent issue
> in Xen if the attribute does not match.
Anything I can do to help debug this? I can run any kind of debug builds, etc.
if needed.
I mean -- at this point it would be really great to get HiKey back to the status
of Xen-on-ARM developer board.
> > Any ideas here would be greatly apprecaited!
> >
> > Thanks,
> > Roman.
> >
> > P.S. Any guess at what these mean?
> >
> > (XEN) traps.c:1973:d0v0 HSR=0x93880006 pc=0x008738
> > gva=0x872f2000 gpa=0x0f
> > (XEN) traps.c:1973:d0v0 HSR=0x93880006 pc=0x00b734e558
> > gva=0xb72eb000 gpa=0x0f
> > (XEN) traps.c:1973:d0v0 HSR=0x93880006 pc=0x008f9d2558
> > gva=0x8f96f000 gpa=0x0f
>
> It means that Linux has tried to access something that has not been
> mapped in stage-2. As Dom0 is mapped 1:1, the GPA also give you the host
> physical address.
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
On Wed, Dec 18, 2019 at 3:50 AM Julien Grall wrote: > > Hi, > > On 18/12/2019 07:36, Roman Shaposhnik wrote: > > On Tue, Dec 17, 2019 at 6:56 PM Roman Shaposhnik wrote: > >> Exactly! That's the other surprising bit -- I noticed that too -- its not > >> like > >> Xen doesn't see any of the memory above 1G -- it just doesn't see enough > >> of it. > >> > >> So the question is -- what is Linux doing that Xen doesn't? > > > > By the way, speaking of running Xen under ARM/qemu -- here's an interesting > > observation: when I run qemu-system-aarch64 with -m 4096 option it seems > > that, again, Linux kernel is perfectly content with having access to 4G of > > RAM, > > while Xen only sees about 2G. > > Linux and Xen should see close to the same amount as memory as long as > you are using the same bootloader... Thanks for confirming. This is what I'm trying to get to on this thread. Any help would be greatly appreciated! > > This may actually have something to do with UEFI I guess. > > ... could you confirm whether you are booting Linux using UEFI or not? The boot sequence in both cases is: HiKey l-loader HiKey Tianocore EDK2 – UEFI GRUB (as a UEFI payload) Xen | Linux GRUB's commands for booting Xen + Dom0: xen_hypervisor /boot/xen.efi console=dtuart dom0_mem=640M dom0_max_vcpus=1 dom0_vcpus_pin xen_module /boot/kernel console=hvc0 root=(hd1,gpt1)/rootfs.img text devicetree (hd1,gpt4)/eve.dtb xen_module (hd1,gpt1)/initrd.img GRUB's commands for booting Linux only: linux /boot/kernel console=ttyAMA0 console=ttyAMA1 console=ttyAMA2 console=ttyAMA3 root=PARTUUID=f71bd987-d99a-4c88-9781-cf4c26cae55e rootdelay=3 devicetree (hd1,gpt4)/eve.dtb So -- nothing boots directly by UEFI -- everything goes through GRUB. However, my understanding is that GRUB will detect devicetree information provided by UEFI (even though devicetree command is supposed to completely replace that). Hence it is possible that Linux relies on some residuals left in memory by GRUB that Xen doesn't pay attention to (but this is a pretty wild speculation only). Thanks, Roman. ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
On 18/12/2019 02:56, Roman Shaposhnik wrote: On Tue, Dec 17, 2019 at 5:51 PM Stefano Stabellini wrote: Interestingly enough, Xen booted, and complained about only 192MB unallocated this time. So, I dropped the size of Dom0 to 640M and I got it boot and here's what I'm seeing as an output of xl info: total_memory : 1120 free_memory: 390 It still nowhere close to 2G. Then I booted the Linux kernel without Xen and it correctly identified all 2G worth of RAM, and in fact, Good! We can work with that. So that is, in fact, my first question -- why is Xen not showing available memory in xl info? I am not entirely sure what exact information you want. The output you dumped above contain the available memory for the memory (see "free_memory"). Are you looking from something different? Cheers, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
Hi,
On 18/12/2019 00:04, Roman Shaposhnik wrote:
memory {
device_type = "memory";
reg = <0x0 0x0 0x0 0x5e0 0x0 0x5f0 0x0 0x1000
0x0 0x5f02000 0x0 0xefd000 0x0 0x6e0 0x0 0x60f000 0x0 0x741
0x0 0x1aaf 0x0 0x21f0 0x0 0x10 0x0 0x2200 0x0
0x1c00>;
};
reserved-memory {
ranges;
#size-cells = <0x2>;
#address-cells = <0x2>;
ramoops@21f0 {
ftrace-size = <0x2>;
console-size = <0x2>;
reg = <0x0 0x21f0 0x0 0x10>;
record-size = <0x2>;
compatible = "ramoops";
};
linux,cma {
linux,cma-default;
reusable;
size = <0x0 0x800>;
compatible = "shared-dma-pool";
};
};
If you look at the REG -- it does now add up to 2Gb, but booting Xen
with it has exactly the
same effect as booting it with: reg = <0x0 0x0 0x0 0x8000>;\
If you boot Xen using EFI, the memory information wil come from EFI and
the DT node will be ignored. So unless UEFI is able to pick up the
modification of the DT memory node, modifying the DT is not going to
affect anything.
I am attaching a full log, and I see the following in the logs:
(XEN) Allocating 1:1 mappings totalling 720MB for dom0:
(XEN) BANK[0] 0x000800-0x001c00 (320MB)
(XEN) BANK[1] 0x004000-0x005800 (384MB)
(XEN) BANK[2] 0x007b00-0x007c00 (16MB)
Which sort of makes sense, I guess -- but I still don't understand
where all these ranges
are coming from and how come Xen doesn't see the full 2Gb even with various
devicetrees I tried.
The range aboves describe the memory range given to Dom0. For all the
memory given to Xen,m you want to look at the top of your log:
(XEN) Checking for initrd in /chosen
(XEN) RAM: - 05df
(XEN) RAM: 05f0 - 06dfefff
(XEN) RAM: 06e0 - 0740efff
(XEN) RAM: 0741 - 1db8dfff
(XEN) RAM: 350f - 3dbd2fff
(XEN) RAM: 3dbd3000 - 3dff
(XEN) RAM: 4000 - 5a653fff
(XEN) RAM: 7ada - 7ada3fff
(XEN) RAM: 7aea8000 - 7afa9fff
(XEN) RAM: 7afaa000 - 7ec73fff
(XEN) RAM: 7ec74000 - 7fdddfff
(XEN) RAM: 7fdde000 - 7fea5fff
(XEN) RAM: 7fea6000 - 7ff6dfff
(XEN) RAM: 7000 - 7fff
Looking at the differences with the Linux logs, there is indeed some
memory not detected by Xen.
On Xen, we only consider usuable memory any EFI description with
EfiConventionalMemory, EfiBootServicesCode and EfiBootServicesData.
Linux include more type here, so this may explain why we see a difference.
While Looking at it, I have also noticed that we don't seem to care
about the memory attribute. I suspect this could be another latent issue
in Xen if the attribute does not match.
Any ideas here would be greatly apprecaited!
Thanks,
Roman.
P.S. Any guess at what these mean?
(XEN) traps.c:1973:d0v0 HSR=0x93880006 pc=0x008738
gva=0x872f2000 gpa=0x0f
(XEN) traps.c:1973:d0v0 HSR=0x93880006 pc=0x00b734e558
gva=0xb72eb000 gpa=0x0f
(XEN) traps.c:1973:d0v0 HSR=0x93880006 pc=0x008f9d2558
gva=0x8f96f000 gpa=0x0f
It means that Linux has tried to access something that has not been
mapped in stage-2. As Dom0 is mapped 1:1, the GPA also give you the host
physical address. In this case, it is trying to access 0xf.
This seems to belong to the RAM, but this part has not been allocated to
Dom0.
You may get more information from Dom0 if you add earlycon=xenboot on
your linux command line. This will give you some output using the
earlyconsole before the console subsytem is actually initialize.
Cheers,
--
Julien Grall
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
Hi, On 18/12/2019 07:36, Roman Shaposhnik wrote: On Tue, Dec 17, 2019 at 6:56 PM Roman Shaposhnik wrote: Exactly! That's the other surprising bit -- I noticed that too -- its not like Xen doesn't see any of the memory above 1G -- it just doesn't see enough of it. So the question is -- what is Linux doing that Xen doesn't? By the way, speaking of running Xen under ARM/qemu -- here's an interesting observation: when I run qemu-system-aarch64 with -m 4096 option it seems that, again, Linux kernel is perfectly content with having access to 4G of RAM, while Xen only sees about 2G. Linux and Xen should see close to the same amount as memory as long as you are using the same bootloader... This may actually have something to do with UEFI I guess. ... could you confirm whether you are booting Linux using UEFI or not? Cheers, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
On Tue, Dec 17, 2019 at 6:56 PM Roman Shaposhnik wrote:
>
> On Tue, Dec 17, 2019 at 5:51 PM Stefano Stabellini
> wrote:
> >
> > On Tue, 17 Dec 2019, Roman Shaposhnik wrote:
> > > On Tue, Dec 17, 2019 at 11:26 AM Stefano Stabellini
> > > wrote:
> > > >
> > > > On Tue, 17 Dec 2019, Roman Shaposhnik wrote:
> > > > > On Tue, Dec 17, 2019 at 10:30 AM Stefano Stabellini
> > > > > wrote:
> > > > > >
> > > > > > On Tue, 17 Dec 2019, Julien Grall wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > On 17/12/2019 04:39, Roman Shaposhnik wrote:
> > > > > > > > On Mon, Dec 16, 2019 at 6:55 PM Stefano Stabellini
> > > > > > > > wrote:
> > > > > > > > > On Mon, 16 Dec 2019, Roman Shaposhnik wrote:
> > > > > > > > > If I sum all the memory sizes together I get 0x3ddfd000 which
> > > > > > > > > is 990M.
> > > > > > > > > If so, I wonder how you could boot succesfully with
> > > > > > > > > dom0_mem=1024M even
> > > > > > > > > on Xen 4.12... :-?
> > > > > > > >
> > > > > > > > That is a very interesting observation indeed! I actually don't
> > > > > > > > remember where that device tree came from, but I think it was
> > > > > > > > from one
> > > > > > > > of the Linaro sites.
> > > > > > >
> > > > > > > This is mostly likely because of:
> > > > > > >
> > > > > > > commit 6341a674573f1834f083f0ab0f5b36b075f9e02e
> > > > > > > Author: Julien Grall
> > > > > > > Date: Wed Aug 21 22:42:31 2019 +0100
> > > > > > >
> > > > > > > xen/arm: domain_build: Don't continue if unable to allocate
> > > > > > > all dom0 banks
> > > > > > >
> > > > > > > Xen will only print a warning if there are memory unallocated
> > > > > > > when using
> > > > > > > 1:1 mapping (only used by dom0). This also includes the case
> > > > > > > where no
> > > > > > > memory has been allocated.
> > > > > > >
> > > > > > > It will bring to all sort of issues that can be hard to
> > > > > > > diagnostic for
> > > > > > > users (the warning can be difficult to spot or disregard).
> > > > > > >
> > > > > > > If the users request 1GB of memory, then most likely they
> > > > > > > want the exact
> > > > > > > amount and not 512MB. So panic if all the memory has not been
> > > > > > > allocated.
> > > > > > >
> > > > > > > After this change, the behavior is the same as for non-1:1
> > > > > > > memory
> > > > > > > allocation (used by domU).
> > > > > > >
> > > > > > > At the same time, reflow the message to have the format on a
> > > > > > > single
> > > > > > > line.
> > > > > > >
> > > > > > > Signed-off-by: Julien Grall
> > > > > > > Acked-by: Stefano Stabellini
> > > > > >
> > > > > > Ah! Roman, could you please post the full boot log of a successful
> > > > > > 4.12
> > > > > > boot?
> > > > > >
> > > > > > If it has a "Failed to allocate requested dom0 memory" message,
> > > > > > then we
> > > > > > know what the issue is.
> > > > >
> > > > > Aha! Our messages seems to have crossed ;-) Full log is attached and
> > > > > yes -- that's
> > > > > the problem indeed.
> > > > >
> > > > > So at least that mystery is solved. But I'm still not able to get to a
> > > > > full 1G of memory
> > > > > even with your update to the device tree file. Any chance you can
> > > > > send me the
> > > > > device tree file that works for you?
> > > >
> > > > I didn't try on real hardware, I only tried on QEMU with a similar
> > > > configuration. I went back and check the HiKey device tree I used and it
> > > > is the same as yours (including the ramoops reserved-memory error).
> > > >
> > > > Apparently there are 1G and 2G variants of the HiKey, obviously both
> > > > yours and my device tree are for the 1G variant. I try to dig through
> > > > the docs but couldn't find the details of the 2G variant. I cannot find
> > > > anywhere the memory range for the top 1G of memory not even on the
> > > > LeMaker docs! :-/
> > >
> > > Yup. That's exactly the issue on my end as well - can't seem to find an
> > > authoritative source for that devicetree.
> > >
> > > I did find this, though:
> > > https://releases.linaro.org/96boards/hikey/linaro/debian/15.11/
> > > which looks like it has the latest (at least file timestamp-wise)
> > > devicetree.
> > >
> > > If you look at the memory and reserved memory nodes there, they
> > > are actually much simpler than what we've got:
> > >
> > > memory {
> > > device_type = "memory";
> > > reg = <0x0 0x0 0x0 0x4000>;
> > > };
> >
> > Which is still 1G, but it is surprisingly simpler.
> >
> >
> > > reserved-memory {
> > > #address-cells = <0x2>;
> > > #size-cells = <0x2>;
> > > ranges;
> > >
> > > mcu-buf@05e0 {
> > > no-map;
> > > reg = <0x0 0x5e0 0x0 0x10 0x0
> > > 0x740f000 0x0 0x1000>;
> > > };
> > >
> > > mbox-buf@06dff000 {
> > >
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
On Tue, Dec 17, 2019 at 5:51 PM Stefano Stabellini
wrote:
>
> On Tue, 17 Dec 2019, Roman Shaposhnik wrote:
> > On Tue, Dec 17, 2019 at 11:26 AM Stefano Stabellini
> > wrote:
> > >
> > > On Tue, 17 Dec 2019, Roman Shaposhnik wrote:
> > > > On Tue, Dec 17, 2019 at 10:30 AM Stefano Stabellini
> > > > wrote:
> > > > >
> > > > > On Tue, 17 Dec 2019, Julien Grall wrote:
> > > > > > Hi,
> > > > > >
> > > > > > On 17/12/2019 04:39, Roman Shaposhnik wrote:
> > > > > > > On Mon, Dec 16, 2019 at 6:55 PM Stefano Stabellini
> > > > > > > wrote:
> > > > > > > > On Mon, 16 Dec 2019, Roman Shaposhnik wrote:
> > > > > > > > If I sum all the memory sizes together I get 0x3ddfd000 which
> > > > > > > > is 990M.
> > > > > > > > If so, I wonder how you could boot succesfully with
> > > > > > > > dom0_mem=1024M even
> > > > > > > > on Xen 4.12... :-?
> > > > > > >
> > > > > > > That is a very interesting observation indeed! I actually don't
> > > > > > > remember where that device tree came from, but I think it was
> > > > > > > from one
> > > > > > > of the Linaro sites.
> > > > > >
> > > > > > This is mostly likely because of:
> > > > > >
> > > > > > commit 6341a674573f1834f083f0ab0f5b36b075f9e02e
> > > > > > Author: Julien Grall
> > > > > > Date: Wed Aug 21 22:42:31 2019 +0100
> > > > > >
> > > > > > xen/arm: domain_build: Don't continue if unable to allocate all
> > > > > > dom0 banks
> > > > > >
> > > > > > Xen will only print a warning if there are memory unallocated
> > > > > > when using
> > > > > > 1:1 mapping (only used by dom0). This also includes the case
> > > > > > where no
> > > > > > memory has been allocated.
> > > > > >
> > > > > > It will bring to all sort of issues that can be hard to
> > > > > > diagnostic for
> > > > > > users (the warning can be difficult to spot or disregard).
> > > > > >
> > > > > > If the users request 1GB of memory, then most likely they want
> > > > > > the exact
> > > > > > amount and not 512MB. So panic if all the memory has not been
> > > > > > allocated.
> > > > > >
> > > > > > After this change, the behavior is the same as for non-1:1
> > > > > > memory
> > > > > > allocation (used by domU).
> > > > > >
> > > > > > At the same time, reflow the message to have the format on a
> > > > > > single
> > > > > > line.
> > > > > >
> > > > > > Signed-off-by: Julien Grall
> > > > > > Acked-by: Stefano Stabellini
> > > > >
> > > > > Ah! Roman, could you please post the full boot log of a successful
> > > > > 4.12
> > > > > boot?
> > > > >
> > > > > If it has a "Failed to allocate requested dom0 memory" message, then
> > > > > we
> > > > > know what the issue is.
> > > >
> > > > Aha! Our messages seems to have crossed ;-) Full log is attached and
> > > > yes -- that's
> > > > the problem indeed.
> > > >
> > > > So at least that mystery is solved. But I'm still not able to get to a
> > > > full 1G of memory
> > > > even with your update to the device tree file. Any chance you can send
> > > > me the
> > > > device tree file that works for you?
> > >
> > > I didn't try on real hardware, I only tried on QEMU with a similar
> > > configuration. I went back and check the HiKey device tree I used and it
> > > is the same as yours (including the ramoops reserved-memory error).
> > >
> > > Apparently there are 1G and 2G variants of the HiKey, obviously both
> > > yours and my device tree are for the 1G variant. I try to dig through
> > > the docs but couldn't find the details of the 2G variant. I cannot find
> > > anywhere the memory range for the top 1G of memory not even on the
> > > LeMaker docs! :-/
> >
> > Yup. That's exactly the issue on my end as well - can't seem to find an
> > authoritative source for that devicetree.
> >
> > I did find this, though:
> > https://releases.linaro.org/96boards/hikey/linaro/debian/15.11/
> > which looks like it has the latest (at least file timestamp-wise)
> > devicetree.
> >
> > If you look at the memory and reserved memory nodes there, they
> > are actually much simpler than what we've got:
> >
> > memory {
> > device_type = "memory";
> > reg = <0x0 0x0 0x0 0x4000>;
> > };
>
> Which is still 1G, but it is surprisingly simpler.
>
>
> > reserved-memory {
> > #address-cells = <0x2>;
> > #size-cells = <0x2>;
> > ranges;
> >
> > mcu-buf@05e0 {
> > no-map;
> > reg = <0x0 0x5e0 0x0 0x10 0x0
> > 0x740f000 0x0 0x1000>;
> > };
> >
> > mbox-buf@06dff000 {
> > no-map;
> > reg = <0x0 0x6dff000 0x0 0x1000>;
> > };
> > };
> >
> > So -- just on a whim -- I changed it to:
> > reg = <0x0 0x0 0x0 0x8000>;
>
> I would have tried that too :-)
>
>
> > Interestingly enough, Xen booted, and complained about
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
On Tue, 17 Dec 2019, Roman Shaposhnik wrote:
> On Tue, Dec 17, 2019 at 11:26 AM Stefano Stabellini
> wrote:
> >
> > On Tue, 17 Dec 2019, Roman Shaposhnik wrote:
> > > On Tue, Dec 17, 2019 at 10:30 AM Stefano Stabellini
> > > wrote:
> > > >
> > > > On Tue, 17 Dec 2019, Julien Grall wrote:
> > > > > Hi,
> > > > >
> > > > > On 17/12/2019 04:39, Roman Shaposhnik wrote:
> > > > > > On Mon, Dec 16, 2019 at 6:55 PM Stefano Stabellini
> > > > > > wrote:
> > > > > > > On Mon, 16 Dec 2019, Roman Shaposhnik wrote:
> > > > > > > If I sum all the memory sizes together I get 0x3ddfd000 which is
> > > > > > > 990M.
> > > > > > > If so, I wonder how you could boot succesfully with
> > > > > > > dom0_mem=1024M even
> > > > > > > on Xen 4.12... :-?
> > > > > >
> > > > > > That is a very interesting observation indeed! I actually don't
> > > > > > remember where that device tree came from, but I think it was from
> > > > > > one
> > > > > > of the Linaro sites.
> > > > >
> > > > > This is mostly likely because of:
> > > > >
> > > > > commit 6341a674573f1834f083f0ab0f5b36b075f9e02e
> > > > > Author: Julien Grall
> > > > > Date: Wed Aug 21 22:42:31 2019 +0100
> > > > >
> > > > > xen/arm: domain_build: Don't continue if unable to allocate all
> > > > > dom0 banks
> > > > >
> > > > > Xen will only print a warning if there are memory unallocated
> > > > > when using
> > > > > 1:1 mapping (only used by dom0). This also includes the case
> > > > > where no
> > > > > memory has been allocated.
> > > > >
> > > > > It will bring to all sort of issues that can be hard to
> > > > > diagnostic for
> > > > > users (the warning can be difficult to spot or disregard).
> > > > >
> > > > > If the users request 1GB of memory, then most likely they want
> > > > > the exact
> > > > > amount and not 512MB. So panic if all the memory has not been
> > > > > allocated.
> > > > >
> > > > > After this change, the behavior is the same as for non-1:1 memory
> > > > > allocation (used by domU).
> > > > >
> > > > > At the same time, reflow the message to have the format on a
> > > > > single
> > > > > line.
> > > > >
> > > > > Signed-off-by: Julien Grall
> > > > > Acked-by: Stefano Stabellini
> > > >
> > > > Ah! Roman, could you please post the full boot log of a successful 4.12
> > > > boot?
> > > >
> > > > If it has a "Failed to allocate requested dom0 memory" message, then we
> > > > know what the issue is.
> > >
> > > Aha! Our messages seems to have crossed ;-) Full log is attached and
> > > yes -- that's
> > > the problem indeed.
> > >
> > > So at least that mystery is solved. But I'm still not able to get to a
> > > full 1G of memory
> > > even with your update to the device tree file. Any chance you can send me
> > > the
> > > device tree file that works for you?
> >
> > I didn't try on real hardware, I only tried on QEMU with a similar
> > configuration. I went back and check the HiKey device tree I used and it
> > is the same as yours (including the ramoops reserved-memory error).
> >
> > Apparently there are 1G and 2G variants of the HiKey, obviously both
> > yours and my device tree are for the 1G variant. I try to dig through
> > the docs but couldn't find the details of the 2G variant. I cannot find
> > anywhere the memory range for the top 1G of memory not even on the
> > LeMaker docs! :-/
>
> Yup. That's exactly the issue on my end as well - can't seem to find an
> authoritative source for that devicetree.
>
> I did find this, though:
> https://releases.linaro.org/96boards/hikey/linaro/debian/15.11/
> which looks like it has the latest (at least file timestamp-wise) devicetree.
>
> If you look at the memory and reserved memory nodes there, they
> are actually much simpler than what we've got:
>
> memory {
> device_type = "memory";
> reg = <0x0 0x0 0x0 0x4000>;
> };
Which is still 1G, but it is surprisingly simpler.
> reserved-memory {
> #address-cells = <0x2>;
> #size-cells = <0x2>;
> ranges;
>
> mcu-buf@05e0 {
> no-map;
> reg = <0x0 0x5e0 0x0 0x10 0x0
> 0x740f000 0x0 0x1000>;
> };
>
> mbox-buf@06dff000 {
> no-map;
> reg = <0x0 0x6dff000 0x0 0x1000>;
> };
> };
>
> So -- just on a whim -- I changed it to:
> reg = <0x0 0x0 0x0 0x8000>;
I would have tried that too :-)
> Interestingly enough, Xen booted, and complained about only 192MB
> unallocated this time.
> So, I dropped the size of Dom0 to 640M and I got it boot and here's
> what I'm seeing as
> an output of xl info:
>total_memory : 1120
>free_memory: 390
> It still nowhere close to 2G.
>
> Then I booted the Linux kernel without Xen and it correctly identified
> all 2G worth
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
On Tue, Dec 17, 2019 at 11:26 AM Stefano Stabellini
wrote:
>
> On Tue, 17 Dec 2019, Roman Shaposhnik wrote:
> > On Tue, Dec 17, 2019 at 10:30 AM Stefano Stabellini
> > wrote:
> > >
> > > On Tue, 17 Dec 2019, Julien Grall wrote:
> > > > Hi,
> > > >
> > > > On 17/12/2019 04:39, Roman Shaposhnik wrote:
> > > > > On Mon, Dec 16, 2019 at 6:55 PM Stefano Stabellini
> > > > > wrote:
> > > > > > On Mon, 16 Dec 2019, Roman Shaposhnik wrote:
> > > > > > If I sum all the memory sizes together I get 0x3ddfd000 which is
> > > > > > 990M.
> > > > > > If so, I wonder how you could boot succesfully with dom0_mem=1024M
> > > > > > even
> > > > > > on Xen 4.12... :-?
> > > > >
> > > > > That is a very interesting observation indeed! I actually don't
> > > > > remember where that device tree came from, but I think it was from one
> > > > > of the Linaro sites.
> > > >
> > > > This is mostly likely because of:
> > > >
> > > > commit 6341a674573f1834f083f0ab0f5b36b075f9e02e
> > > > Author: Julien Grall
> > > > Date: Wed Aug 21 22:42:31 2019 +0100
> > > >
> > > > xen/arm: domain_build: Don't continue if unable to allocate all
> > > > dom0 banks
> > > >
> > > > Xen will only print a warning if there are memory unallocated when
> > > > using
> > > > 1:1 mapping (only used by dom0). This also includes the case where
> > > > no
> > > > memory has been allocated.
> > > >
> > > > It will bring to all sort of issues that can be hard to diagnostic
> > > > for
> > > > users (the warning can be difficult to spot or disregard).
> > > >
> > > > If the users request 1GB of memory, then most likely they want the
> > > > exact
> > > > amount and not 512MB. So panic if all the memory has not been
> > > > allocated.
> > > >
> > > > After this change, the behavior is the same as for non-1:1 memory
> > > > allocation (used by domU).
> > > >
> > > > At the same time, reflow the message to have the format on a single
> > > > line.
> > > >
> > > > Signed-off-by: Julien Grall
> > > > Acked-by: Stefano Stabellini
> > >
> > > Ah! Roman, could you please post the full boot log of a successful 4.12
> > > boot?
> > >
> > > If it has a "Failed to allocate requested dom0 memory" message, then we
> > > know what the issue is.
> >
> > Aha! Our messages seems to have crossed ;-) Full log is attached and
> > yes -- that's
> > the problem indeed.
> >
> > So at least that mystery is solved. But I'm still not able to get to a
> > full 1G of memory
> > even with your update to the device tree file. Any chance you can send me
> > the
> > device tree file that works for you?
>
> I didn't try on real hardware, I only tried on QEMU with a similar
> configuration. I went back and check the HiKey device tree I used and it
> is the same as yours (including the ramoops reserved-memory error).
>
> Apparently there are 1G and 2G variants of the HiKey, obviously both
> yours and my device tree are for the 1G variant. I try to dig through
> the docs but couldn't find the details of the 2G variant. I cannot find
> anywhere the memory range for the top 1G of memory not even on the
> LeMaker docs! :-/
Yup. That's exactly the issue on my end as well - can't seem to find an
authoritative source for that devicetree.
I did find this, though:
https://releases.linaro.org/96boards/hikey/linaro/debian/15.11/
which looks like it has the latest (at least file timestamp-wise) devicetree.
If you look at the memory and reserved memory nodes there, they
are actually much simpler than what we've got:
memory {
device_type = "memory";
reg = <0x0 0x0 0x0 0x4000>;
};
reserved-memory {
#address-cells = <0x2>;
#size-cells = <0x2>;
ranges;
mcu-buf@05e0 {
no-map;
reg = <0x0 0x5e0 0x0 0x10 0x0
0x740f000 0x0 0x1000>;
};
mbox-buf@06dff000 {
no-map;
reg = <0x0 0x6dff000 0x0 0x1000>;
};
};
So -- just on a whim -- I changed it to:
reg = <0x0 0x0 0x0 0x8000>;
Interestingly enough, Xen booted, and complained about only 192MB
unallocated this time.
So, I dropped the size of Dom0 to 640M and I got it boot and here's
what I'm seeing as
an output of xl info:
total_memory : 1120
free_memory: 390
It still nowhere close to 2G.
Then I booted the Linux kernel without Xen and it correctly identified
all 2G worth of RAM, and in fact,
when I converted /sys/firmware/devicetree/base back into dts, here's
what I've got:
memory {
device_type = "memory";
reg = <0x0 0x0 0x0 0x5e0 0x0 0x5f0 0x0 0x1000
0x0 0x5f02000 0x0 0xefd000 0x0 0x6e0 0x0 0x60f000 0x0 0x741
0x0 0x1aaf 0x0 0x21f0 0x0 0x10 0x0 0x2200 0x0
0x1c00>;
};
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
On Tue, 17 Dec 2019, Roman Shaposhnik wrote: > On Tue, Dec 17, 2019 at 10:30 AM Stefano Stabellini > wrote: > > > > On Tue, 17 Dec 2019, Julien Grall wrote: > > > Hi, > > > > > > On 17/12/2019 04:39, Roman Shaposhnik wrote: > > > > On Mon, Dec 16, 2019 at 6:55 PM Stefano Stabellini > > > > wrote: > > > > > On Mon, 16 Dec 2019, Roman Shaposhnik wrote: > > > > > If I sum all the memory sizes together I get 0x3ddfd000 which is 990M. > > > > > If so, I wonder how you could boot succesfully with dom0_mem=1024M > > > > > even > > > > > on Xen 4.12... :-? > > > > > > > > That is a very interesting observation indeed! I actually don't > > > > remember where that device tree came from, but I think it was from one > > > > of the Linaro sites. > > > > > > This is mostly likely because of: > > > > > > commit 6341a674573f1834f083f0ab0f5b36b075f9e02e > > > Author: Julien Grall > > > Date: Wed Aug 21 22:42:31 2019 +0100 > > > > > > xen/arm: domain_build: Don't continue if unable to allocate all dom0 > > > banks > > > > > > Xen will only print a warning if there are memory unallocated when > > > using > > > 1:1 mapping (only used by dom0). This also includes the case where no > > > memory has been allocated. > > > > > > It will bring to all sort of issues that can be hard to diagnostic for > > > users (the warning can be difficult to spot or disregard). > > > > > > If the users request 1GB of memory, then most likely they want the > > > exact > > > amount and not 512MB. So panic if all the memory has not been > > > allocated. > > > > > > After this change, the behavior is the same as for non-1:1 memory > > > allocation (used by domU). > > > > > > At the same time, reflow the message to have the format on a single > > > line. > > > > > > Signed-off-by: Julien Grall > > > Acked-by: Stefano Stabellini > > > > Ah! Roman, could you please post the full boot log of a successful 4.12 > > boot? > > > > If it has a "Failed to allocate requested dom0 memory" message, then we > > know what the issue is. > > Aha! Our messages seems to have crossed ;-) Full log is attached and > yes -- that's > the problem indeed. > > So at least that mystery is solved. But I'm still not able to get to a > full 1G of memory > even with your update to the device tree file. Any chance you can send me the > device tree file that works for you? I didn't try on real hardware, I only tried on QEMU with a similar configuration. I went back and check the HiKey device tree I used and it is the same as yours (including the ramoops reserved-memory error). Apparently there are 1G and 2G variants of the HiKey, obviously both yours and my device tree are for the 1G variant. I try to dig through the docs but couldn't find the details of the 2G variant. I cannot find anywhere the memory range for the top 1G of memory not even on the LeMaker docs! :-/ ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
On Tue, Dec 17, 2019 at 10:30 AM Stefano Stabellini wrote: > > On Tue, 17 Dec 2019, Julien Grall wrote: > > Hi, > > > > On 17/12/2019 04:39, Roman Shaposhnik wrote: > > > On Mon, Dec 16, 2019 at 6:55 PM Stefano Stabellini > > > wrote: > > > > On Mon, 16 Dec 2019, Roman Shaposhnik wrote: > > > > If I sum all the memory sizes together I get 0x3ddfd000 which is 990M. > > > > If so, I wonder how you could boot succesfully with dom0_mem=1024M even > > > > on Xen 4.12... :-? > > > > > > That is a very interesting observation indeed! I actually don't > > > remember where that device tree came from, but I think it was from one > > > of the Linaro sites. > > > > This is mostly likely because of: > > > > commit 6341a674573f1834f083f0ab0f5b36b075f9e02e > > Author: Julien Grall > > Date: Wed Aug 21 22:42:31 2019 +0100 > > > > xen/arm: domain_build: Don't continue if unable to allocate all dom0 > > banks > > > > Xen will only print a warning if there are memory unallocated when using > > 1:1 mapping (only used by dom0). This also includes the case where no > > memory has been allocated. > > > > It will bring to all sort of issues that can be hard to diagnostic for > > users (the warning can be difficult to spot or disregard). > > > > If the users request 1GB of memory, then most likely they want the exact > > amount and not 512MB. So panic if all the memory has not been allocated. > > > > After this change, the behavior is the same as for non-1:1 memory > > allocation (used by domU). > > > > At the same time, reflow the message to have the format on a single > > line. > > > > Signed-off-by: Julien Grall > > Acked-by: Stefano Stabellini > > Ah! Roman, could you please post the full boot log of a successful 4.12 > boot? > > If it has a "Failed to allocate requested dom0 memory" message, then we > know what the issue is. Aha! Our messages seems to have crossed ;-) Full log is attached and yes -- that's the problem indeed. So at least that mystery is solved. But I'm still not able to get to a full 1G of memory even with your update to the device tree file. Any chance you can send me the device tree file that works for you? Thanks, Roman. Using modules provided by bootloader in FDT Xen 4.12.0 (c/s ) EFI loader Xen 4.12.0 (XEN) Xen version 4.12.0 (@) (gcc (Alpine 6.4.0) 6.4.0) debug=n Fri Jun 7 17:32:08 UTC 2019 (XEN) Latest ChangeSet: (XEN) Processor: 410fd033: "ARM Limited", variant: 0x0, part 0xd03, rev 0x3 (XEN) 64-bit Execution: (XEN) Processor Features: (XEN) Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32 (XEN) Extensions: FloatingPoint AdvancedSIMD (XEN) Debug Features: 10305106 (XEN) Auxiliary Features: (XEN) Memory Model Features: 1122 (XEN) ISA Features: 00011120 (XEN) 32-bit Execution: (XEN) Processor Features: 0131:00011011 (XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle (XEN) Extensions: GenericTimer Security (XEN) Debug Features: 03010066 (XEN) Auxiliary Features: (XEN) Memory Model Features: 10101105 4000 0126 02102211 (XEN) ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121 (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 1200 KHz (XEN) GICv2 initialization: (XEN) gic_dist_addr=f6801000 (XEN) gic_cpu_addr=f6802000 (XEN) gic_hyp_addr=f6804000 (XEN) gic_vcpu_addr=f6806000 (XEN) gic_maintenance_irq=25 (XEN) GICv2: 160 lines, 8 cpus, secure (IID 0200143b). (XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2) (XEN) Initializing Credit2 scheduler (XEN) Allocated console ring of 16 KiB. (XEN) Bringing up CPU1 (XEN) Bringing up CPU2 (XEN) Bringing up CPU3 (XEN) Bringing up CPU4 (XEN) Bringing up CPU5 (XEN) Bringing up CPU6 (XEN) Bringing up CPU7 (XEN) Brought up 8 CPUs (XEN) P2M: 40-bit IPA with 40-bit PA and 8-bit VMID (XEN) P2M: 3 levels with order-1 root, VTCR 0x80023558 (XEN) I/O virtualisation disabled (XEN) *** LOADING DOMAIN 0 *** (XEN) Loading Domd0 kernel from boot module @ 48d38000 (XEN) Loading ramdisk from boot module @ 47aa6000 (XEN) Allocating 1:1 mappings totalling 1024MB for dom0: (XEN) WARNING: Failed to allocate requested dom0 memory. 624MB unallocated (XEN) BANK[0] 0x000800-0x001000 (128MB) (XEN) BANK[1] 0x003600-0x003e00 (128MB) (XEN) BANK[2] 0x004000-0x004700 (112MB) (XEN) BANK[3] 0x007b00-0x007c00 (16MB) (XEN) BANK[4] 0x007e00-0x007f00 (16MB) (XEN) Grant table range: 0x0047998000-0x00479d8000 (XEN) Allocating PPI 16 for event channel interrupt (XEN) Loading zImage from 48d38000 to 0808-09233200 (XEN) Loading dom0 initrd from 47aa6000 to
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
Hi Julien, On Tue, Dec 17, 2019 at 3:30 AM Julien Grall wrote: > > Hi, > > On 17/12/2019 04:39, Roman Shaposhnik wrote: > > On Mon, Dec 16, 2019 at 6:55 PM Stefano Stabellini > > wrote: > >> On Mon, 16 Dec 2019, Roman Shaposhnik wrote: > >> If I sum all the memory sizes together I get 0x3ddfd000 which is 990M. > >> If so, I wonder how you could boot succesfully with dom0_mem=1024M even > >> on Xen 4.12... :-? > > > > That is a very interesting observation indeed! I actually don't > > remember where that device tree came from, but I think it was from one > > of the Linaro sites. > > This is mostly likely because of: > > commit 6341a674573f1834f083f0ab0f5b36b075f9e02e > Author: Julien Grall > Date: Wed Aug 21 22:42:31 2019 +0100 > > xen/arm: domain_build: Don't continue if unable to allocate all > dom0 banks > > Xen will only print a warning if there are memory unallocated when > using > 1:1 mapping (only used by dom0). This also includes the case where no > memory has been allocated. > > It will bring to all sort of issues that can be hard to diagnostic for > users (the warning can be difficult to spot or disregard). > > If the users request 1GB of memory, then most likely they want the > exact > amount and not 512MB. So panic if all the memory has not been > allocated. > > After this change, the behavior is the same as for non-1:1 memory > allocation (used by domU). > > At the same time, reflow the message to have the format on a single > line. > > Signed-off-by: Julien Grall > Acked-by: Stefano Stabellini It seems you're absolutely right. Looking at the logs from Xen 4.12 I'm seeing: (XEN) Allocating 1:1 mappings totalling 1024MB for dom0: (XEN) WARNING: Failed to allocate requested dom0 memory. 624MB unallocated (XEN) BANK[0] 0x000800-0x001000 (128MB) (XEN) BANK[1] 0x003600-0x003e00 (128MB) (XEN) BANK[2] 0x004000-0x004700 (112MB) (XEN) BANK[3] 0x007b00-0x007c00 (16MB) (XEN) BANK[4] 0x007e00-0x007f00 (16MB) (XEN) Grant table range: 0x0047998000-0x00479d8000 (XEN) Allocating PPI 16 for event channel interrupt So yes -- it was a warning that now turned an ERROR. So at least that part is clear now. What isn't clear still is the interplay between device trees and Xen memory allocation -- I'll reply to Stefano on that. Thanks, Roman. ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
On Tue, 17 Dec 2019, Julien Grall wrote: > Hi, > > On 17/12/2019 04:39, Roman Shaposhnik wrote: > > On Mon, Dec 16, 2019 at 6:55 PM Stefano Stabellini > > wrote: > > > On Mon, 16 Dec 2019, Roman Shaposhnik wrote: > > > If I sum all the memory sizes together I get 0x3ddfd000 which is 990M. > > > If so, I wonder how you could boot succesfully with dom0_mem=1024M even > > > on Xen 4.12... :-? > > > > That is a very interesting observation indeed! I actually don't > > remember where that device tree came from, but I think it was from one > > of the Linaro sites. > > This is mostly likely because of: > > commit 6341a674573f1834f083f0ab0f5b36b075f9e02e > Author: Julien Grall > Date: Wed Aug 21 22:42:31 2019 +0100 > > xen/arm: domain_build: Don't continue if unable to allocate all dom0 banks > > Xen will only print a warning if there are memory unallocated when using > 1:1 mapping (only used by dom0). This also includes the case where no > memory has been allocated. > > It will bring to all sort of issues that can be hard to diagnostic for > users (the warning can be difficult to spot or disregard). > > If the users request 1GB of memory, then most likely they want the exact > amount and not 512MB. So panic if all the memory has not been allocated. > > After this change, the behavior is the same as for non-1:1 memory > allocation (used by domU). > > At the same time, reflow the message to have the format on a single > line. > > Signed-off-by: Julien Grall > Acked-by: Stefano Stabellini Ah! Roman, could you please post the full boot log of a successful 4.12 boot? If it has a "Failed to allocate requested dom0 memory" message, then we know what the issue is. ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
Hi, On 17/12/2019 04:39, Roman Shaposhnik wrote: On Mon, Dec 16, 2019 at 6:55 PM Stefano Stabellini wrote: On Mon, 16 Dec 2019, Roman Shaposhnik wrote: If I sum all the memory sizes together I get 0x3ddfd000 which is 990M. If so, I wonder how you could boot succesfully with dom0_mem=1024M even on Xen 4.12... :-? That is a very interesting observation indeed! I actually don't remember where that device tree came from, but I think it was from one of the Linaro sites. This is mostly likely because of: commit 6341a674573f1834f083f0ab0f5b36b075f9e02e Author: Julien Grall Date: Wed Aug 21 22:42:31 2019 +0100 xen/arm: domain_build: Don't continue if unable to allocate all dom0 banks Xen will only print a warning if there are memory unallocated when using 1:1 mapping (only used by dom0). This also includes the case where no memory has been allocated. It will bring to all sort of issues that can be hard to diagnostic for users (the warning can be difficult to spot or disregard). If the users request 1GB of memory, then most likely they want the exact amount and not 512MB. So panic if all the memory has not been allocated. After this change, the behavior is the same as for non-1:1 memory allocation (used by domU). At the same time, reflow the message to have the format on a single line. Signed-off-by: Julien Grall Acked-by: Stefano Stabellini Cheers, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
On Mon, Dec 16, 2019 at 6:55 PM Stefano Stabellini
wrote:
>
> On Mon, 16 Dec 2019, Roman Shaposhnik wrote:
> > Hi!
> >
> > it appears that something has broken in 4.13 RC5 so that
> > I'm now getting the following on ARM (full logs are attached).
> >
> > (XEN)
> > (XEN) Panic on CPU 0:
> > (XEN) Failed to allocate requested dom0 memory. 672MB unallocated
> > (XEN)
> >
> > My GRUB boot sequence hasn't changed:
> >
> > xen_hypervisor /boot/xen.efi console=dtuart dom0_mem=1024M,max:1024M
> > dom0_max_vcpus=1 dom0_vcpus_pin
>
> FYI we don't actually support the ",max:1024M" part of the dom0_mem
> argument on ARM. On ARM, it should just be:
>
> dom0_mem=1024M
I tried with just dom0_mem=1024M and it is the same result -- can't boot.
> > xen_module /boot/kernel console=hvc0 root=(hd1,gpt1)/rootfs.img text
> > devicetree (hd1,gpt4)/eve.dtb
> > xen_module (hd1,gpt1)/initrd.img
> >
> > In fact, if I use Xen 4.12 instead of 4.13 -- everything seems to work
> > as it used to.
>
> I spoke too early: I am unable to reproduce it on my end. On what
> platforms did you see this error? Was it the HiKey?
Yup. It is HiKey.
> Could you please
> post the devicetree that you are using (eve.dtb from this example)?
Looks like you've found it ;-) Btw, can you please send me the device
tree with which you're booting fine? I can try that very quickly.
> If the dts is this one:
> https://github.com/lf-edge/eve/blob/master/conf/eve.dts
> then I might have an idea.
Yup. That's the one.
> In Xen 4.13 we introduced support for
> reserved-memory. It looks like the HiKey has a few reserved-memory
> regions and I wonder if that is the problem. Could you please remove the
> whole "reserved-memory" node and try again to see if that is the issue?
Will do tomorrow (but I'd also appreciate getting your devicetree).
> Also looking at eve.dts above, I am a bit puzzled because the memory
> node is:
>
> memory@0 {
> device_type = "memory";
> reg = <0x0 0x0 0x0 0x5e0 0x0 0x5f0 0x0 0x1000 0x0
> 0x5f02000 0x0 0xefd000 0x0 0x6e0 0x0 0x60f000 0x0 0x741 0x0
> 0x1aaf 0x0 0x2200 0x0 0x1c00>;
> };
>
> If I sum all the memory sizes together I get 0x3ddfd000 which is 990M.
> If so, I wonder how you could boot succesfully with dom0_mem=1024M even
> on Xen 4.12... :-?
That is a very interesting observation indeed! I actually don't
remember where that device tree came from, but I think it was from one
of the Linaro sites.
> If we look at ramoops@21f0 under reserved-memory:
>
> ramoops@21f0 {
> record-size = <0x2>;
> compatible = "ramoops";
> console-size = <0x2>;
> reg = <0x0 0x21f0 0x0 0x10>;
> ftrace-size = <0x2>;
> };
>
> the memory range "0x0 0x21f0 0x0 0x10" is supposed to fall in
> any of the memory ranges of the memory node (the ones I copy/pasted
> above). But actually it doesn't. That is a device tree error.
>
> Maybe you could try booting on the HiKey changing
Btw, speaking of which: silly question -- it seems I can boot Linux just
fine with this device tree and it appears functional with one caveat: try
as hard as I may I can't get the console output (note that Xen has no
problem outputting to the console -- well at least until it crashes and
with Xen 4.12 Linux is very happy using hvc0).
Any chance you can suggest what would be a reasonable setting
for booting a pure Linux kernel when it comes to console output?
That way, at least, I can do some additional experimenting.
> the device tree so that the
> memory node includes the ramoops range, like this:
>
> memory@0 {
> device_type = "memory";
> reg = <0x0 0x0 0x0 0x5e0 0x0 0x5f0 0x0 0x1000 0x0
> 0x5f02000 0x0 0xefd000 0x0 0x6e0 0x0 0x60f000 0x0 0x741 0x0
> 0x1aaf 0x0 0x21f0 0x0 0x10 0x0 0x2200 0x0 0x1c00>;
> };
>
> (Note that I added "0x0 0x21f0 0x0 0x10" to the list of ranges in
> order.)
>
> Let me know!
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
On Mon, 16 Dec 2019, Roman Shaposhnik wrote:
> Hi!
>
> it appears that something has broken in 4.13 RC5 so that
> I'm now getting the following on ARM (full logs are attached).
>
> (XEN)
> (XEN) Panic on CPU 0:
> (XEN) Failed to allocate requested dom0 memory. 672MB unallocated
> (XEN)
>
> My GRUB boot sequence hasn't changed:
>
> xen_hypervisor /boot/xen.efi console=dtuart dom0_mem=1024M,max:1024M
> dom0_max_vcpus=1 dom0_vcpus_pin
FYI we don't actually support the ",max:1024M" part of the dom0_mem
argument on ARM. On ARM, it should just be:
dom0_mem=1024M
> xen_module /boot/kernel console=hvc0 root=(hd1,gpt1)/rootfs.img text
> devicetree (hd1,gpt4)/eve.dtb
> xen_module (hd1,gpt1)/initrd.img
>
> In fact, if I use Xen 4.12 instead of 4.13 -- everything seems to work
> as it used to.
I spoke too early: I am unable to reproduce it on my end. On what
platforms did you see this error? Was it the HiKey? Could you please
post the devicetree that you are using (eve.dtb from this example)?
If the dts is this one: https://github.com/lf-edge/eve/blob/master/conf/eve.dts
then I might have an idea. In Xen 4.13 we introduced support for
reserved-memory. It looks like the HiKey has a few reserved-memory
regions and I wonder if that is the problem. Could you please remove the
whole "reserved-memory" node and try again to see if that is the issue?
Also looking at eve.dts above, I am a bit puzzled because the memory
node is:
memory@0 {
device_type = "memory";
reg = <0x0 0x0 0x0 0x5e0 0x0 0x5f0 0x0 0x1000 0x0
0x5f02000 0x0 0xefd000 0x0 0x6e0 0x0 0x60f000 0x0 0x741 0x0 0x1aaf
0x0 0x2200 0x0 0x1c00>;
};
If I sum all the memory sizes together I get 0x3ddfd000 which is 990M.
If so, I wonder how you could boot succesfully with dom0_mem=1024M even
on Xen 4.12... :-?
If we look at ramoops@21f0 under reserved-memory:
ramoops@21f0 {
record-size = <0x2>;
compatible = "ramoops";
console-size = <0x2>;
reg = <0x0 0x21f0 0x0 0x10>;
ftrace-size = <0x2>;
};
the memory range "0x0 0x21f0 0x0 0x10" is supposed to fall in
any of the memory ranges of the memory node (the ones I copy/pasted
above). But actually it doesn't. That is a device tree error.
Maybe you could try booting on the HiKey changing the device tree so that the
memory node includes the ramoops range, like this:
memory@0 {
device_type = "memory";
reg = <0x0 0x0 0x0 0x5e0 0x0 0x5f0 0x0 0x1000 0x0
0x5f02000 0x0 0xefd000 0x0 0x6e0 0x0 0x60f000 0x0 0x741 0x0 0x1aaf
0x0 0x21f0 0x0 0x10 0x0 0x2200 0x0 0x1c00>;
};
(Note that I added "0x0 0x21f0 0x0 0x10" to the list of ranges in
order.)
Let me know!
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
Thanks for the report, I'll give it a look! On Mon, 16 Dec 2019, Roman Shaposhnik wrote: > Hi! > > it appears that something has broken in 4.13 RC5 so that > I'm now getting the following on ARM (full logs are attached). > > (XEN) > (XEN) Panic on CPU 0: > (XEN) Failed to allocate requested dom0 memory. 672MB unallocated > (XEN) > > My GRUB boot sequence hasn't changed: > > xen_hypervisor /boot/xen.efi console=dtuart dom0_mem=1024M,max:1024M > dom0_max_vcpus=1 dom0_vcpus_pin > xen_module /boot/kernel console=hvc0 root=(hd1,gpt1)/rootfs.img text > devicetree (hd1,gpt4)/eve.dtb > xen_module (hd1,gpt1)/initrd.img > > In fact, if I use Xen 4.12 instead of 4.13 -- everything seems to work > as it used to. > > Thanks, > Roman. > ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
[Xen-devel] REGRESSION: Xen 4.13 RC5 fails to bootstrap Dom0 on ARM
Hi! it appears that something has broken in 4.13 RC5 so that I'm now getting the following on ARM (full logs are attached). (XEN) (XEN) Panic on CPU 0: (XEN) Failed to allocate requested dom0 memory. 672MB unallocated (XEN) My GRUB boot sequence hasn't changed: xen_hypervisor /boot/xen.efi console=dtuart dom0_mem=1024M,max:1024M dom0_max_vcpus=1 dom0_vcpus_pin xen_module /boot/kernel console=hvc0 root=(hd1,gpt1)/rootfs.img text devicetree (hd1,gpt4)/eve.dtb xen_module (hd1,gpt1)/initrd.img In fact, if I use Xen 4.12 instead of 4.13 -- everything seems to work as it used to. Thanks, Roman. Using modules provided by bootloader in FDT Xen 4.13.0-rc (c/s ) EFI loader Xen 4.13.0-rc (XEN) Xen version 4.13.0-rc (@) (gcc (Alpine 6.4.0) 6.4.0) debug=y Fri Dec 16 07:29:10 UTC 2019 (XEN) Latest ChangeSet: (XEN) build-id: bf2e5dde8df8199230a1b0a25fb8f220be3714a0 (XEN) Processor: 410fd033: "ARM Limited", variant: 0x0, part 0xd03, rev 0x3 (XEN) 64-bit Execution: (XEN) Processor Features: (XEN) Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32 (XEN) Extensions: FloatingPoint AdvancedSIMD (XEN) Debug Features: 10305106 (XEN) Auxiliary Features: (XEN) Memory Model Features: 1122 (XEN) ISA Features: 00011120 (XEN) 32-bit Execution: (XEN) Processor Features: 0131:00011011 (XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle (XEN) Extensions: GenericTimer Security (XEN) Debug Features: 03010066 (XEN) Auxiliary Features: (XEN) Memory Model Features: 10101105 4000 0126 02102211 (XEN) ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121 (XEN) Using SMC Calling Convention v1.0 (XEN) Using PSCI v1.0 (XEN) SMP: Allowing 8 CPUs (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 1200 KHz (XEN) GICv2 initialization: (XEN) gic_dist_addr=f6801000 (XEN) gic_cpu_addr=f6802000 (XEN) gic_hyp_addr=f6804000 (XEN) gic_vcpu_addr=f6806000 (XEN) gic_maintenance_irq=25 (XEN) GICv2: 160 lines, 8 cpus, secure (IID 0200143b). (XEN) XSM Framework v1.0.0 initialized (XEN) Initialising XSM SILO mode (XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2) (XEN) Initializing Credit2 scheduler (XEN) load_precision_shift: 18 (XEN) load_window_shift: 30 (XEN) underload_balance_tolerance: 0 (XEN) overload_balance_tolerance: -3 (XEN) runqueues arrangement: socket (XEN) cap enforcement granularity: 10ms (XEN) load tracking window length 1073741824 ns (XEN) Allocated console ring of 64 KiB. (XEN) CPU0: Guest atomics will try 10 times before pausing the domain (XEN) Bringing up CPU1 (XEN) CPU1: Guest atomics will try 23 times before pausing the domain (XEN) CPU 1 booted. (XEN) Bringing up CPU2 (XEN) CPU2: Guest atomics will try 22 times before pausing the domain (XEN) CPU 2 booted. (XEN) Bringing up CPU3 (XEN) CPU3: Guest atomics will try 18 times before pausing the domain (XEN) CPU 3 booted. (XEN) Bringing up CPU4 (XEN) CPU4: Guest atomics will try 21 times before pausing the domain (XEN) CPU 4 booted. (XEN) Bringing up CPU5 (XEN) CPU5: Guest atomics will try 20 times before pausing the domain (XEN) CPU 5 booted. (XEN) Bringing up CPU6 (XEN) CPU6: Guest atomics will try 18 times before pausing the domain (XEN) CPU 6 booted. (XEN) Bringing up CPU7 (XEN) CPU7: Guest atomics will try 17 times before pausing the domain (XEN) Brought up 8 CPUs (XEN) CPU 7 booted. (XEN) I/O virtualisation disabled (XEN) P2M: 40-bit IPA with 40-bit PA and 8-bit VMID (XEN) P2M: 3 levels with order-1 root, VTCR 0x80023558 (XEN) Adding cpu 0 to runqueue 0 (XEN) First cpu on runqueue, activating (XEN) Adding cpu 1 to runqueue 0 (XEN) Adding cpu 2 to runqueue 0 (XEN) Adding cpu 3 to runqueue 0 (XEN) Adding cpu 4 to runqueue 0 (XEN) Adding cpu 5 to runqueue 0 (XEN) Adding cpu 6 to runqueue 0 (XEN) Adding cpu 7 to runqueue 0 (XEN) alternatives: Patching with alt table 002cc068 -> 002cc77c (XEN) *** LOADING DOMAIN 0 *** (XEN) Loading d0 kernel from boot module @ 48d09000 (XEN) Allocating 1:1 mappings totalling 1024MB for dom0: (XEN) (XEN) (XEN) Panic on CPU 0: (XEN) Failed to allocate requested dom0 memory. 672MB unallocated (XEN) (XEN) (XEN) Reboot in five seconds... ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
