Re: bugs with EFI "reserved" memory and LMB?
Hello all, re-opening this old email thread ... On Sat, Aug 12, 2023 at 08:23:32AM +0200, Heinrich Schuchardt wrote: > On 7/27/23 14:14, Emanuele Ghidoli wrote: > > On 27/07/2023 13:21, Heinrich Schuchardt wrote: > > > On 27.07.23 13:08, Emanuele Ghidoli wrote: > > > > Efi loader module have its own memory management that flags as reserved > > > > the > > > > area between ram_top to ram_end (currently where dt reserved-memory is > > > > falling). uboot lmb updates reserved-memory by adding these > > > > efi loader module areas (see lmb_reserve_common->efi_lmb_reserve). > > > > > > > > > > > > On our system (AM62xx SoC, 512MB RAM) we have some reserved-memory > > > > specified in > > > > the device tree (used for comunication with other microcontroller on > > > > the SoC, > > > > OPTEE, ...) falling in the DDR range between 456 and 512 MB. > > > > > > > > Normally U-Boot relocate itself at the end of DDR so it overlap this > > > > reserved-memory and this of course does not work [1]. > > > > > > > > In this case U-Boot prints the following errors: > > > > > > > > ERROR: reserving fdt memory region failed (addr=9c80 > > > > size=30) > > > > ERROR: reserving fdt memory region failed (addr=9cb0 > > > > size=10) > > > > ... > > > > ERROR: reserving fdt memory region failed (addr=9e80 > > > > size=180) > > > > > > > > because U-Boot reserved area (lmb_flag LMB_NONE) overlaps with the FDT > > > > reserve > > > > memories (lmb_flag LMB_NOMAP). > > > > > > > > > > > > To fix this I moved the U-Boot relocation address, implementing > > > > board_get_usable_ram_top() and therefore setting gd->ram_top to a lower > > > > value > > > > with no overlap (448 MB). > > > > > > > > > > > > The memory map is: > > > > +---+ 512 MB > > > > |DT reserved-mem| > > > > +---+ 456 MB > > > > |free space | > > > > +---+ 448 MB (ram_top) > > > > | | > > > > |uboot | > > > > | | > > > > +---+ 0 MB > > > > > > > > This is working fine (the board is able to boot, no memory overlaps) ... > > > > > > > > however, in this configuration U-Boot we still have some errors prints > > > > while > > > > loading linux dtb: > > > > > > > > ERROR: reserving fdt memory region failed (addr=9cb0 > > > > size=10 flags=4) > > > > ERROR: reserving fdt memory region failed (addr=9cc0 > > > > size=e0 flags=4) > > > > ERROR: reserving fdt memory region failed (addr=9da0 > > > > size=10 flags=4) > > > > ERROR: reserving fdt memory region failed (addr=9db0 > > > > size=c0 flags=4) > > > > ERROR: reserving fdt memory region failed (addr=9e78 size=8 > > > > flags=4) > > > > ERROR: reserving fdt memory region failed (addr=9e80 > > > > size=180 flags=4) > > > > > > > > And the reason of these error is complete different: The efi loader lmb > > > > memory > > > > reservation (lib/lmb.c:efi_lmb_reserve()) is wrongly reserving some > > > > area. > > > > > > > > > > > > 1) lib/lmb.c:lmb_reserve_common() reserve these memories: > > > > ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) > > > > 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) > > > > > > > > 2) lib/lmb.c:efi_lmb_reserve() reserve a region just before u-boot > > > > reserved > > > > area. This is not coalesced/merged with the u-boot reserved area > > > > because it is > > > > not contiguous. > > > > > > > > This is the reserved-memory array at this point: > > > > ~414M - ~414M efi loader reserved area (lmb_flag LMB_NONE) > > > > ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) > > > > 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) > > > > > > > > 3) lib/lmb.c:efi_lmb_reserve() reserve some more areas. > > > > In the last two steps efi loader add some reserved area that overlap > > > > u-boot and fdt > > > > reserved area BUT they are also contiguous to the "efi loader reserved > > > > area". > > > > The very last step reserve also area between ram_top (448M) to ram_end > > > > (512M) > > > > > > > > We fall in this condition (where we have overlapping areas!): > > > > ~414M - 512M efi loader reserved area (lmb_flag LMB_NONE) > > > > ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) > > > > 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) > > > > > > > > 4) Now, while loading the linux fdt the reserved-memory areas are > > > > checked > > > > toward efi loader reserved area, they overlap, BUT they have different > > > > lmb_flag. So we get the ERROR print. > > > > > > > > Hopefully this is clear, I undestand is not that obvious ... > > > > > > efidebug memmap > > Verdin AM62 # efidebug memmap > > MMC: no card present > > No EFI system partition > > No EFI system partition > > Failed to persist EFI variables > > Type StartEnd Attributes > >
Re: bugs with EFI "reserved" memory and LMB?
On 7/27/23 14:14, Emanuele Ghidoli wrote: On 27/07/2023 13:21, Heinrich Schuchardt wrote: On 27.07.23 13:08, Emanuele Ghidoli wrote: Efi loader module have its own memory management that flags as reserved the area between ram_top to ram_end (currently where dt reserved-memory is falling). uboot lmb updates reserved-memory by adding these efi loader module areas (see lmb_reserve_common->efi_lmb_reserve). On our system (AM62xx SoC, 512MB RAM) we have some reserved-memory specified in the device tree (used for comunication with other microcontroller on the SoC, OPTEE, ...) falling in the DDR range between 456 and 512 MB. Normally U-Boot relocate itself at the end of DDR so it overlap this reserved-memory and this of course does not work [1]. In this case U-Boot prints the following errors: ERROR: reserving fdt memory region failed (addr=9c80 size=30) ERROR: reserving fdt memory region failed (addr=9cb0 size=10) ... ERROR: reserving fdt memory region failed (addr=9e80 size=180) because U-Boot reserved area (lmb_flag LMB_NONE) overlaps with the FDT reserve memories (lmb_flag LMB_NOMAP). To fix this I moved the U-Boot relocation address, implementing board_get_usable_ram_top() and therefore setting gd->ram_top to a lower value with no overlap (448 MB). The memory map is: +---+ 512 MB |DT reserved-mem| +---+ 456 MB |free space | +---+ 448 MB (ram_top) | | |uboot | | | +---+ 0 MB This is working fine (the board is able to boot, no memory overlaps) ... however, in this configuration U-Boot we still have some errors prints while loading linux dtb: ERROR: reserving fdt memory region failed (addr=9cb0 size=10 flags=4) ERROR: reserving fdt memory region failed (addr=9cc0 size=e0 flags=4) ERROR: reserving fdt memory region failed (addr=9da0 size=10 flags=4) ERROR: reserving fdt memory region failed (addr=9db0 size=c0 flags=4) ERROR: reserving fdt memory region failed (addr=9e78 size=8 flags=4) ERROR: reserving fdt memory region failed (addr=9e80 size=180 flags=4) And the reason of these error is complete different: The efi loader lmb memory reservation (lib/lmb.c:efi_lmb_reserve()) is wrongly reserving some area. 1) lib/lmb.c:lmb_reserve_common() reserve these memories: ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) 2) lib/lmb.c:efi_lmb_reserve() reserve a region just before u-boot reserved area. This is not coalesced/merged with the u-boot reserved area because it is not contiguous. This is the reserved-memory array at this point: ~414M - ~414M efi loader reserved area (lmb_flag LMB_NONE) ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) 3) lib/lmb.c:efi_lmb_reserve() reserve some more areas. In the last two steps efi loader add some reserved area that overlap u-boot and fdt reserved area BUT they are also contiguous to the "efi loader reserved area". The very last step reserve also area between ram_top (448M) to ram_end (512M) We fall in this condition (where we have overlapping areas!): ~414M - 512M efi loader reserved area (lmb_flag LMB_NONE) ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) 4) Now, while loading the linux fdt the reserved-memory areas are checked toward efi loader reserved area, they overlap, BUT they have different lmb_flag. So we get the ERROR print. Hopefully this is clear, I undestand is not that obvious ... Thanks Emanuele for reporting the issue. Could you, please, provide the output of efidebug memmap Verdin AM62 # efidebug memmap MMC: no card present No EFI system partition No EFI system partition Failed to persist EFI variables Type StartEnd Attributes == CONVENTIONAL 8000-98eec000 WB BOOT DATA98eec000-98eee000 WB RUNTIME DATA 98eee000-98eef000 WB|RT BOOT DATA98eef000-98ef2000 WB RUNTIME DATA 98ef2000-98ef3000 WB|RT BOOT DATA98ef3000-98ef4000 WB RUNTIME DATA 98ef4000-98ef6000 WB|RT BOOT DATA98ef6000-98ef7000 WB RUNTIME DATA 98ef7000-98f07000 WB|RT BOOT DATA98f07000-98f1 WB BOOT CODE98f1-9bf2 WB RUNTIME CODE 9bf2-9bf3 WB|RT BOOT CODE9bf3-9c00 WB BOOT DATA9c00-a000 WB and of bdinfo With this "debug" patch I hope giving you a better picture: diff --git a/lib/lmb.c b/lib/lmb.c index
Re: bugs with EFI "reserved" memory and LMB?
On 27/07/2023 13:21, Heinrich Schuchardt wrote: > On 27.07.23 13:08, Emanuele Ghidoli wrote: >> Efi loader module have its own memory management that flags as reserved the >> area between ram_top to ram_end (currently where dt reserved-memory is >> falling). uboot lmb updates reserved-memory by adding these >> efi loader module areas (see lmb_reserve_common->efi_lmb_reserve). >> >> >> On our system (AM62xx SoC, 512MB RAM) we have some reserved-memory specified >> in >> the device tree (used for comunication with other microcontroller on the SoC, >> OPTEE, ...) falling in the DDR range between 456 and 512 MB. >> >> Normally U-Boot relocate itself at the end of DDR so it overlap this >> reserved-memory and this of course does not work [1]. >> >> In this case U-Boot prints the following errors: >> >> ERROR: reserving fdt memory region failed (addr=9c80 size=30) >> ERROR: reserving fdt memory region failed (addr=9cb0 size=10) >> ... >> ERROR: reserving fdt memory region failed (addr=9e80 size=180) >> >> because U-Boot reserved area (lmb_flag LMB_NONE) overlaps with the FDT >> reserve >> memories (lmb_flag LMB_NOMAP). >> >> >> To fix this I moved the U-Boot relocation address, implementing >> board_get_usable_ram_top() and therefore setting gd->ram_top to a lower value >> with no overlap (448 MB). >> >> >> The memory map is: >> +---+ 512 MB >> |DT reserved-mem| >> +---+ 456 MB >> |free space | >> +---+ 448 MB (ram_top) >> | | >> |uboot | >> | | >> +---+ 0 MB >> >> This is working fine (the board is able to boot, no memory overlaps) ... >> >> however, in this configuration U-Boot we still have some errors prints while >> loading linux dtb: >> >> ERROR: reserving fdt memory region failed (addr=9cb0 size=10 >> flags=4) >> ERROR: reserving fdt memory region failed (addr=9cc0 size=e0 >> flags=4) >> ERROR: reserving fdt memory region failed (addr=9da0 size=10 >> flags=4) >> ERROR: reserving fdt memory region failed (addr=9db0 size=c0 >> flags=4) >> ERROR: reserving fdt memory region failed (addr=9e78 size=8 >> flags=4) >> ERROR: reserving fdt memory region failed (addr=9e80 size=180 >> flags=4) >> >> And the reason of these error is complete different: The efi loader lmb >> memory >> reservation (lib/lmb.c:efi_lmb_reserve()) is wrongly reserving some area. >> >> >> 1) lib/lmb.c:lmb_reserve_common() reserve these memories: >> ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) >> 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) >> >> 2) lib/lmb.c:efi_lmb_reserve() reserve a region just before u-boot reserved >> area. This is not coalesced/merged with the u-boot reserved area because it >> is >> not contiguous. >> >> This is the reserved-memory array at this point: >> ~414M - ~414M efi loader reserved area (lmb_flag LMB_NONE) >> ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) >> 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) >> >> 3) lib/lmb.c:efi_lmb_reserve() reserve some more areas. >> In the last two steps efi loader add some reserved area that overlap u-boot >> and fdt >> reserved area BUT they are also contiguous to the "efi loader reserved area". >> The very last step reserve also area between ram_top (448M) to ram_end (512M) >> >> We fall in this condition (where we have overlapping areas!): >> ~414M - 512M efi loader reserved area (lmb_flag LMB_NONE) >> ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) >> 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) >> >> 4) Now, while loading the linux fdt the reserved-memory areas are checked >> toward efi loader reserved area, they overlap, BUT they have different >> lmb_flag. So we get the ERROR print. >> >> Hopefully this is clear, I undestand is not that obvious ... > > Thanks Emanuele for reporting the issue. > > Could you, please, provide the output of > > efidebug memmap Verdin AM62 # efidebug memmap MMC: no card present No EFI system partition No EFI system partition Failed to persist EFI variables Type StartEnd Attributes == CONVENTIONAL 8000-98eec000 WB BOOT DATA98eec000-98eee000 WB RUNTIME DATA 98eee000-98eef000 WB|RT BOOT DATA98eef000-98ef2000 WB RUNTIME DATA 98ef2000-98ef3000 WB|RT BOOT DATA98ef3000-98ef4000 WB RUNTIME DATA 98ef4000-98ef6000 WB|RT BOOT DATA98ef6000-98ef7000 WB RUNTIME DATA 98ef7000-98f07000 WB|RT BOOT DATA98f07000-98f1 WB BOOT CODE98f1-9bf2 WB RUNTIME CODE 9bf2-9bf3 WB|RT BOOT CODE
Re: bugs with EFI "reserved" memory and LMB?
On 27.07.23 13:08, Emanuele Ghidoli wrote: Efi loader module have its own memory management that flags as reserved the area between ram_top to ram_end (currently where dt reserved-memory is falling). uboot lmb updates reserved-memory by adding these efi loader module areas (see lmb_reserve_common->efi_lmb_reserve). On our system (AM62xx SoC, 512MB RAM) we have some reserved-memory specified in the device tree (used for comunication with other microcontroller on the SoC, OPTEE, ...) falling in the DDR range between 456 and 512 MB. Normally U-Boot relocate itself at the end of DDR so it overlap this reserved-memory and this of course does not work [1]. In this case U-Boot prints the following errors: ERROR: reserving fdt memory region failed (addr=9c80 size=30) ERROR: reserving fdt memory region failed (addr=9cb0 size=10) ... ERROR: reserving fdt memory region failed (addr=9e80 size=180) because U-Boot reserved area (lmb_flag LMB_NONE) overlaps with the FDT reserve memories (lmb_flag LMB_NOMAP). To fix this I moved the U-Boot relocation address, implementing board_get_usable_ram_top() and therefore setting gd->ram_top to a lower value with no overlap (448 MB). The memory map is: +---+ 512 MB |DT reserved-mem| +---+ 456 MB |free space | +---+ 448 MB (ram_top) | | |uboot | | | +---+ 0 MB This is working fine (the board is able to boot, no memory overlaps) ... however, in this configuration U-Boot we still have some errors prints while loading linux dtb: ERROR: reserving fdt memory region failed (addr=9cb0 size=10 flags=4) ERROR: reserving fdt memory region failed (addr=9cc0 size=e0 flags=4) ERROR: reserving fdt memory region failed (addr=9da0 size=10 flags=4) ERROR: reserving fdt memory region failed (addr=9db0 size=c0 flags=4) ERROR: reserving fdt memory region failed (addr=9e78 size=8 flags=4) ERROR: reserving fdt memory region failed (addr=9e80 size=180 flags=4) And the reason of these error is complete different: The efi loader lmb memory reservation (lib/lmb.c:efi_lmb_reserve()) is wrongly reserving some area. 1) lib/lmb.c:lmb_reserve_common() reserve these memories: ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) 2) lib/lmb.c:efi_lmb_reserve() reserve a region just before u-boot reserved area. This is not coalesced/merged with the u-boot reserved area because it is not contiguous. This is the reserved-memory array at this point: ~414M - ~414M efi loader reserved area (lmb_flag LMB_NONE) ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) 3) lib/lmb.c:efi_lmb_reserve() reserve some more areas. In the last two steps efi loader add some reserved area that overlap u-boot and fdt reserved area BUT they are also contiguous to the "efi loader reserved area". The very last step reserve also area between ram_top (448M) to ram_end (512M) We fall in this condition (where we have overlapping areas!): ~414M - 512M efi loader reserved area (lmb_flag LMB_NONE) ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) 4) Now, while loading the linux fdt the reserved-memory areas are checked toward efi loader reserved area, they overlap, BUT they have different lmb_flag. So we get the ERROR print. Hopefully this is clear, I undestand is not that obvious ... Thanks Emanuele for reporting the issue. Could you, please, provide the output of efidebug memmap and of bdinfo Best regards Heinrich IMO we have two different bugs: - there is nothing that prevent that coalesced area may overlap other areas (while lmb module expect that there aren't overlapping areas) - efi loader (correctly) reserve between ram_top and ram_end BUT this area is reserved by fdt. One potential solution could be to override efi_add_known_memory() and set ram_top = ram_end, e.g. - efi_add_conventional_memory_map(ram_start, ram_end, ram_top); + efi_add_conventional_memory_map(ram_start, ram_end, ram_end); but this does not really seems like something that should ve overridden at the board level. Any suggestion? Emanuele
bugs with EFI "reserved" memory and LMB?
Efi loader module have its own memory management that flags as reserved the area between ram_top to ram_end (currently where dt reserved-memory is falling). uboot lmb updates reserved-memory by adding these efi loader module areas (see lmb_reserve_common->efi_lmb_reserve). On our system (AM62xx SoC, 512MB RAM) we have some reserved-memory specified in the device tree (used for comunication with other microcontroller on the SoC, OPTEE, ...) falling in the DDR range between 456 and 512 MB. Normally U-Boot relocate itself at the end of DDR so it overlap this reserved-memory and this of course does not work [1]. In this case U-Boot prints the following errors: ERROR: reserving fdt memory region failed (addr=9c80 size=30) ERROR: reserving fdt memory region failed (addr=9cb0 size=10) ... ERROR: reserving fdt memory region failed (addr=9e80 size=180) because U-Boot reserved area (lmb_flag LMB_NONE) overlaps with the FDT reserve memories (lmb_flag LMB_NOMAP). To fix this I moved the U-Boot relocation address, implementing board_get_usable_ram_top() and therefore setting gd->ram_top to a lower value with no overlap (448 MB). The memory map is: +---+ 512 MB |DT reserved-mem| +---+ 456 MB |free space | +---+ 448 MB (ram_top) | | |uboot | | | +---+ 0 MB This is working fine (the board is able to boot, no memory overlaps) ... however, in this configuration U-Boot we still have some errors prints while loading linux dtb: ERROR: reserving fdt memory region failed (addr=9cb0 size=10 flags=4) ERROR: reserving fdt memory region failed (addr=9cc0 size=e0 flags=4) ERROR: reserving fdt memory region failed (addr=9da0 size=10 flags=4) ERROR: reserving fdt memory region failed (addr=9db0 size=c0 flags=4) ERROR: reserving fdt memory region failed (addr=9e78 size=8 flags=4) ERROR: reserving fdt memory region failed (addr=9e80 size=180 flags=4) And the reason of these error is complete different: The efi loader lmb memory reservation (lib/lmb.c:efi_lmb_reserve()) is wrongly reserving some area. 1) lib/lmb.c:lmb_reserve_common() reserve these memories: ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) 2) lib/lmb.c:efi_lmb_reserve() reserve a region just before u-boot reserved area. This is not coalesced/merged with the u-boot reserved area because it is not contiguous. This is the reserved-memory array at this point: ~414M - ~414M efi loader reserved area (lmb_flag LMB_NONE) ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) 3) lib/lmb.c:efi_lmb_reserve() reserve some more areas. In the last two steps efi loader add some reserved area that overlap u-boot and fdt reserved area BUT they are also contiguous to the "efi loader reserved area". The very last step reserve also area between ram_top (448M) to ram_end (512M) We fall in this condition (where we have overlapping areas!): ~414M - 512M efi loader reserved area (lmb_flag LMB_NONE) ~415M - 448M u-boot reserved area (lmb_flag LMB_NONE) 456M - 512M fdt reserved-memory (lmb_flag LMB_NOMAP) 4) Now, while loading the linux fdt the reserved-memory areas are checked toward efi loader reserved area, they overlap, BUT they have different lmb_flag. So we get the ERROR print. Hopefully this is clear, I undestand is not that obvious ... IMO we have two different bugs: - there is nothing that prevent that coalesced area may overlap other areas (while lmb module expect that there aren't overlapping areas) - efi loader (correctly) reserve between ram_top and ram_end BUT this area is reserved by fdt. One potential solution could be to override efi_add_known_memory() and set ram_top = ram_end, e.g. - efi_add_conventional_memory_map(ram_start, ram_end, ram_top); + efi_add_conventional_memory_map(ram_start, ram_end, ram_end); but this does not really seems like something that should ve overridden at the board level. Any suggestion? Emanuele
