Since commit a3075db94d49 ("lmb: Reinstate access to memory above
ram_top") secondary CPUs cannot be brought online on the E850-96 board
in the Linux kernel:
smp: Bringing up secondary CPUs ...
CPU1: failed to come online
CPU1: failed in unknown state : 0x0
CPU2: failed to come online
CPU2: failed in unknown state : 0x0
...
This is happening because EL3_MON (the Secure Monitor firmware, which
also implements PSCI) cannot handle 36-bit addresses in its CPU_ON PSCI
function implementation. The E850-96 board has two 2 GiB RAM banks: the
first one is mapped at 0x8000_0000, and the second one at 0x8_8000_0000.
The commit mentioned above changed the LMB behavior (in U-Boot) in a way
that the memory from the second RAM bank (which is mapped at 36-bit
addresses) is not reserved anymore. The change in 'bdinfo' output:
- reserved[4] [0x8_8000_0000 - 0x8_ffff_ffff], 0x8000_0000 bytes
+ reserved[3] [0x8_ffff_3000 - 0x8_ffff_ffff], 0xd000 bytes
That in turn leads to the Linux kernel image being placed in the second
RAM bank with 36-bit addresses. The corresponding change in /proc/iomem:
- f480_0000 - f67e_ffff : Kernel code
+ 8_fa80_0000 - 8_fc7e_ffff : Kernel code
- f6b6_0000 - f71a_ffff : Kernel data
+ 8_fcb6_0000 - 8_fd1a_ffff : Kernel data
As evident from the Exynos850 device tree, it uses "psci" as a CPU
enable method. That means the kernel relies on PSCI firmware (which is
implemented as a part of EL3_MON firmware in case of Exynos850 SoC) to
bring up secondary CPU cores. So when bringing up secondary CPUs the
kernel makes an SMC call (for each secondary CPU) to execute CPU_ON PSCI
function (see PSCI spec [1]). The kernel passes an "entry point address"
to that SMC call, which is the address of the secondary_entry() function
in the kernel, which should be executed when PSCI returns to the kernel
from the SMC call:
static int cpu_psci_cpu_boot(unsigned int cpu)
{
phys_addr_t pa_secondary_entry = __pa_symbol(secondary_entry);
int err = psci_ops.cpu_on(cpu_logical_map(cpu),
pa_secondary_entry);
...
The pa_secondary_entry is a 64-bit variable, which is perfectly fine
according to the PSCI spec. Unfortunately, in case when that address is
actually bigger than 32-bit value, the EL3_MON firmware doesn't execute
the function at that address -- that's the reason the secondary CPUs
don't come up. And as discussed above, the kernel image is now located
at a 36-bit address, which means the secondary_entry() function is at a
36-bit address too:
- secondary_entry() address: f67d_f5a8 // 32-bit address
+ secondary_entry() address: 8_fc7d_f5a8 // 36-bit address
It seems like a bug in the EL3_MON firmware, e.g. it might truncate the
address to 32-bit by using a u32 variable (instead of u64) to store it
internally. So the right way to fix this issue would be to fix the
EL3_MON, but unfortunately it's closed source. So instead just enable
LMB_LIMIT_DMA_BELOW_RAM_TOP for ARCH_EXYNOS9 to limit the available
memory to the 32-bit boundary.
Although this platform could benefit from having all memory available in
U-Boot (e.g. for transferring big images over TFTP), and it looks like
there are no hardware limitations on this platform too (DMA has 36-bit
support, etc.), limiting the available memory in U-Boot is still the
easiest way to workaround the EL3_MON problem, and 2 GiB is enough for
all use cases in U-Boot currently. And of course all 4 GiB of RAM is
available to use in Linux in any case, so it's not a bad tradeoff, and
in fact it just brings back the previous behavior (before the "Reinstate
access ..." patch).
Enabling LMB_LIMIT_DMA_BELOW_RAM_TOP for the whole ARCH_EXYNOS9 family
also seems justified, as it's likely that all modern ARM64 Exynos chips
may share similar EL3_MON software, and thus possibly the same PSCI bug.
For example, a similar issue was mentioned in exynos-mobile related
patch [2], stating that after "Reinstate access ..." patch only CPU0 is
able to come online on some variants of the Samsung Galaxy J6 (based on
Exynos7870 SoC).
This was tested on the E850-96 board:
1. Using 'bootefi' to boot the Linux kernel image
- before this patch: the issue is reproducible
- with this patch applied: the issue is not reproducible
2. Using 'booti' to boot the Linux kernel image from 0x8_8000_0000
- before this patch: the issue is reproducible
- with this patch applied: the image can't be loaded there anymore
("Reading file would overwrite reserved memory"), as the second
RAM bank is reserved in LMB again
3. Using 'booti' to boot the Linux kernel image from 0x8000_0000
- the issue is not reproducible
[1] https://support.arm.com/documentation/den0022/fb/
[2]
https://lore.kernel.org/u-boot/[email protected]/
Signed-off-by: Sam Protsenko <[email protected]>
---
lib/Kconfig | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/Kconfig b/lib/Kconfig
index 021a82d47092..e0bfd10f2b7f 100644
--- a/lib/Kconfig
+++ b/lib/Kconfig
@@ -1371,7 +1371,8 @@ config SPL_LMB_ARCH_MEM_MAP
config LMB_LIMIT_DMA_BELOW_RAM_TOP
bool
depends on LMB
- default y if ARCH_BCM283X || ARCH_MVEBU || ARCH_ROCKCHIP ||
STARFIVE_JH7110
+ default y if ARCH_BCM283X || ARCH_EXYNOS9 || ARCH_MVEBU || \
+ ARCH_ROCKCHIP || STARFIVE_JH7110
help
Some architectures can not DMA above ram_top boundary,
which is after 4 GiB or 32-bit boundary too. Limit the
base-commit: 8d534cec0cc0109a37c0ded8854da03f1526dc0e
--
2.47.3
