hujun260 opened a new pull request, #22080:
URL: https://github.com/apache/nuttx/pull/22080

   # arch/arm64: consolidate common boot sequence and centralize D-Cache 
lifecycle
   
   ## Summary
   
   Two related cleanups to the arm64 boot path:
   
   1. **Consolidate the duplicated boot sequence.** Every SoC 
`arm64_chip_boot()`
      repeated the same MMU/MPU init, MTE init, FDT registration, per-CPU data
      copy, PSCI init and early serial init. These move into the common
      `arm64_boot_primary_c_routine()`, removing ~340 lines across 13 boot 
files.
   
   2. **Centralize the D-Cache lifecycle.** Bracketing MPU/MMU setup with
      `up_disable_dcache()` / `up_enable_dcache()` in both the primary and
      secondary CPU paths, so the cache is off while the translation/protection
      tables are being built. `arm64_mpu_enable/disable()` and
      `enable_mmu_el1/el3()` stop touching `SCTLR_C_BIT`, so cache state is
      controlled in exactly one place instead of being scattered across helpers.
   
   ## Problem
   
   ### Duplicated boot sequence
   
   Before the change, each `*_boot.c` carried its own copy of the same sequence:
   
   ```c
   void arm64_chip_boot(void)
   {
     arm64_mmu_init(true);              /* identical in every SoC   */
   
   #ifdef CONFIG_ARM64_MTE
     arm64_mte_init();                  /* identical                */
   #endif
   
   #if defined(CONFIG_ARM64_PSCI)
     arm64_psci_init("smc");            /* identical                */
   #endif
   
     xxx_board_initialize();            /* genuinely SoC-specific   */
   
   #ifdef USE_EARLYSERIALINIT
     arm64_earlyserialinit();           /* identical                */
   #endif
   }
   ```
   
   With nine SoC files holding the same block, every policy change (PSCI conduit
   selection, cache ordering, per-CPU data copy) had to be applied 9+ times, and
   drifted in the meantime. Concretely, the call order was *not* uniform before
   this change:
   
   | SoC | order of `psci_init` vs `board_initialize` |
   | --- | --- |
   | `a64`, `bcm2711`, `rk3399` | before |
   | `imx9` | before `earlyserialinit`, after `lowsetup` |
   | `qemu`, `goldfish` | after `fdt_register`, before board init |
   
   and `imx9` additionally guarded `arm64_mmu_init()` with
   `CONFIG_ARCH_USE_MMU` while every other SoC used it unconditionally.
   
   ### D-Cache state during table setup
   
   `setup_page_tables()` walks `g_mmu_config.mmu_regions[]` and writes the
   translation tables *before* the MMU is enabled, i.e. while all memory is 
still
   treated as Device-nGnRnE. The old code enabled the D-Cache as a side effect 
of
   the last helper that happened to run:
   
   ```c
   /* arm64_mmu.c, before */
   write_sysreg((value | SCTLR_M_BIT
   #ifndef CONFIG_ARM64_DCACHE_DISABLE
                  | SCTLR_C_BIT
   #endif
                  ), sctlr_el1);
   ```
   
   This made the cache state depend on *which* helper ran last, and on whether a
   particular SoC used the MPU or the MMU path. It also meant the MPU path and 
the
   MMU path could disagree about cache state for the same configuration.
   
   ## Change
   
   ### 1. Common boot sequence
   
   `arm64_boot_primary_c_routine()` now performs the shared steps:
   
   ```c
   void arm64_boot_primary_c_routine(void)
   {
   #ifdef CONFIG_ARCH_HAVE_MPU
     arm64_mpu_init(true);
   #endif
   
   #if defined(CONFIG_ARCH_HAVE_MMU)
   #  if defined(CONFIG_ARCH_CHIP_IMX9) && !defined(CONFIG_ARCH_USE_MMU)
     /* i.MX9 BL2 bootloaders deliberately boot without the MMU: TF-A
      * reconfigures it afterwards and the translation tables cost ~50 KB.
      * MMU setup is only performed when CONFIG_ARCH_USE_MMU is selected.
      */
   #  else
     arm64_mmu_init(true);
   #  endif
   #endif
   
   #ifdef CONFIG_ARM64_MTE
     arm64_mte_init();
   #endif
   
   #ifndef CONFIG_ARM64_DCACHE_DISABLE
     up_enable_dcache();
   #endif
   
   #ifdef CONFIG_PERCPU_SECTION
     for (int cpu = 1; cpu < CONFIG_NCPUS; cpu++)
       {
         memcpy((void *)((uintptr_t)_sdata_percpu + PERCPU_OFFSET * cpu),
                _ldata_percpu,
                (uintptr_t)_edata_percpu - (uintptr_t)_sdata_percpu);
       }
   #endif
   
   #ifdef CONFIG_ARM64_PSCI
   #  if defined(CONFIG_ARCH_CHIP_QEMU_WITH_HV)
     arm64_psci_init("hvc");
   #  elif defined(CONFIG_ARCH_CHIP_AM62X) && !defined(CONFIG_SMP)
     /* Keep PSCI out of the initial UP bring-up path until the basic
      * boot-to-NSH flow is stable on AM62x. */
   #  elif CONFIG_ARCH_ARM64_EXCEPTION_LEVEL < 3
     arm64_psci_init("smc");
   #  endif
   #endif
   
     arm64_chip_boot();
   
   #ifdef USE_EARLYSERIALINIT
     arm64_earlyserialinit();
   #endif
     nx_start();
   }
   ```
   
   Each SoC file keeps only what is genuinely SoC-specific:
   
   ```c
   /* after */
   void arm64_chip_boot(void)
   {
     a64_board_initialize();
   }
   ```
   
   `imx9` keeps its BL2 pre-initialization and clock/TRDC setup, which must run
   before `nx_start()` but is otherwise self-contained.
   
   ### 2. D-Cache lifecycle
   
   `up_enable_dcache()` is called once, after the tables are live, on both 
paths:
   
   ```c
     /* MPU/MMU setup above runs with the D-Cache disabled; invalidate and
      * re-enable it before touching shared data structures.
      */
   
   #ifndef CONFIG_ARM64_DCACHE_DISABLE
     up_enable_dcache();
   #endif
   ```
   
   `up_enable_dcache()` already invalidates before enabling, so stale lines from
   the pre-MMU window cannot survive. The `SCTLR_C_BIT` handling is removed from
   the helpers that used to do it implicitly:
   
   ```diff
   -  write_sysreg((value | SCTLR_M_BIT
   -#ifndef CONFIG_ARM64_DCACHE_DISABLE
   -               | SCTLR_C_BIT
   -#endif
   -               ), sctlr_el1);
   +  write_sysreg((value | SCTLR_M_BIT), sctlr_el1);
   ```
   
   ## Design notes
   
   **Why `#ifndef CONFIG_ARM64_DCACHE_DISABLE` around the new calls.** Removing 
the
   `SCTLR_C_BIT` handling from the helpers left the 
`CONFIG_ARM64_DCACHE_DISABLE`
   promise ("Disable DCACHE at `__start`", `arch/arm64/Kconfig:534`) with no
   enforcement anywhere. Five in-tree configurations select it:
   
   ```text
   boards/arm64/zynq-mpsoc/zcu111/configs/{nsh,jtag,netnsh,netjtag}/defconfig
   boards/arm64/imx9/imx93-evk/configs/bootloader/defconfig
   ```
   
   Guarding the new `up_enable_dcache()` calls keeps those builds cache-off 
while
   still giving the normal builds a single, explicit re-enable point.
   
   **Why `imx9` keeps its `CONFIG_ARCH_USE_MMU` distinction.** Upstream commit
   `5d63c3e460e` ("arch/arm64: Allow non MMU builds") deliberately made BL2
   bootloaders able to skip MMU setup: TF-A reconfigures it afterwards, and the
   translation tables cost ~50 KB. `arm64_mpu_init`/`arm64_mmu_init` keying off
   `CONFIG_ARCH_HAVE_MMU` alone would silently reintroduce MMU setup into those
   builds, so the chip-specific exception is preserved in the common routine
   rather than dropped.
   
   **Why `am62x` keeps its PSCI exception.** Its UP path intentionally skips 
PSCI
   init until the boot-to-NSH flow is stable, and only enables it for SMP. That
   carve-out moves into the common routine so behaviour is unchanged.
   
   **Includes dropped along the way.** Moving the calls out left these unused in
   the SoC files: `<stdint.h>`, `<assert.h>`, `<nuttx/debug.h>`, 
`<nuttx/cache.h>`,
   the `CONFIG_LEGACY_PAGING`-guarded `<nuttx/page.h>`, `"arm64_smp.h"`,
   `"arm64_arch.h"` (`arm64_internal.h` already pulls it in) and the per-SoC
   `*_serial.h` headers. Headers still needed by the remaining code
   (`<arch/barriers.h>`, `arm64_mmu.h`/`arm64_mpu.h`, the MMU region tables) are
   kept.
   
   ## Verification
   
   ### Build matrix
   
   All configurations built from clean with
   `aarch64-none-elf-gcc`, `tools/configure.sh -a ../apps <board>:<config>`,
   `make -j$(nproc)`:
   
   | Configuration | MMU/MPU | SMP | `DCACHE_DISABLE` | Result |
   | --- | --- | --- | --- | --- |
   | `qemu-armv8a:nsh` | MMU | - | - | PASS |
   | `qemu-armv8a:nsh_smp` | MMU | yes | - | PASS |
   | `qemu-armv8a:mte` | MMU | - | - | PASS |
   | `qemu-armv8a:knsh` | MMU | - | - | PASS |
   | `pinephone:nsh` (a64) | MMU | - | - | PASS |
   | `nanopi_m4:nsh` (rk3399) | MMU | - | - | PASS |
   | `imx8qm-mek:nsh` | MMU | - | - | PASS |
   | `raspberrypi-4b:nsh` (bcm2711) | MMU | - | - | PASS |
   | `zcu111:nsh` (zynq-mpsoc) | MMU | - | **yes** | PASS |
   | `avaota-a1:nsh` (a527) | MMU | - | - | PASS |
   | `beagleplay:nsh` (am62x) | MMU | - | - | PASS |
   | `orangepi-5-pro:nsh` (rk3588) | MMU | - | - | PASS |
   | `imx93-evk:nsh` | - | - | - | PASS |
   | `imx93-evk:bootloader` | - | - | **yes** | PASS |
   | `fvp-armv8r:nsh` | MPU | - | - | PASS |
   | `fvp-armv8r:nsh_smp` | MPU | yes | - | PASS |
   
   ### Symbol-level checks
   
   The `CONFIG_ARM64_DCACHE_DISABLE` carve-out and the i.MX9 exception were both
   confirmed at object level rather than by reading the preprocessor result:
   
   ```text
   $ tools/configure.sh -a ../apps zcu111:nsh && make -C arch/arm64/src 
arm64_boot.o
   $ aarch64-none-elf-nm arch/arm64/src/arm64_boot.o | grep up_enable_dcache
     (no output)                     -> D-Cache left disabled, as configured
   
   $ tools/configure.sh -a ../apps qemu-armv8a:nsh_smp
   $ aarch64-none-elf-nm arch/arm64/src/arm64_boot.o     | grep -c "U 
up_enable_dcache"
   1
   $ aarch64-none-elf-nm arch/arm64/src/arm64_cpustart.o | grep -c "U 
up_enable_dcache"
   1                                 -> both boot paths re-enable the cache
   
   $ tools/configure.sh -a ../apps imx93-evk:bootloader
   $ aarch64-none-elf-nm arch/arm64/src/arm64_boot.o | grep -E 
"mmu_init|mte_init|psci_init"
     (no output)                     -> MMU/MTE/PSCI correctly compiled out
   ```
   
   ### Runtime
   
   `qemu-armv8a:nsh` on `qemu-system-aarch64` with the board's own arguments:
   
   ```text
   $ qemu-system-aarch64 -cpu cortex-a53 -nographic \
       -machine virt,virtualization=on,gic-version=3 \
       -net none -chardev stdio,id=con,mux=on \
       -serial chardev:con -monitor none -kernel nuttx
   ```
   
   ```text
   - Ready to Boot Primary CPU
   - Boot from EL2
   - Boot from EL1
   - Boot to C runtime for OS Initialize
   nsh> uname -a
   NuttX 13.1.1-RC0 117248aad55 Nov 20 2026 09:41:24 arm64 qemu-armv8a
   nsh> ps
     TID   PID  PPID PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK        
    STACK    USED FILLED COMMAND
       0     0     0   0 FIFO     Kthread   - Ready              
0000000000000000 0008144 0001744  21.4%  Idle_Task
       1     0     0 192 RR       Kthread   - Waiting  Semaphore 
0000000000000000 0008080 0001152  14.2%  hpwork 0x403ec3a0 0x403ec428
       3     3     0 100 RR       Task      - Waiting  Semaphore 
0000000000000000 0008112 0002528  31.1%  init_main
       4     4     3 100 RR       Task      - Running            
0000000000000000 0008112 0003200  39.4%  sh
   ```
   
   `qemu-armv8a:nsh_smp` with `-smp 4`, exercising the secondary-CPU cache path:
   
   ```text
   - Ready to Boot Primary CPU
   - Boot from EL2
   - Boot from EL1
   - Boot to C runtime for OS Initialize
   - Ready to Boot Second CPU
   - Boot from EL2
   - Boot from EL1
   - Boot to C runtime for OS Initialize
   nsh> ps
     TID   PID  PPID CPU PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK    
        STACK    USED FILLED COMMAND
       0     0     0   0   0 FIFO     Kthread   - Assigned           
0000000000000000 0016336 0001568   9.5%  CPU0 IDLE
       1     0     0   1   0 FIFO     Kthread   - Running            
0000000000000000 0016336 0000992   6.0%  CPU1 IDLE
       2     0     0 --- 192 RR       Kthread   - Waiting  Semaphore 
0000000000000000 0016272 0001184   7.2%  hpwork 0x4046dbd0 0x4046dc58
       4     4     0 --- 100 RR       Task      - Waiting  Semaphore 
0000000000000000 0016304 0002032  12.4%  init_main
       5     5     4   0 100 RR       Task      - Running            
0000000000000000 0016304 0003216  19.7%  sh
   nsh> getprime
   Set thread priority to 10
   Set thread policy to SCHED_RR
   Start thread #0
   thread #0 started, looking for primes < 10000, doing 10 run(s)
   thread #0 finished, found 1230 primes, last one was 9973
   Done
   getprime took 82 msec
   ```
   
   Both CPUs come up and `getprime` completes with the expected prime count, so
   the D-Cache is coherent on the secondary path.
   
   ### checkpatch
   
   ```text
   $ tools/checkpatch.sh -c -u -m -g ed122374690..HEAD
   Used config files:
       1: .codespellrc
   arch/arm64/src/a527/a527_boot.c:93:6: error: Missing blank line after 
declarations
   Some checks failed.
   ```
   
   This one report is **pre-existing and unrelated**: line 93 is
   `if (dest <= src)` inside `a527_copy_overlap()`, which is unchanged by this
   series (identical at `apache/master`). checkpatch scans the whole file once 
the
   file is touched. All newly added lines pass.
   
   ## Change size
   
   ```text
    arch/arm64/src/a527/a527_boot.c         | 29 +---------------
    arch/arm64/src/a64/a64_boot.c           | 32 ------------------
    arch/arm64/src/am62x/am62x_boot.c       | 29 ----------------
    arch/arm64/src/bcm2711/bcm2711_boot.c   | 27 ---------------
    arch/arm64/src/common/arm64_boot.c      | 59 
+++++++++++++++++++++++++++++++++
    arch/arm64/src/common/arm64_cpustart.c  |  4 +++
    arch/arm64/src/common/arm64_mmu.c       | 12 ++-----
    arch/arm64/src/common/arm64_mpu.c       |  5 +--
    arch/arm64/src/fvp-v8r/fvp_boot.c       | 23 -------------
    arch/arm64/src/goldfish/goldfish_boot.c | 31 -----------------
    arch/arm64/src/imx8/imx8_boot.c         | 24 +-------------
    arch/arm64/src/imx9/imx9_boot.c         | 30 +----------------
    arch/arm64/src/qemu/qemu_boot.c         | 37 ---------------------
    arch/arm64/src/rk3399/rk3399_boot.c     | 32 ------------------
    arch/arm64/src/rk3588/rk3588_boot.c     | 17 ----------
    arch/arm64/src/zynq-mpsoc/zynq_boot.c   | 40 ----------------------
    16 files changed, 69 insertions(+), 362 deletions(-)
   ```
   
   Net −293 lines, with no new public API and no behaviour change on any
   configuration that keeps the D-Cache enabled.
   
   ## Commits
   
   ```text
   b63e4ec83f2 arch/arm64: consolidate common boot sequence into 
arm64_boot_primary_c_routine
   117248aad55 arch/arm64: disable dcache during MPU/MMU initialization
   ```
   
   The two are kept separate because the second one's effect is only meaningful
   once the cache-affecting calls live in one place. Splitting them also makes 
it
   possible to bisect a cache-coherency regression against the pure refactor.
   
   ## Honest notes
   
   - **The patch does not itself call `up_disable_dcache()`.** The commit 
message
     says initialization is "bracketed" by disable/enable, but the disable side 
is
     implicit: `arm64_boot_el1_init()` already leaves `SCTLR_C_BIT` clear before
     `arm64_boot_primary_c_routine()` runs, and `arm64_mpu_disable()` no longer
     clears it either. So the cache is off on entry by construction rather than 
by
     an explicit call here. Anyone refactoring `arm64_boot_el1_init()` later 
needs
     to preserve that.
   - **Cache-ordering issue is argued, not reproduced.** I did not construct a
     failing case for "stale cache entries during memory protection 
transitions".
     The change is justified by making the invariant explicit and 
single-sourced,
     not by a captured crash.
   - **`imx9:bootloader` with `CONFIG_ARCH_USE_MMU=n`** keeps `ARCH_HAVE_MMU=y`,
     so the MMU carve-out depends on `CONFIG_ARCH_CHIP_IMX9` matching. A new SoC
     wanting the same behaviour would need its own clause.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to