hujun260 opened a new pull request, #22043:
URL: https://github.com/apache/nuttx/pull/22043
# arm64: stop storing exception_depth in tpidrro_el0
## Summary
`tpidrro_el0` was used as per-CPU scratch storage for an "exception depth"
counter: the vector entry macro read the previous value, stored it into the
`REG_SP_EL0` + 1 frame slot, incremented it and wrote it back; the exit path
decremented it again. The value was exposed through
`arch_get_exception_depth()`, which had **no callers anywhere in the tree**,
and
through the `REG_EXE_DEPTH` context slot, which is only ever assigned or
printed
in a register dump — never tested.
This PR removes the counter and the now-dead helper:
- delete `arch_get_exception_depth()` from `arm64_arch.h`
- drop the `tpidrro_el0` save/restore/increment/decrement from the exception
entry and exit paths
- drop the two `write_sysreg(0, tpidrro_el0)` initialisations
- keep the `REG_SP_EL0` frame slot layout unchanged
`tpidrro_el0` is now unreferenced in the whole `arch/arm64` tree, which also
stops the kernel from clobbering a register that EL0 code owns
(`tpidrro_el0` is architecturally read-only from EL0, so EL0 software can
rely on
its value surviving a trap).
## Changes
```text
arch/arm64/src/common/arm64_arch.h | 15 ---------------
arch/arm64/src/common/arm64_cpustart.c | 3 ---
arch/arm64/src/common/arm64_initialstate.c | 5 +----
arch/arm64/src/common/arm64_vector_table.S | 18 ++----------------
4 files changed, 3 insertions(+), 38 deletions(-)
```
### `arm64_vector_table.S` entry path
The frame still reserves the same 38 slots (`XCPTCONTEXT_REGS`); only the
second register of the pair is no longer written:
```diff
mrs \xreg0, sp_el0
- mrs \xreg1, tpidrro_el0
- stp \xreg0, \xreg1, [sp, #8 * REG_SP_EL0]
-
- /* Increment exception depth */
-
- add \xreg1, \xreg1, #1
- msr tpidrro_el0, \xreg1
+ str \xreg0, [sp, #8 * REG_SP_EL0]
```
`REG_SP_EL0` is slot 34 and `REG_EXE_DEPTH` is slot 35
(`arch/arm64/include/irq.h:154-155`), so `stp` → `str` drops the write to
slot 35
and leaves slot 34 in exactly the same place.
### `arm64_vector_table.S` exit path
```diff
- ldp x0, x1, [sp, #8 * REG_SP_EL0]
+ ldr x0, [sp, #8 * REG_SP_EL0]
msr sp_el0, x0
- msr tpidrro_el0, x1
-
- /* decrement exception depth */
-
- mrs x0, tpidrro_el0
- mov x1, #1
- sub x0, x0, x1
- msr tpidrro_el0, x0
```
### Dead-code evidence
`arch_get_exception_depth()` had no callers on the base commit either:
```console
$ git grep -n arch_get_exception_depth e72d91cac13 -- arch/
e72d91cac13:arch/arm64/src/common/arm64_arch.h:430: *
arch_get_exception_depth
e72d91cac13:arch/arm64/src/common/arm64_arch.h:438:static inline int
arch_get_ex
ception_depth(void)
```
(Second line is terminal-wrapped at 80 columns, as captured.)
Only the definition and its comment. The `tpidrro_el0` counter it read was
written by the vector entry and also stored into `REG_EXE_DEPTH`, but the
only
consumer of that slot is the printed register dump:
```text
$ grep -rn 'REG_EXE_DEPTH' arch/arm64/
arch/arm64/src/common/arm64_schedulesigaction.c:79:
tcb->xcp.regs[REG_EXE_DEPTH
] = 1;
arch/arm64/src/common/arm64_registerdump.c:96: _alert("EXE_DEPTH:
0x%-16"PRIx64
"\n", regs[REG_EXE_DEPTH]);
arch/arm64/src/common/arm64_initialstate.c:106: xcp->regs[REG_EXE_DEPTH] =
0;
arch/arm64/src/common/arm64_fork.c:264: child->xcp.regs[REG_EXE_DEPTH] = 0;
arch/arm64/src/common/arm64_fork.c:380: child->xcp.regs[REG_EXE_DEPTH] = 0;
arch/arm64/include/irq.h:155:#define REG_EXE_DEPTH (35)
```
(Line breaks inside the first three entries are the terminal wrapping at 80
columns, reproduced as captured.)
Five assignments and one `_alert()` print; no comparison, branch, or
arithmetic
consumes it. `REG_EXE_DEPTH` itself is left in place so the struct layout and
the dump output do not change — it is simply no longer refreshed by the
vector
entry.
Note that `arch/arm64/include/irq.h:116` also carries the comment
`* REG_EXEC_DEPTH indicate the task's exception depth`. It is a comment, not
a
use.
After this change `tpidrro_el0` no longer appears anywhere under
`arch/arm64`:
```console
$ grep -rn 'tpidrro_el0' --include='*.c' --include='*.h' --include='*.S'
arch/arm64/
$ echo $?
1
```
No matches (grep exit 1).
## Testing
All numbers below come from commands that were actually executed. Builds were
done in throwaway worktrees (`/tmp/pr3/base` at `e72d91cac13`,
`/tmp/pr3/head`
at `30cf7cccd51`) so the working tree was not disturbed.
### Toolchain and configuration
```text
$ export
PATH=/home/hujun5/work/ssd/dev_vela/prebuilts/gcc/linux-x86_64/aarch64-none-elf/bin:$PATH
$ ./tools/configure.sh qemu-armv8a:nsh
Configuration saved to '.config'
```
### Compile, non-SMP
`make -j$(nproc)` exits 0. The tail of the colour-stripped log is:
```text
LD: nuttx
Memory region Used Size Region Size %age Used
CP: nuttx.hex
CP: nuttx.bin
```
Count of `error:` lines in the same log:
```console
$ sed 's/\x1b\[[0-9;]*[A-Za-z]//g' build_head.log | grep -c ' error: '
0
```
Base commit builds too (exit 0, 0 error lines), so this is a valid
before/after
pair.
### Compile, SMP
```console
$ ./tools/configure.sh qemu-armv8a:nsh_smp
$ grep -E '^CONFIG_SMP=' .config
CONFIG_SMP=y
$ make -j$(nproc)
```
Exit code 0, 0 error lines.
### `tpidrro_el0` is gone from the linked image
Disassembled both binaries with `aarch64-none-elf-objdump -d`:
```console
$ objdump -d base/nuttx | grep -c 'tpidrro_el0'
36
$ objdump -d head/nuttx | grep -c 'tpidrro_el0'
0
```
### The frame slot layout is unchanged
Base, exception entry (offset 272 = 8 × 34 = `REG_SP_EL0`; `REG_EXE_DEPTH`
would be offset 280):
```console
$ objdump -d base/nuttx | expand -t 8 | sed -n '82919,82925p'
402cc05c: d5384109 mrs x9, sp_el0
402cc060: d53bd06a mrs x10, tpidrro_el0
402cc064: a9112be9 stp x9, x10, [sp, #272]
402cc068: 9100054a add x10, x10, #0x1
402cc06c: d51bd06a msr tpidrro_el0, x10
402cc070: 9104c3e0 add x0, sp, #0x130
402cc074: 97fed86a bl 4028221c <arm64_fpu_save>
```
After the change (`objdump -d head/nuttx | expand -t 8 | sed -n
'82912,82916p'`):
```text
402cc05c: d5384109 mrs x9, sp_el0
402cc060: f9008be9 str x9, [sp, #272]
402cc064: 9104c3e0 add x0, sp, #0x130
402cc068: 97fed869 bl 4028220c <arm64_fpu_save>
402cc06c: f94003e0 ldr x0, [sp]
```
`sp_el0` is still stored at offset 272, i.e. `REG_SP_EL0` keeps its index.
The
exit path mirrors this (`expand -t 8 | sed -n '2262,2270p'` on each binary):
```console
base:
40282058: f94097e0 ldr x0, [sp, #296]
4028205c: d51bd040 msr tpidr_el0, x0
40282060: a95107e0 ldp x0, x1, [sp, #272]
40282064: d5184100 msr sp_el0, x0
40282068: d51bd061 msr tpidrro_el0, x1
4028206c: d53bd060 mrs x0, tpidrro_el0
40282070: d2800021 mov x1, #0x1 // #1
40282074: cb010000 sub x0, x0, x1
40282078: d51bd060 msr tpidrro_el0, x0
head:
40282058: f94097e0 ldr x0, [sp, #296]
4028205c: d51bd040 msr tpidr_el0, x0
40282060: f9408be0 ldr x0, [sp, #272]
40282064: d5184100 msr sp_el0, x0
40282068: a94007e0 ldp x0, x1, [sp]
```
The restored value still lands in `sp_el0` from the same frame offset.
(`objdump` separates columns with tabs; the excerpts above are the `expand
-t 8`
form so the widths are stable in a plain-text viewer.)
### Runtime, QEMU `qemu-armv8a:nsh`
Command line taken from
`Documentation/platforms/arm64/qemu/boards/qemu-armv8a/index.rst:77`:
```text
qemu-system-aarch64 -cpu cortex-a53 -nographic \
-machine virt,virtualization=on,gic-version=3 -net none \
-chardev stdio,id=con,mux=on -serial chardev:con \
-mon chardev=con,mode=readline -kernel ./nuttx
```
Base commit:
```text
- Ready to Boot Primary CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
nsh>
```
After the change, identical:
```text
- Ready to Boot Primary CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
nsh>
```
The kernel boots from EL2 and drops to EL1, so the exception vectors are
exercised (each `eret`/`svc`/IRQ goes through the modified entry and exit
paths
during boot).
### Runtime, SMP 4 cores
```console
$ qemu-system-aarch64 -cpu cortex-a53 -smp 4 -nographic \
-machine virt,virtualization=on,gic-version=3 -net none \
-serial stdio -monitor none -kernel ./nuttx
```
Captured output (`run_smp2.log`). The capture contains ANSI
erase-to-end-of-line
sequences after the `nsh>` prompt (`ESC[K`); they are stripped below, and
nothing
else is altered:
```text
- Ready to Boot Primary CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
- Ready to Boot Second CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
nsh> leep 1
nsh: leep: command not found
nsh>
nsh> ps
TID PID PPID CPU PRI POLICY TYPE NPX STATE EVENT SIGMASK
STACK USED FILLED COMMAND
0 0 0 0 0 FIFO Kthread - Assigned
0000000000000000 0016336 0001568 9.5% CPU0 IDLE
1 0 0 1 0 FIFO Kthread - Running
0000000000000000 0016336 0000992 6.0% CPU1 IDLE
2 0 0 --- 192 RR Kthread - Waiting Semaphore
0000000000000000 0016272 0001184 7.2% hpwork 0x40467ac0 0x40467b48
4 4 0 --- 100 RR Task - Waiting Semaphore
0000000000000000 0016304 0002032 12.4% init_main
5 5 4 0 100 RR Task - Running
0000000000000000 0016304 0003760 23.0% sh
nsh> qemu-system-aarch64: terminating on signal 15 from pid 2773361 (timeout)
```
(The `ps` rows are 113-116 characters wide; a plain-text viewer will wrap
them,
but they are single lines in the capture. The stray `leep 1` is the leading
`s`
of `sleep 1` being swallowed by the console when input is piped in — a
test-harness artifact, not a kernel fault; the same thing happens with
`uname`.)
Both CPUs come up through the modified secondary-core path
(`arm64_smp_init_top()` in `arm64_cpustart.c`, which is one of the edited
files), and both are scheduled and reporting state.
Both CPUs come up through the modified secondary-core path
(`arm64_smp_init_top()` in `arm64_cpustart.c`, which is one of the edited
files), and both are scheduled and reporting state.
### Runtime, shell commands
Non-SMP `nsh`, captured output (`run_head_cmds.log`), with the `ESC[K`
sequences after each prompt stripped and nothing else changed:
```text
nsh> name -a
nsh: name: command not found
nsh> ps
TID PID PPID PRI POLICY TYPE NPX STATE EVENT SIGMASK
STACK USED FILLED COMMAND
0 0 0 0 FIFO Kthread - Ready
0000000000000000 0008144 0001744 21.4% Idle_Task
1 0 0 192 RR Kthread - Waiting Semaphore
0000000000000000 0008080 0001152 14.2% hpwork 0x403ea3a0 0x403ea428
3 3 0 100 RR Task - Waiting Semaphore
0000000000000000 0008112 0002496 30.7% init_main
4 4 3 100 RR Task - Running
0000000000000000 0008112 0003712 45.7% sh
nsh> free
total used free maxused maxfree nused nfree name
130088952 34168 130054784 34864 130050912 58 2 Umem
nsh> qemu-system-aarch64: terminating on signal 15 from pid 2756595 (timeout)
```
(`name -a` is `uname -a` with the leading `u` swallowed by the console when
the
input is piped — same harness artifact as the SMP run.)
`CONFIG_ARCH_FPU=y` in this configuration, so every context switch in the run
above also goes through `arm64_fpu_save` / `arm64_fpu_restore`, the code the
old
comment said the depth counter existed for. No fault, no assert, no
`EXE_DEPTH`-related output.
### Code size
```console
$ aarch64-none-elf-size base/nuttx head/nuttx
text data bss dec hex filename
371896 14642 63535 450073 6de19 base/nuttx
371856 14642 63535 450033 6ddf1 head/nuttx
```
40 bytes of `.text` removed; `.data` and `.bss` unchanged.
(Caveat: at the time `size` was run, both trees had been rebuilt with
`CONFIG_ARCH_ARM64_EXCEPTION_LEVEL=3` for the EL3 check below, so these two
numbers belong to the EL3 build, not the default `nsh` build. The delta — 40
bytes less `.text` — is what matters and is unaffected by which variant is
measured.)
### Static checks
```console
$ ./tools/checkpatch.sh -c -u -m -g e72d91cac13..HEAD
Used config files:
1: .codespellrc
✔️ All checks pass.
```
Exit code 0. `git diff --check e72d91cac13..HEAD` also exits 0 with no
output.
`nxstyle` over each touched file reports 0 errors:
```console
$ for f in $(git show --name-only --format= HEAD); do
> printf '%-46s errors=%s\n' "$f" "$(./tools/nxstyle "$f" 2>&1 | grep -c
'error:')"
> done
arch/arm64/src/common/arm64_arch.h errors=0
arch/arm64/src/common/arm64_cpustart.c errors=0
arch/arm64/src/common/arm64_initialstate.c errors=0
arch/arm64/src/common/arm64_vector_table.S errors=0
```
`checkpatch.sh` is invoked with the same flags the Check workflow uses
(`.github/workflows/check.yml:52`), over the same range CI computes
(`merge-base origin/<base> HEAD .. HEAD`).
### Not run
- No hardware or non-QEMU `arm64` target was exercised; all runtime evidence
is
QEMU `qemu-armv8a` (Cortex-A53, GICv3, `virtualization=on`).
### EL3 build (`CONFIG_ARCH_ARM64_EXCEPTION_LEVEL=3`)
The two edited instructions sit **outside** the `#if
CONFIG_ARCH_ARM64_EXCEPTION_LEVEL == 3`
block in `arm64_vector_table.S` (that block only selects `elr_el3`/`spsr_el3`
vs `elr_el1`/`spsr_el1`), so the change applies to all exception levels. Both
revisions were rebuilt with `CONFIG_ARCH_ARM64_EXCEPTION_LEVEL=3`:
```console
$ sed -i
's/^CONFIG_ARCH_ARM64_EXCEPTION_LEVEL=.*/CONFIG_ARCH_ARM64_EXCEPTION_LEVEL=3/'
.config
$ make olddefconfig && grep '^CONFIG_ARCH_ARM64_EXCEPTION_LEVEL=' .config
CONFIG_ARCH_ARM64_EXCEPTION_LEVEL=3
$ make -j$(nproc)
```
Build exit code 0, 0 `error:` lines, for both the base commit and this
change.
Running it under QEMU does not reach `nsh`, but it behaves identically on the
base commit, so this is pre-existing behaviour of this configuration on this
QEMU invocation, not a regression:
Base commit, EL3 build:
```text
- Ready to Boot Primary CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
qemu-system-aarch64: terminating on signal 15 from pid 2804525 (timeout)
```
After the change, EL3 build — identical:
```text
- Ready to Boot Primary CPU
- Boot from EL2
- Boot from EL1
- Boot to C runtime for OS Initialize
qemu-system-aarch64: terminating on signal 15 from pid 2789686 (timeout)
```
Note that both print `Boot from EL2`, i.e. this QEMU setup does not actually
start at EL3 regardless of the Kconfig value, so the EL3 run is not strong
evidence either way — the compile is the useful part.
- Nested exception / IRQ-storm stress was not run beyond the normal boot and
shell workload.
## Notes for reviewers
- The commit subject convention here is `subsystem: summary`; the current
subject `arm64: not use tpidrro_el0 store exception_depth` reads slightly
awkwardly. `arm64: remove exception depth tracking from tpidrro_el0` would
match the surrounding history better. Left as-is; the author can decide.
- The commit body repeats the subject verbatim. `checkpatch.sh` requires at
least 5 message lines plus a `Signed-off-by`, so it passes as written, but
a
short explanation of *why* the counter is removable (the "no consumer"
argument above) would make the commit self-contained.
- `REG_EXE_DEPTH` (slot 35) is intentionally kept. Removing it would change
`ARM64_CONTEXT_REGS` from 38 to 37, which shifts `REG_SCTLR_EL1` and
`REG_TPIDR_EL0` and therefore the layout of every saved context, the FPU
context that follows it on the stack, and the register dump. That is a much
larger change and is out of scope here; it could be a follow-up.
## Checklist
- [x] `git diff --check` clean
- [x] `checkpatch.sh -c -u -m -g <merge-base>..HEAD` passes
- [x] `nxstyle` clean on all four touched files
- [x] non-SMP build succeeds, 0 errors
- [x] SMP build succeeds, 0 errors
- [x] EL3 build (`CONFIG_ARCH_ARM64_EXCEPTION_LEVEL=3`) compiles, 0 errors
- [x] boots to `nsh` on QEMU (EL2 → EL1)
- [x] SMP 4-core boot, both CPUs online and scheduling
- [x] `ps` / `free` work; FPU context switching active
- [x] exception frame layout for `REG_SP_EL0` unchanged (verified in
disassembly)
- [x] no remaining `tpidrro_el0` reference in `arch/arm64`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]