PiyushPatle26 opened a new issue, #20582:
URL: https://github.com/apache/nuttx/issues/20582
### Description / Steps to reproduce the issue
In RISC-V SMP kernel builds, unmapping user pages invalidates the TLB only
on the hart performing the unmap. Other harts running threads of the same
address environment can retain a valid translation to the unmapped physical
page.
I reproduced this on rv-virt:ksmp64 under QEMU. After shmdt() and
shmctl(IPC_RMID), a reader thread on another hart continued accessing the
unmapped shared-memory page. When the physical page was reallocated to a writer
process, the stale reader observed the writer's data through the old mapping.
In three first-run-after-boot tests, the reader observed the new writer's
0xb0b0b0b0 marker in all 1,024 words of the page. The same-hart control faults
as expected, and a proof of concept using SBI remote SFENCE.VMA makes the
cross-hart reader fault as expected.
This demonstrates cross-process data exposure through a stale translation in
the tested configuration. The experiment has so far been conducted in QEMU, not
on physical hardware.
Root cause
The shared-memory unmap path reaches a local-only TLB invalidation:
```
shmdt() mm/shm/shmdt.c
munmap_shm() mm/shm/shmat.c
up_shmdt() arch/risc-v/src/common/riscv_addrenv_shm.c
riscv_unmap_pages() arch/risc-v/src/common/riscv_addrenv_utils.c
mmu_ln_clear() arch/risc-v/src/common/riscv_mmu.c
sfence.vma x0, va riscv_mmu.h:284, local hart only
```
SFENCE.VMA invalidates translations on the executing hart; it does not
invalidate the corresponding entries on other harts. The remote harts are not
notified when the page table is changed.
The SBI RFENCE function IDs are defined in riscv_sbi.h, but the current
RISC-V code does not use them for this path.
The same riscv_unmap_pages() function is also used by:
- User pgmap unmapping (riscv_addrenv_pgmap.c).
- Kernel kmap unmapping (up_addrenv_kunmap_pages()).
These paths therefore warrant investigation for the same local-only
invalidation problem.
### Steps to reproduce
Environment
- NuttX configuration: rv-virt:ksmp64
- Build options: CONFIG_BUILD_KERNEL=y, CONFIG_SMP=y
- QEMU: 8.2.2
- Firmware: OpenSBI v1.3
- Toolchain: riscv64-unknown-elf-gcc
- NuttX revisions tested: 5fda288a2a and 284d4d4ea4 (NuttX-13.1.0)
- Four RISC-V harts, running in S-mode
#### Reproducer
The test application is testing/mm/tlbstale, available on these branches:
- NuttX: https://github.com/PiyushPatle26/nuttx/tree/riscv-smp-stale-tlb
- NuttX apps:
https://github.com/PiyushPatle26/nuttx-apps/tree/riscv-smp-stale-tlb
The NuttX branch includes a change needed to link the test application in a
kernel build; details are in the notes below.
To build and run it (with `nuttx` and `apps` checked out side by side):
```bash
cd nuttx
./tools/configure.sh -l rv-virt:ksmp64
kconfig-tweak --file .config --enable CONFIG_MM_SHM \
--set-val CONFIG_ARCH_SHM_VBASE 0xC2000000 \
--set-val CONFIG_ARCH_SHM_NPAGES 16 \
--set-val CONFIG_ARCH_SHM_MAXREGIONS 4 \
--enable CONFIG_TESTING_TLBSTALE
make olddefconfig
make -j$(nproc) && make export -j$(nproc)
cd ../apps
./tools/mkimport.sh -z -x ../nuttx/nuttx-export-*.tar.gz
make import -j$(nproc)
cd ../nuttx
qemu-system-riscv64 -semihosting -M virt,aclint=on -cpu rv64 -smp 4 -kernel
nuttx -nographic
nsh> tlbstale -x # as the first command after boot
nsh> tlbstale -l # control
```
##### How the reproducer works
1. The main thread is pinned to CPU0. It creates a one-page shared-memory
segment and fills it with 0x5a5aa5a5.
2. A reader thread in the same process is pinned to CPU1 and continuously
reads the page.
3. With -x, three writer processes are created first and blocked on SIGUSR1.
4. The main thread calls shmdt() and shmctl(IPC_RMID), unmapping the page
and returning it to the page pool. It then publishes a flag with release
semantics and signals the writers.
5. Writer 0 performs shmget() and obtains the freed page. Its allocation
zeroes the page, after which it fills the page with 0xb0b0b0b0.
6. The reader observes the flag with acquire semantics and continues
scanning the virtual address through its stale translation. On a correctly
invalidated mapping, its first post-unmap access should fault instead.
The -l option runs the reader on the same hart as the unmapper to provide a
control case.
### Results
| Case | Result |
|---|---|
| `tlbstale` (reader on CPU1), 3 runs | stale: ~39.9M successful loads after
`shmdt()` returned, every run |
| `tlbstale -c 2`, `-c 3` | stale: 39.99M / 39.17M loads |
| **`tlbstale -x`, first run after a fresh boot, 3 of 3 boots (incl. the
published branches)** | **leak: all 1024 words went `0x5a5aa5a5` → `0` →
`0xb0b0b0b0`, i.e. the reader saw writer 0's `shmget()` zero the reused page
and then its marker, ~53M stale loads** |
| `tlbstale -x`, later runs in the same boot | stale, but the freed page
wasn't reallocated within the window (see notes) |
| `tlbstale -l` (control, same hart) | load page fault, SIGSEGV (correct) |
| PoC: SBI `remote_sfence_vma` added to `riscv_unmap_pages()` | reader on
CPU1 / CPU3 faults like the control (correct) |
```
nsh> tlbstale -x
tlbstale: reader on CPU1, main on CPU0 (cross-CPU unmap, cross-process reuse)
tlbstale -w0: filled 16 shm pages and 256 heap blocks with 0xb0b0b0b0
...
tlbstale: FAIL: the reader on CPU1 made 53014550 loads after shmdt()
returned.
tlbstale: scanned the page 51772 times: 1024 of 1024 words changed after the
free, 1024 held the writer's marker 0xb0b0b0b0
tlbstale: load #27881 word 210 read 0x00000000 (unknown owner's data)
...
tlbstale: last view of the page: 0 words our old data, 1024 the writer's
marker, 0 zero, 0 other
tlbstale: LEAK: the freed page was reallocated to a writer process, and this
process read the writer's data
nsh> tlbstale -l
tlbstale: reader on CPU0, main on CPU0 (control: same CPU)
[CPU0] riscv_exception: EXCEPTION: Load page fault. MCAUSE:
000000000000000d, EPC: 00000000c000014e, MTVAL: 00000000c2000b90
[CPU0] riscv_fault_handler: Segmentation fault in tlbstale (PID 29: tlbstale)
```
#### Proof of concept
The diagnostic PoC adds an SBI remote `SFENCE.VMA` call to
`riscv_unmap_pages()`.
It broadcasts to all harts unconditionally, does not probe for RFENCE
support, has no IPI fallback, and does not support NuttSBI. It confirms the
suspected cause in the tested OpenSBI configuration but is not proposed as a
production fix.
#### Proposed direction
Following the discussion on Discord, I suggest addressing this in two stages.
1. Fix RISC-V SMP correctness first
Ensure that riscv_unmap_pages() does not return until the necessary remote
invalidations have completed. Use SBI RFENCE when the firmware advertises the
required extension and provide a synchronous IPI-based fallback for
configurations without RFENCE, including NuttSBI. The fallback must be reviewed
for deadlocks involving critical sections and locks.
2. Discuss a generic TLB-maintenance contract separately
A synchronous architecture-independent interface could allow common unmap
paths to express the required completion semantics while architecture backends
implement them appropriately. ARM/ARM64 could use their existing broadcast
maintenance, while x86_64 would need to ensure remote invalidation has
completed before returning. This interface would also provide a foundation for
future RISC-V ASID support.
I would welcome feedback on whether this separation is appropriate and on
the best place to establish the generic interface.
### On which OS does this issue occur?
[OS: Linux]
### What is the version of your OS?
Ubuntu 24
### NuttX Version
13.0.1
### Issue Architecture
[Arch: risc-v]
### Issue Area
[Area: Memory Management]
### Host information
_No response_
### Verification
- [x] I have verified before submitting the report.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
