casaroli opened a new pull request, #22069:
URL: https://github.com/apache/nuttx/pull/22069

   ## Summary
   
   When a thread ends its process (`exit()`, `_exit()` or a fatal signal), 
`group_kill_children()` cancels the other threads, and `nxtask_terminate()` 
ends each of them where it is. A thread in a system call can hold locks and 
references there, and kernel objects can be on its kernel stack (a request that 
a device completes, a callback that a driver calls). Nothing gives them up: a 
lock stays held for ever, a file is never closed, or a driver later writes into 
the freed stack. With the default `CONFIG_CANCELLATION_POINTS=n` this happens 
to every thread that is in a system call when its process ends.
   
   This PR ends such a thread only when its system call is over. Each commit 
fixes one thing:
   
   1. **`libs/libc: Check the lock of the task information.`** The kernel copy 
of libc takes `ta_lock` (`getenv()` for `$PWD` in the inode search, `getcwd()`, 
`lib_getfullpath()`, `env_dup()`, `env_sanitize_secure()`, the `setenv()` 
family, `pthread_atfork()`, `fork()`). Most callers ignored the result of 
`nxrmutex_lock()`, and then unlocked a lock that they did not hold. Now each 
caller checks it. With commit 3 the lock fails with `ECANCELED` for a thread 
that its process end canceled; a relative path lookup then fails instead of 
using the home directory.
   2. **`sched/pthread: Leave the join queue when pthread_join() is 
canceled.`** A cancellation ends the wait of `pthread_join()` with `ECANCELED` 
(today with `CONFIG_CANCELLATION_POINTS`, and with commit 3 at the end of the 
process). `pthread_join()` ignored it and left the caller in the join queue of 
the other thread. When the caller ended first, `pthread_completejoin()` later 
walked into its freed TCB. Now the caller leaves the queue and `pthread_join()` 
returns the error.
   3. **`sched/task: End a thread in a system call when the call is over.`** 
For the forced cancellation of an ending process, a thread in a system call 
(`TCB_FLAG_SYSCALL`) is marked in a new TCB field, `kcancel`, and not ended. A 
wait that a signal ends, and a wait for a mutex in user memory (its holder can 
have ended with the process), end with `ECANCELED`. A wait that no signal ends 
(`TCB_FLAG_UNINTERRUPTIBLE`) and a wait for a kernel mutex complete first. The 
marked thread does not start such a wait again. `nxsig_unmask_pendingsignal()`, 
which the architecture calls at the end of each system call, then ends the 
thread; it holds nothing of the kernel there. `nxtask_terminate()` and its SMP 
call check again, because the thread can enter a system call after 
`nxnotify_cancellation()` looked at it. `group_kill_children()` walks the 
member list again after each cancellation, because a woken member can end and 
leave the list. A thread that ends by itself at the same time (`_exit()`, `p
 thread_exit()`) is left to that end: it is not marked, and the SMP call keeps 
its `TCB_FLAG_EXIT_PROCESSING`, which the thread can have set itself and which 
`nxtask_exithook()` needs.
   4. **`Documentation: The end of a thread that is in a system call.`** 
`cancellation_points.rst`.
   
   This needs an architecture that ends a system call in the context of the 
thread. ARM64 and RISC-V do (`dispatch_syscall()`) and select the new 
`CONFIG_ARCH_HAVE_SYSCALL_THREAD_RETURN`. The other architectures with system 
calls (ARMv6-M/7-M/8-M, ARMv7-A/R, ARMv8-R, MIPS, MISOC, Xtensa) end a system 
call in the SVC exception handler, where the thread cannot end; they keep the 
old behaviour. x86_64 does not set `TCB_FLAG_SYSCALL` (see the note in 
`x86_64_syscall()`), so it keeps the old behaviour too.
   
   ## Impact
   
   PROTECTED and KERNEL builds on ARM64 and RISC-V:
   
   - A thread in a system call ends when the call returns, not inside it. No 
cleanup handler of the canceled thread runs (the process ends as with 
`_exit()`). The exit status of the process does not change.
   - A thread that waits in the kernel for a device that never answers keeps 
its process from its end, as a single-threaded process is kept today.
   - `nxsig_timedwait()` takes a canceled wait (`SIG_CANCEL_TIMEOUT`) also 
without `CONFIG_CANCELLATION_POINTS`.
   - An asynchronous `pthread_cancel()` still ends a thread at once.
   - One byte in `struct tcb_s`. Text, measured on an earlier version of this 
PR without the rule for a thread in its own end (a few more instructions): 
qemu-armv8a:knsh +1024 bytes, rv-virt:knsh64 +860, rv-virt:pnsh64 (kernel part) 
+1016.
   
   All builds:
   
   - Commit 1: when the environment lock fails, `getenv()` returns NULL with 
`errno`, `setenv()` and `fork()` return -1. `env_sanitize_secure()` returns 
`int` now (internal; one caller, `exec_module()`).
   - Commit 2: `pthread_join()` returns `ECANCELED` when a cancellation ended 
its wait, instead of 0 with an old exit value.
   - FLAT builds and the other architectures: no change of behaviour beyond 
commits 1 and 2. Text (the same earlier version): qemu-armv8a:nsh +340 bytes, 
mps2-an500:citest_protected (kernel part) +168.
   
   ## Testing
   
   Test programs (standalone, from `/system/bin` over 9p, qemu-armv8a under 
HVF, `CONFIG_MM_FILL_ALLOCATIONS=y`):
   
   - `sysexit OP 40 DIR MODE`: 40 rounds; each round forks a child with 3 
worker threads in one kind of system call, and the child ends after a random 
time up to 100 ms: its main thread calls `_exit(7)` (exit), the parent sends 
`SIGKILL` (kill), or a fourth thread calls `_exit(7)` (texit). After each round 
the parent uses the same kind of object (a lock that stays held stops it). It 
checks the exit status and the page pool and kernel heap of `/proc/meminfo`. 
OP: `tmpfile` (open, write, close, unlink; in a tmpfs and in the 9p root), 
`pipe`, `local` (AF_UNIX socketpair), `mmap`, `poll`, `semwait`, `mutex` 
(`pthread_mutex_lock()` of a mutex that the main thread holds), `sleep`, `net` 
(UDP `recv()`), `tty` (pty `read()`), `join` (`pthread_join()` of a thread in 
`pause()`), `vfork`, `spin` (user code only).
   - `exitrace ROUNDS MODE`: each round forks a child with 3 threads that wait 
for a start flag. Then the main thread calls `_exit(7)`, and at the same moment 
(a random spin of a few microseconds) each thread calls `pthread_exit()` 
(pexit) or `_exit(7)` (texit). On SMP the end of the process then cancels 
threads that are in their own end.
   - `ptyintr`: see below.
   
   qemu-armv8a:knsh, 1 CPU, master 3e66f0a9b4c vs this PR:
   
   | Test | master | this PR |
   |---|---|---|
   | ostest | status 0 | status 0 |
   | sysexit tmpfile (tmpfs) exit / kill / texit | PASS / hang / not run | PASS 
/ PASS* / PASS |
   | sysexit tmpfile (9p root) | kernel fault in the virtio interrupt (nested 
exception) | PASS / PASS* / PASS |
   | sysexit pipe, local | hang in the first round | PASS / PASS* / PASS |
   | sysexit poll | kernel fault | PASS / PASS* / PASS |
   | sysexit mmap | `mm_granfree.c:69` assertion (page pool) | PASS / PASS* / 
PASS, pool +0 |
   | sysexit join | kernel fault | exit PASS, kill hangs (see below) |
   | sysexit vfork | kernel fault | PASS / PASS* / PASS |
   | sysexit mutex | exit PASS, kill hangs | exit PASS, kill hangs (see below) |
   | sysexit semwait, sleep, net, spin, tty | PASS / PASS* / PASS | PASS / 
PASS* / PASS |
   
   \* "kill" runs report "wrong status" in every round on master and with this 
PR: `waitpid()` gives exit status 0 for a child that `SIGKILL` ended. Not 
changed by this PR.
   
   The "join kill" and "mutex kill" hangs are the same on master: `SIGKILL` of 
the process goes to a thread in a wait that no signal ends (`pthread_join()`, a 
mutex), and stays pending there. Not changed by this PR.
   
   `ptyintr`: 40 children each open a pty pair, start a thread that blocks in 
`read()` of the slave, and `_exit()` after 20 ms. master: the kernel heap grows 
by 50 KiB, and after about 33 rounds no pty pair can be opened any more (the 
pairs of the ended processes are never freed). This PR: +128 bytes, all pairs 
freed (the same as a run with no reader).
   
   qemu-armv8a:knsh, 4 CPUs (`CONFIG_SMP`): ostest status 0 on both. sysexit: 
master as on 1 CPU. This PR: all `exit` and `texit` cases PASS with pool +0, 
except `vfork`, which corrupts the user heap of the child on master and with 
this PR alike (`mm_extend.c:91`). `tmpfile` in the 9p root needs the separate 
fix "fs/v9fs: Do not release the directory fid twice in unlink." (#22065): on 4 
CPUs, master and this PR fault in `fs/v9fs` without it (a fid used after it is 
freed), also with no end of a process; with it, `tmpfile` in the 9p root passes 
in all three modes.
   
   On 4 CPUs the `kill` mode of `tmpfile` in a tmpfs sometimes stops (4 of 11 
runs, with and without the rule of commit 3 for a thread in its own end; master 
stops in this case on 1 CPU already): the other threads of the child wait for 
the tmpfs lock, and the thread that holds it has ended. So in the `SIGKILL` 
path on SMP a thread can still end inside a system call. Not analyzed further. 
The `exit` and `texit` modes do not show it.
   
   `exitrace` on 4 CPUs: master: kernel fault in the first rounds 
(`group_foreachchild()` walks the freed TCB of a member that ended). This PR: 2 
x 1000 pexit rounds and 1000 texit rounds, all with the right exit status, in 
each of 2 boots. (On 1 CPU `exitrace` is too slow to be useful: its threads 
wait in busy loops.)
   
   With `CONFIG_DEBUG_ASSERTIONS`, fewer than 1 pexit round in 100 (26 in 4000) 
shows the assertion of `TCB_FLAG_EXIT_PROCESSING` in `nxtask_exithook()` (it 
does not stop the test): another CPU changes `tcb->flags` of the running thread 
with a plain read and write (`TCB_FLAG_FORCED_CANCEL` in 
`group_cancel_children_handler()`, `TCB_FLAG_JOIN_COMPLETED` in 
`pthread_completejoin()`) while the thread sets `TCB_FLAG_SYSCALL` and 
`TCB_FLAG_EXIT_PROCESSING` itself, and the write of one CPU removes the bits of 
the other. The thread was then in `pthread_exit()` with neither bit set. Master 
has the same race. Without the rule of commit 3 for a thread in its own end, 
466 rounds in 2000 showed it.
   
   qemu-armv8a:knsh, 1 CPU, `CONFIG_CANCELLATION_POINTS=y`: ostest status 0; 
sysexit as without it.
   
   qemu-armv8a:nsh (flat), 1 and 4 CPUs: ostest status 0 on master and with 
this PR.
   
   rv-virt:knsh64 (QEMU TCG, 1 hart): ostest status 0. sysexit `exit` PASS for 
mutex, pipe, semwait and join (master: pipe hangs). The `kill` mode hangs on 
RISC-V with master and with this PR alike. rv-virt:pnsh64 (protected), 1 and 4 
harts: ostest status 0.
   
   Builds in the NuttX CI image (`EXTRAFLAGS="-Wno-cpp -Werror"`), this PR: all 
OK, no compiler warnings (only the linker's RWX-segment note, also on master): 
qemu-armv8a:knsh (also with `CANCELLATION_POINTS`, `SMP`, 
`GROUP_KILL_CHILDREN_TIMEOUT_MS` -1 and 100, `DISABLE_ALL_SIGNALS`), 
qemu-armv8a:nsh, nsh_smp, rv-virt:knsh64, ksmp64, knsh (rv32), pnsh64 (also 
with `CANCELLATION_POINTS`, `SMP`, `ENABLE_PARTIAL_SIGNALS`, without 
`MM_KERNEL_HEAP`), pnsh, nsh64, smp64, mps2-an500:citest_protected, 
mps2-an500:knsh, qemu-armv7a:knsh, qemu-intel64:knsh_romfs, qemu-intel64:nsh, 
sim:ostest. sim:smp fails in apps `testing/ostest/intstack.c` on master and 
with this PR alike. sim:ostest (CI image): 4 of 5 loops in 900 s on a loaded 
host, master and this PR alike.
   
   Rebased on master fa7cb3c2e94: qemu-armv8a:knsh and qemu-armv8a:nsh build 
with no warning, and ostest exits with status 0 on both (1 CPU).
   
   `tools/checkpatch.sh -c -u -m -g` passes. Documentation: commit 4.
   
   ## Not in this PR
   
   - FLAT builds still end a thread where it is, also inside the OS.
   - The other architectures with system calls (see above) keep the old 
behaviour. They need to end a system call in the context of the thread first.
   - Atomic changes of `tcb->flags` of a running thread from another CPU (see 
the race above). That change touches many files.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to