casaroli opened a new pull request, #21393:
URL: https://github.com/apache/nuttx/pull/21393
## Summary
ostest hangs in the first case of its spinlock test (`Thread num: 1`) on
qemu-armv8a:nsh_smp with 2 or 4 CPUs under QEMU with HVF, and sometimes under
TCG. The test pins itself to CPU0 and starts a thread pinned to CPU1 at the
highest priority. Two scheduler defects can each lose a task here; this PR
fixes both.
1. A lost task delivery blocks a CPU for ever. `nxsched_deliver_task()` sets
`g_delivertasks[cpu]` and sends an inter-processor interrupt; the CPU clears
the request in `nxsched_process_delivered()`. While the request is set, no
other CPU sends that CPU an interrupt again. The other CPUs start in parallel
with CPU0 (on arm64 since 5e2eadacf77), so CPU0 can deliver a task to CPU1
before CPU1 has initialized its GIC. That SGI is lost (`gicv3_cpuif_init()`
also clears the pending SGIs of the CPU; on risc-v, `riscv_cpu_boot()` clears
the IPI of the hart in the same way), and the request stays set. After that, a
task that can run only on CPU1 never runs, and `sched_unlock()` on CPU1 itself
also skips its delivery. The first commit clears the request in
`nx_idle_trampoline()`, where the CPU can take interrupts, before
`sched_unlock()` looks at the ready-to-run list.
2. `sched_setaffinity()` on the calling task does not move the task when no
other task is ready to run. `nxsched_running_setpriority()` checks the affinity
change (since 01f7ff2bb73; 791c1a4ab5e added the move of the calling task) only
inside `if (nxttcb && ...)`, and `nxttcb` is NULL when the ready-to-run list is
empty, which is the usual case. The task keeps running on a CPU that its
affinity excludes. When a higher priority task then preempts it on that CPU, it
goes back to the ready-to-run list, and no CPU of its affinity is told. The
third commit checks the affinity independently of the ready-to-run list. The
second commit fixes two older nxstyle errors in the same file (whitespace only).
Fix 2 alone makes the hang of fix 1 easier to hit (a task that moves itself
to CPU1 then hangs), so both are in one PR.
## Impact
SMP only. Fix 1 runs once on each of CPUs 1 to N-1 when it starts. With fix
2, `sched_setaffinity()` on the calling task moves it to a CPU of the new mask
before it returns, as it already did when another task was ready to run. For a
task that runs on another CPU, fix 2 only adds an equal-priority delivery to
that CPU; that case is not changed otherwise. Common code, so every SMP
architecture and build mode. No API, ABI or documentation change.
## Testing
Base 0e22bd5bd98, apps b017038e5 (some master runs used 6bc6ffef30b or
2eeaf9b6a33; no commit between them touches sched/ or arch/arm64/).
qemu-armv8a:nsh_smp (flat) with `CONFIG_ARCH_CHIP_QEMU_WITH_HV=y` (HVF needs
hvc for PSCI) and without semihosting, 4 CPUs unless noted; QEMU 11.0 with
`-accel hvf -cpu host` on an Apple M4, and the same image on TCG (`-cpu max`).
Three test programs, built in as apps: `spinhang` (the first spinlock case, 20
rounds of 100000 lock/unlock, with progress lines), `smppin` (a thread pinned
to each CPU must run within 5 s, 10 rounds) and `selfmig`
(`sched_setaffinity(0, {c})` must move the caller to CPU c, 10 rounds x 4 CPUs).
| Test | master | this PR |
|---|---|---|
| ostest, HVF | hangs in the spinlock test, 3 of 3 (also 1 of 1 with the
head of apps #3936) | status 0, 5 of 5 |
| ostest, TCG | status 0, 1 of 1 | status 0, 2 of 2 |
| spinhang, HVF | hangs in round 0, 3 of 3 (2 CPUs: 2 of 2;
`CONFIG_TICKET_SPINLOCK`: 1 of 1) | PASS 5 of 5 (2 CPUs: 2 of 2) |
| spinhang, TCG | hangs 2 of 5 (when it starts on CPU1) | PASS 3 of 3 |
| smppin, HVF | FAIL 3 of 3: the thread for CPU1 never runs | PASS 3 of 3 |
| selfmig, HVF and TCG | FAIL 5 of 5: 30 of 40 checks, the caller stays on
CPU0 | PASS 5 of 5 |
On master, gdb (QEMU gdbstub) at the HVF hang shows all CPUs in the idle
loop with IRQs enabled, the test thread in `g_readytorun`, `g_delivertasks[1]
== SWITCH_HIGHER`, and no SGI pending or active in CPU1's redistributor. At the
idle nsh prompt, before any test, `g_delivertasks[1]` is already
`SWITCH_HIGHER` in some boots. A debug patch (not in this PR) that records
`g_delivertasks[cpu]` just before the GIC init of each secondary CPU shows
`SWITCH_HIGHER` for CPU1, 3 of 3 boots. Under HVF the secondary CPUs print
their boot lines after the first `nsh>` prompt; under TCG they boot before it,
which is why TCG does not show fix 1. At the TCG hang, gdb shows the test's
main task in `g_readytorun` with affinity CPU0 and all CPUs idle; the console
shows it was still on CPU1 after `sched_setaffinity()`. With only fix 2,
`selfmig` hangs under HVF (the moved task meets fix 1's problem on CPU1).
qemu-armv8a:knsh (kernel build) with `CONFIG_SMP=y`, `CONFIG_SMP_NCPUS=4`,
HVF, /system over virtio 9p: master: `smppin` PASS, `selfmig` FAIL (30 of 40),
ostest status 0, 2 of 2; this PR: `smppin` PASS, `selfmig` PASS, ostest status
0, 3 of 3. With the head of apps #3936 (5b4c38fb1) the three commits (on
2eeaf9b6a33) also give ostest status 0 under HVF (2 of 2) and TCG (2 of 2).
Build with `EXTRAFLAGS="-Wno-cpp -Werror"`, each OK: qemu-armv8a:nsh_smp,
rv-virt:smp64, rv-virt:ksmp64 (kernel, SMP), sim:smp (macOS host),
qemu-armv7a:smp, qemu-armv8a:knsh, qemu-armv8a:nsh. The only warnings are the
linker RWX/empty segment notices of rv-virt:smp64 and qemu-armv7a:smp, the same
on master.
`tools/checkpatch.sh -c -u -m -g origin/master..HEAD` passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
