CV-Bowen opened a new pull request, #21971:
URL: https://github.com/apache/nuttx/pull/21971
## Summary
rpmsg_virtio receives with its own kthread per device: the rx interrupt
moves every non-RT buffer to a readyrx list and wakes the thread, which
then runs the endpoint callbacks one by one. A callback that blocks (for
example in `rpmsg_wait()` for a tx buffer) stalls the whole device, and
other transports cannot reuse the scheme.
This PR moves rx scheduling into the rpmsg framework:
- `drivers/rpmsg: dispatch rx in per-device work queues`: each device
creates `CONFIG_RPMSG_WQUEUE_NUMBER` work queues; the rx interrupt
queues a buffer with `rpmsg_queue_rx_work()` on the queue picked by the
endpoint priority (bound RT endpoints are still handled in the
interrupt). `rpmsg_wait()` called from one of these queues runs the
queue's pending works while it waits, bounded by a recursion limit, so
a blocking callback no longer stalls rx. The kthread and the
readyrx/freerx lists in rpmsg_virtio are removed.
- `drivers/rpmsg: add an option to share the work queues between
devices`: `CONFIG_RPMSG_WQUEUE_GLOBAL` creates one set of queues in
`rpmsg_initialize()` for all devices.
- `drivers/rpmsg: fix rpmsg_wait() not returning in the recursive path`:
at the recursion limit `rpmsg_wait()` returned an error although the
semaphore could still be posted; keep waiting instead. Adds
`rpmsg_tickwait()` with a timeout (`rpmsg_wait()` becomes a wrapper)
and passes the remaining time to the transport wait op.
## Impact
- Kconfig: `RPMSG_VIRTIO_PRIORITY` / `RPMSG_VIRTIO_STACKSIZE` are replaced
by `RPMSG_WQUEUE_PRIORITY` / `RPMSG_WQUEUE_STACKSIZE` (same defaults),
plus `RPMSG_WQUEUE_NUMBER` (default 1) and `RPMSG_WQUEUE_GLOBAL`
(default n). No in-tree defconfig sets the old options. `RPMSG` now
selects `SCHED_WORKQUEUE`.
- The rx thread is named `rpmsg-<cpu>-<n>` instead of `rpmsg-virtio`.
- Public API: new `rpmsg_tickwait()`; `rpmsg_wait()` keeps its signature
(now a macro), so the existing callers are unchanged.
- Private API in `drivers/rpmsg/rpmsg.h`: the transport `wait`/`post` ops
lose the semaphore argument, and `rpmsg_register()` takes the number of
rx works to preallocate; rpmsg_port and rpmsg_router_edge pass 0.
## Testing
Host: Ubuntu 22.04.5 LTS x86_64, QEMU version 10.2.1, sim: cc (Ubuntu
13.1.0-8ubuntu1~22.04) 13.1.0, qemu-armv8a: aarch64-none-elf-gcc (Arm GNU
Toolchain 12.2.Rel1 (Build arm-12.24)) 12.2.1 20221205
qemu-armv8a:rpserver + qemu-armv8a:rpproxy (rptun ivshmem; the pair also
talks over rpmsg_port_uart). A test-only patch (not in this PR) logs, once
per endpoint, whether the ping callback runs in the interrupt and in
which task. `rpmsg ping all 20 1024 3 0` pings with the normal endpoint,
`... 11 0` with the RT one:
| | master | this PR, `RPMSG_WQUEUE_NUMBER=1` | `=3` |
|---|---|---|---|
| normal endpoint, virtio | task `rpmsg-virtio` | task `rpmsg-<cpu>-0` |
task `rpmsg-<cpu>-1` (default priority maps to the middle queue) |
| RT endpoint, virtio | in irq | in irq | in irq |
| RT endpoint, port_uart | task `rpmsg-uart-rx` | same | same |
| 20 x 1024 pings, both sides, both endpoints | pass | pass | pass |
With `=3` each device has three queues at priority 224..226 (`ps`), and
with `RPMSG_WQUEUE_GLOBAL=y`, `RPMSG_WQUEUE_NUMBER=2` there is one shared
pair `rpmsg-glb-0/1`; pings and rpmsgfs (`cat /sp/version` from the
proxy) pass there too.
<details><summary>ping path, this PR, RPMSG_WQUEUE_NUMBER=3</summary>
```
[proxy] PINGPATH rpmsg-ping in irq=0 task=rpmsg-server-1
[proxy] PINGPATH rpmsg-ping-rt in irq=0 task=rpmsg-uart-rx
[proxy] PINGPATH rpmsg-ping-rt in irq=1 task=-
[server] PINGPATH rpmsg-ping in irq=0 task=rpmsg-proxy-1
[server] PINGPATH rpmsg-ping-rt in irq=0 task=rpmsg-uart-rx
[server] PINGPATH rpmsg-ping-rt in irq=1 task=-
```
</details>
<details><summary>ping path, master</summary>
```
[proxy] PINGPATH rpmsg-ping in irq=0 task=rpmsg-virtio
[proxy] PINGPATH rpmsg-ping-rt in irq=0 task=rpmsg-uart-rx
[proxy] PINGPATH rpmsg-ping-rt in irq=1 task=-
[server] PINGPATH rpmsg-ping in irq=0 task=rpmsg-virtio
[server] PINGPATH rpmsg-ping-rt in irq=0 task=rpmsg-uart-rx
[server] PINGPATH rpmsg-ping-rt in irq=1 task=-
```
</details>
<details><summary>ps, RPMSG_WQUEUE_NUMBER=3 / RPMSG_WQUEUE_GLOBAL=y</summary>
```
TID PID PPID PRI POLICY TYPE NPX STATE EVENT SIGMASK
STACK USED FILLED COMMAND
9 0 0 224 FIFO Kthread - Waiting Semaphore
0000000000000000 0008064 0001280 15.8% rpmsg-proxy-0
10 0 0 225 FIFO Kthread - Waiting Semaphore
0000000000000000 0008064 0002224 27.5% rpmsg-proxy-1
11 0 0 226 FIFO Kthread - Waiting Semaphore
0000000000000000 0008064 0002128 26.3% rpmsg-proxy-2
1 0 0 224 FIFO Kthread - Waiting Semaphore
0000000000000000 0008064 0001280 15.8% rpmsg-glb-0
2 0 0 225 FIFO Kthread - Waiting Semaphore
0000000000000000 0008064 0002224 27.5% rpmsg-glb-1
```
</details>
Every commit builds with no warning on sim:rpserver,
qemu-armv8a:rpproxy, and sim:rpserver with `RPMSG_WQUEUE_GLOBAL=y`.
No regression in rpmsg: sim:rpserver/rpproxy and
qemu-armv8a:rpserver/rpproxy boot and pass the fixed rpmsg cases (dump, ping,
rpmsgfs; 15/15). qemu-armv8a was built with LIBC_ENVPATH and BINFMT_CHECKEXEC
disabled, which current master needs for nxinit to start nsh there; unrelated
to this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]