CV-Bowen opened a new pull request, #22098:
URL: https://github.com/apache/nuttx/pull/22098

   ## Summary
   
   Priority handling of the rpmsg rx workers and the tx done interrupt of
   rpmsg_virtio:
   
   - `openamp: add the priority field to the rpmsg virtio config`: new
     open-amp patch 0025. It puts a `priority` byte into
     `struct fw_rsc_config`, taken from the reserved space so the structure
     keeps its size, and adds the feature bit `VIRTIO_RPMSG_F_PRIORITY`.
   - `take the rx worker priority from the virtio config`:
     `rpmsg_init_wqueues()` gets the priority of the device's work queues.
     rpmsg_virtio reads it from the resource table when the device offers
     `VIRTIO_RPMSG_F_PRIORITY`, else it stays
     `CONFIG_RPMSG_WQUEUE_PRIORITY`.
   - `enable the tx done interrupt only while waiting`: the tx virtqueue
     callback was enabled once at start-up, so the peer interrupted the
     sender for every returned tx buffer. Now the first sender that waits
     for a buffer in `rpmsg_virtio_notify_wait()` enables it and the last
     one disables it again (as Linux virtio_rpmsg_bus does with
     `rpmsg_upref_sleepers()`/`rpmsg_downref_sleepers()`), with a recheck
     of the virtqueue under the device lock so a buffer returned just
     before the callback was enabled is not missed.
   - `give each rx worker the priority of the work it runs`: the work
     queues were created at `priority + i` and `rpmsg_queue_work()` chose
     one by the distance of the endpoint priority from
     `RPMSG_PRIO_DEFAULT`. Now all queues start at the device priority and
     get a priority on demand with `work_queue_setpriority_wq()`: the
     endpoint priority is mapped around the device priority, an existing
     queue at that priority is reused, else a free queue is assigned, else
     the closest one is used. With `CONFIG_RPMSG_WQUEUE_GLOBAL` the shared
     queues are assigned under a spinlock.
   - `keep the rx worker priority in range`: the mapped priority was stored
     in a `uint8_t` and wrapped. A name service message inherited the
     priority of the endpoint it announces, so the announcement of an RT
     endpoint (255) mapped to 255 - 127 + 224 = 352 and ran at 96. Name
     service messages no longer inherit `RPMSG_PRIO_RT`, and the result is
     clamped to the scheduler range.
   
   Depends on apache/nuttx#22058 (its commits are the first 10 of this
   branch; review the last 5).
   
   ## Impact
   
   - New open-amp patch 0025: `struct fw_rsc_config` gets `priority`, the
     former `reserved[6]` becomes `reserved1[3]` and `reserved2[5]` (same
     size, no in-tree user of the field). New feature bit
     `VIRTIO_RPMSG_F_PRIORITY` (5). No in-tree resource table sets it yet,
     so the worker priority stays `CONFIG_RPMSG_WQUEUE_PRIORITY` unless a
     platform fills the field.
   - With `CONFIG_RPMSG_WQUEUE_NUMBER` > 1 the extra workers no longer run
     at fixed `CONFIG_RPMSG_WQUEUE_PRIORITY + i`; their priority follows the
     endpoints that use them. With the default of 1 nothing changes.
   - The tx done interrupt of rpmsg_virtio is only enabled while a sender
     waits for a buffer. With `CONFIG_RPMSG_VIRTIO_PM=y` and
     `CONFIG_RPMSG_VIRTIO_PM_AUTORELAX=n` the PM wakelock taken on send is
     released from that callback, so it now stays held until the next wait
     (`PM_AUTORELAX` defaults to y and releases it from a timer instead; no
     in-tree config enables `RPMSG_VIRTIO_PM`).
   - `rpmsg_init_wqueues()` takes a priority, `rpmsg_queue_work()` takes
     the priority as `int`; both are internal to drivers/rpmsg.
   
   ## Testing
   
   Host: Ubuntu 22.04.5 LTS x86_64, QEMU version 10.2.1, sim: cc (Ubuntu 
13.1.0-8ubuntu1~22.04) 13.1.0, qemu-armv8a: aarch64-none-elf-gcc (Arm GNU 
Toolchain 12.2.Rel1 (Build arm-12.24)) 12.2.1 20221205
   
   Every commit builds sim:rpserver, sim:rpproxy, qemu-armv8a:rpserver and
   qemu-armv8a:rpproxy with 0 warnings (open-amp re-extracted for each
   commit, so patch 0025 is applied from scratch). sim:rpserver/rpproxy
   also build with `CONFIG_RPMSG_WQUEUE_GLOBAL=y` with 0 warnings.
   
   **Worker priorities**, sim:rpserver + sim:rpproxy, `ps` on both nodes
   after boot and after a 200 x 2048 byte ping in each direction (2 runs
   each, same result). "before" is the base of this PR (#22058 head).
   "config 150" adds a test-only change (not in this PR) to
   `arch/sim/src/sim/sim_rptun.c` that sets `VIRTIO_RPMSG_F_PRIORITY` in
   the rpmsg vdev `dfeatures` and `rpmsg0_config.priority = 150`.
   
   | `CONFIG_RPMSG_WQUEUE_NUMBER` | before | this PR | this PR, config 150 |
   |---|---|---|---|
   | 1 | server 224, proxy 224 | (same as before) | server 150, proxy 150 |
   | 3 | 224 / 225 / 226 on both | server 224 / 223 / 224, proxy 224 / 224 / 
224 | server 150 / 149 / 150, proxy 150 / 150 / 150 |
   
   With 3 queues the server queue at 223 serves the rpmsg-syslog server
   endpoint (endpoint priority 126); the others serve endpoints at the
   default priority. Without the last commit the same run showed a queue
   at 96 (22 with config 150) on both nodes: the name service message of
   `rpmsg-ping-rt`.
   
   **Tx buffer exhaustion**: `rpmsg ping all 1000 2048 0 0` (no ack, back
   to back, so senders wait in `rpmsg_virtio_notify_wait()`) on both nodes,
   then a normal ping, 2 runs per target.
   
   | | before | this PR |
   |---|---|---|
   | sim, 1000 pings each way | pass, pass | pass, pass |
   | qemu-armv8a ivshmem (rpmsg_virtio) channel | pass in every run that 
reached it | pass in every run |
   | qemu-armv8a rpmsg_port_uart channel | stalls with all 8 tx buffers in use 
in 3 of 4 directions | stalls in 1 of 4 |
   
   The rpmsg_port_uart stall happens before and after this PR and that
   transport is not touched here.
   
   **Ping latency**, 200 x 2048 byte with ack (test.sh extra cases):
   
   | average | before | this PR |
   |---|---|---|
   | sim server / proxy | 9.998 / 9.998 ms | 9.998 / 9.998 ms |
   | qemu-armv8a ivshmem, server / proxy | 202 / 222 us | 203 / 203 us |
   
   Not tested: the number of tx done interrupts (no interrupt counter in
   these configs), a platform that fills `fw_rsc_config.priority` in its
   resource table, `CONFIG_RPMSG_VIRTIO_PM`, and endpoints with priorities
   other than the defaults present in these configs.
   
   No regression in rpmsg: sim:rpserver/rpproxy and 
qemu-armv8a:rpserver/rpproxy boot and pass the fixed rpmsg cases (dump, ping, 
rpmsgfs; 22/22). qemu-armv8a was built with LIBC_ENVPATH and BINFMT_CHECKEXEC 
disabled, which current master needs for nxinit to start nsh there; unrelated 
to this change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to