daniel-p-carvalho commented on PR #20576: URL: https://github.com/apache/nuttx/pull/20576#issuecomment-6100447582
Thanks @xiaoxiang781216, I measured what the copies cost on the board (100 Mbit/s, direct cable, TCP iperf) before answering. **Cost of the copies** (CP0 Count around `netpkt_copyin`/`netpkt_copyout`, 4000-frame samples, default 4 RX / 2 TX descriptors, IOB 24x196): | | Throughput | Copy per frame | CPU spent copying | |---|---|---|---| | TX | 51.7 Mbit/s | 42 us (1419 B) | 20% (board is 91% busy) | | RX | 26.9 Mbit/s | 46 us (1514 B) | 11% (board is 60% busy) | **RX:** I kept the copy. The RX ring has 4-16 descriptors, and wrapping a DMA buffer in an IOB would hold the descriptor until the application reads the data (TCP read-ahead, sockets), so the ring would drain. Removing the copy would gain 11% of CPU at most, and RX was not CPU bound anyway; what limited it is below. **TX:** it is worth doing, so I pushed it as a separate commit on top (bb18da5030, "send the TX packets in place"): one descriptor per IOB of the packet (SOP/EOP), data cache flush of each buffer, packet held until its last descriptor is reclaimed, and one TX quota unit per descriptor so the stack never queues more than the free descriptors. A packet that does not fit in the free descriptors is still copied to the DMA buffer of the first one. | IOB 1600 x 24, 2 TX / 16 RX | copy | in place | |---|---|---| | TCP TX | 55.1 Mbit/s | 65 Mbit/s (+18%) | | UDP TX (packets in ~12.5 s) | 95.2k | 100.3k (link limited) | With the default IOB size (196 B) a data frame needs 8 descriptors, so it mostly falls back to the copy and the result is neutral. Since the IOBs are held until the DMA is done, `CONFIG_IOB_NBUFFERS` has to cover the descriptors in use (24 IOBs with 10 TX descriptors made `send()` fail with ENOMEM). **Something I found while measuring** (not caused by the copy, it reproduces with the previous driver): TCP RX collapses to ~1 Mbit/s with two different causes, both about sizing: * Small IOBs: the TCP window is free IOBs x IOB size, but a 1514-byte frame takes 8 IOBs, so the pool runs out and `netpkt_copyin()` fails with ENOMEM. `CONFIG_IOB_THROTTLE=8` fixes it (64x196: 21.8 -> 33.2 Mbit/s, no drops). * Large IOBs (1600): the window is ~38 KB, the sender bursts it at line rate and 4 RX descriptors overrun in hardware (not counted anywhere). 16 RX descriptors give 48.6 Mbit/s. With IOB 1600 x 24, `NRXDESC=16`, `IOB_THROTTLE=4`: 65 Mbit/s TX and 48 Mbit/s RX, against 52.9 and 27 for the first revision of this PR. I also fixed the `NETDEV_xxERRORS()` macros, which were given the lower half instead of the `net_driver_s` and did not build with `CONFIG_NETDEV_STATISTICS=y` (folded into the conversion commit). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
