casaroli opened a new pull request, #22089: URL: https://github.com/apache/nuttx/pull/22089
## Summary Fixes #22088. Since ea5d2c9ae49 (#22000), `tcp_datahandler()` drops the received data while fewer than `IOB_THROTTLE` IOBs are free. With `CONFIG_NET_TCP_WRITE_BUFFERS`, the write buffers can use all IOBs above the throttle, and the received segment holds some of the others. When the peer is on this host (127.0.0.1, or a local address of a network device), the write buffers that hold the IOBs are the buffers of that peer, and they free their IOBs only when the dropped data is acknowledged. Every segment is dropped and the transfer stops. With the default pool (96 IOBs of 196 bytes), a TCP transfer over 127.0.0.1 stops after about 20 KB, also between two threads of one process; on master the sender resets the connection after about 13.5 s. A log at each drop shows the pool at that time: 1 IOB free, 87 IOBs in the write queue of the sender, 8 IOBs in the received segment, the read-ahead buffer of the receiver empty, the drop in `tcp_data_event()`. The throttle keeps IOBs for sending and for the RX buffers of the drivers. A segment of a peer on this host comes through `devif_loopback()`, not from the RX buffer of a driver. This PR keeps the data of such a peer as before ea5d2c9ae49 (new `tcp_peer_is_local()`: the segment came through the loopback device, or the remote address of the connection is an address of the device). The data of other peers is dropped below the throttle as before, so the case of #22000 does not change. Another possible fix is the rule that `tcp_datahandler()` had until 34d2cde8a8f (an empty read-ahead buffer takes the data, bf210560017). It was tested and is not sufficient: the transfer goes on at about 5 KB/s, because the segments that arrive while the read-ahead buffer holds one segment are dropped and sent again after the retransmission timeout. ## Impact TCP with `CONFIG_IOB_THROTTLE > 0` (the default with write buffers and read-ahead): the received data of a connection whose peer is on this host is no longer dropped below the throttle. Other connections, UDP and `IOB_THROTTLE` 0 do not change. No API, ABI or Kconfig change. For such a connection, the read-ahead buffer can again take IOBs below the throttle, as before ea5d2c9ae49. The sender on this host takes the IOBs of each segment from the free pool, so it cannot send more when the pool is empty; the IOBs come back when the application reads. Build modes and architectures: common network code, the same in flat, protected and kernel builds; SMP: the check runs under the same locks as the drop before. ## Testing qemu-armv8a under HVF (Apple M-series), TCP over the loopback device, 96 IOBs of 196 bytes, `IOB_THROTTLE` 8, `CONFIG_NET_TCP_WRITE_BUFFERS` unless noted; master ed122374690 (the PR is rebased on 11c71e16384 after the runs, with no change in `net/`; the final head, which also takes every segment of the loopback device as local, was run again with `cmocka_net_tcp`, the reproducers, `tcpthru` and ostest on knsh, 1 CPU, with the same results), the parent of #22000 is 9df78aa1e69. The programs run from `/system/bin` (kernel build). - `tcplomin`: one thread sends 8 MiB to another thread of the same process over 127.0.0.1 (the program of #22088). - `tcplobulk`: the same with a progress line every second and a 60 s limit; also with 4 KiB `send()`/`recv()`. - `tcpthru loop 20`: 20 MiB between two threads. - `cmocka_net_tcp` with the new case `test_tcp_loopback_bulk` (apache/nuttx-apps#APPS): 4 MiB from a thread with blocking `send()`, 30 s limit (an earlier 1 MiB version on the parent and on ea5d2c9ae49). | Test (knsh, HVF) | 9df78aa1e69 (before #22000) | ea5d2c9ae49 | master | this PR | |---|---|---|---|---| | `tcplomin`, 1 CPU | PASS | not done after 300 s | FAIL: `send()` gives `ENOTCONN`, 14674 of 8 MiB received | PASS | | `tcplobulk`, 64 KiB calls, 1 CPU | PASS, 79.9 MiB/s | about 4.5 KB/s | FAIL at 14674 bytes (`ECONNRESET`) | PASS, 79.6 MiB/s | | `tcplobulk`, 4 KiB calls, 1 CPU | PASS, 79.8 MiB/s | | FAIL at 18540 bytes | PASS, 79.3 MiB/s | | `tcpthru loop 20`, 1 CPU | PASS, 46.0 MiB/s | | FAIL: `ENOTCONN` after 13.5 s | PASS, 46.5 MiB/s | | `test_tcp_loopback_bulk`, 1 CPU, 2 runs | PASS (1 MiB version) | FAIL: 63008 of 1 MiB after 31 s (1 MiB version) | FAIL: `ECONNRESET` after 23330 bytes, 13 s | PASS | | `tcplobulk` to the address of eth0 (virtio-net), 1 CPU | | | FAIL at 17308 bytes (`ECONNRESET`) | PASS, 79.3 MiB/s | | `tcplomin`, 4 CPUs | | | FAIL: 223106 bytes received | PASS | | `tcpthru loop 20`, 4 CPUs | | | FAIL: `ENOTCONN` after 13.5 s | PASS, 455.9 MiB/s | | `test_tcp_loopback_bulk`, 4 CPUs | | | FAIL in 4 of 4 runs (2 boots): `ECONNRESET` after 70 KB to 2 MB | PASS in 2 of 2 runs | | flat qemu-armv8a:nsh: `test_tcp_loopback_bulk` | | | FAIL: `ECONNRESET` after 23330 bytes | PASS | | no TCP write buffers (`IOB_THROTTLE` 8): all rows of 1 CPU | | | PASS | PASS | | `IOB_THROTTLE` 0: `tcplomin`, `tcplobulk` | | | | PASS | | `ostest` (each boot) | status 0 | status 0 | status 0 | status 0 | A slow reader (1 KiB `recv()` and a 2 ms pause) gets about 4 to 16 KB/s over 127.0.0.1 with write buffers, before #22000 and with this PR alike (master resets the connection); this is not changed here. `test_tcp_connect_ipv4` of `cmocka_net_tcp` fails in this configuration on all versions (`ECONNREFUSED` from the group server). #22000's test (4 host connections echo 256 KiB each through a NuttX node at the same time, random writes of 1 to 4096 bytes; sim without write buffers, 100 IOBs of 196 bytes, `IOB_THROTTLE` 8, `examples/tcpecho`): | | master | this PR | master without ea5d2c9ae49 | |---|---|---|---| | connections that echoed all 256 KiB (6 runs x 4) | 5 of 24 | 4 of 24 | 0 of 24 | | free IOBs after the runs | 100 of 100 | 100 of 100 | 1 of 100 | Without ea5d2c9ae49 the received data keeps the whole pool and every connection stops; with it, on master and with this PR, the pool comes back. With `examples/tcpecho` (one thread, 64-byte reads) no run completes all 4 connections, also on master (the other connections stop, and later ones find no free TCP connection), so these runs show the pool, not the pass rate of #22000. The peers of this test are not on this host, so the changed condition does not apply to them. Builds in the NuttX CI image (`EXTRAFLAGS="-Wno-cpp -Werror"`), master and this PR: all OK, 0 warnings. Text size (bytes): | Configuration | Build mode | `IOB_THROTTLE` | master | this PR | |---|---|---|---|---| | qemu-armv8a:netnsh (no TCP write buffers) | flat | 8 | 554132 | 554148 (+16) | | qemu-armv8a:netnsh + `NET_TCP_WRITE_BUFFERS` | flat | 8 | 557108 | 557124 (+16) | | the same, CMake | flat | 8 | 557908 | 557924 (+16) | | the same + `CONFIG_IOB_THROTTLE=0` | flat | 0 | 556856 | 556856 (+0) | | qemu-armv8a:netnsh + write buffers + `NET_IPv6` (both domains) | flat | 8 | 569780 | 569832 (+52) | | qemu-armv8a:netnsh_smp + write buffers | flat, SMP | 8 | 532804 | 532820 (+16) | | rv-virt:pnsh64 + TCP, write buffers | protected | 8 | 184672 | 184672 (+0) | | rv-virt:knetnsh64 | kernel | 2 | 229103 | 229109 (+6) | | rv-virt:knetnsh64 + write buffers, dynamic pools | kernel | 2 | 231483 | 231489 (+6) | | lc823450-xgevk:krndis | protected | 0 | 193476 | 193476 (+0) | | sim:tcpblaster (IPv4 and IPv6) | sim | 16 | 997922 | 998066 (+144) | | sim:tcpblaster without write buffers | sim | 16 | 989234 | 989378 (+144) | | lm3s6432-s2e:nsh | flat | 0 | 64932 | 64932 (+0) | IPv6 only is not built here: qemu-armv8a:netnsh selects the IPv4 DHCP client, which does not build without IPv4 (master too). The IPv6 condition is built in the two dual-stack rows. `tools/checkpatch.sh -c -u -m -g` passes. Documentation: no page describes this drop (`Documentation/os/memory/iob.rst` describes `IOB_THROTTLE` in general terms that stay true). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
