casaroli opened a new pull request, #22132:
URL: https://github.com/apache/nuttx/pull/22132

   ## Summary
   
   With `CONFIG_NET_TCP_WRITE_BUFFERS`, `psock_send_eventhandler()` freed a 
write buffer at its `TCP_MAXRTX` retransmission, with its data, and only 
counted it in `conn->expired`. The connection stayed open until the next 
timeout of `tcp_timer()`. Meanwhile the next write buffer took its sequence 
number from `conn->sent`, which still counted the sent part of the freed 
buffer. So its data went out at the sequence numbers of the freed data: the 
peer received wrong data without an error, and then a reset.
   
   The count of a write buffer also grew while the peer acknowledged a part of 
it at each timeout. So a connection that made progress at each timeout was 
reset after `TCP_MAXRTX` timeouts, and lost data.
   
   Since a91c50243ef, `tcp_timer()` also closes a connection with write buffers 
when the retry count of the connection (`conn->nrtx`) reaches `TCP_MAXRTX`: it 
sends a RST and gives `TCP_TIMEDOUT` (`ETIMEDOUT`) to the socket. An ACK of all 
the data sent sets that count to zero. This PR removes the expiry of a write 
buffer and `conn->expired`. A write buffer now stays until its data is 
acknowledged or the connection is closed, as without write buffers.
   
   Two examples on master:
   
   - A lossy link: the peer gets each segment only the second time and 
acknowledges it. At the eighth timeout NuttX freed the write buffer. After a 
window update of the peer, NuttX sent the data of stream offset 5840 at offset 
1456, then a RST.
   - Two sim nodes, an echo client and server (`tcpechod`) with 256 IOBs and 
`IOB_THROTTLE` 16: the server freed 3 write buffers 1.85 s after boot and sent 
the data of offset 49152 at offset 46844. The client got 26 segments of wrong 
data, then a reset.
   
   ## Impact
   
   TCP with `CONFIG_NET_TCP_WRITE_BUFFERS` only (`net/tcp/tcp_send_buffered.c`, 
`tcp_timer.c`, `tcp_conn.c`, `tcp.h`). No API, ABI or Kconfig change; 
`conn->expired` was internal.
   
   Behaviour:
   
   - No data is sent at the sequence numbers of other data, and no data that 
the peer did not acknowledge is dropped while the connection is open.
   - A connection whose peer acknowledges nothing ends after `TCP_MAXRTX` 
timeouts, at the same time as before, with a RST and `ETIMEDOUT`. It now also 
retransmits at the eighth timeout.
   - A connection that makes progress at each timeout is no longer reset after 
8 timeouts.
   - The count of a write buffer (`wb_nrtx`) stays for the coalescing check and 
the debug output; it now stops at 255.
   
   Build modes and architectures: common network code, the same in flat, 
protected and kernel builds and on every architecture. With and without 
`NET_TCP_FAST_RETRANSMIT`, `NET_TCP_SELECTIVE_ACK`, `NET_TCP_CC_NEWRENO`, 
`NET_SENDFILE`: see the builds below.
   
   ## Testing
   
   qemu-armv8a:knsh (kernel build) and qemu-armv8a:nsh (flat build) under HVF 
(Apple M-series), 96 IOBs of 196 bytes, `IOB_THROTTLE` 8, write buffers, a TUN 
device (`NET_TUN_PKTSIZE` 1500). Master is 5e4ecd1d603.
   
   - `cmocka_net_tun` (apache/nuttx-apps#APPS): a TCP peer that the test plays 
on the TUN device. The link loses the first transmission of each segment; the 
peer acknowledges each retransmission, so the connection makes progress at each 
timeout. After 10 rounds the window opens. The case checks each byte against 
its sequence number, the whole stream, the FIN, and no RST.
   - `wrbexp`: the same peer as a standalone program, with two more modes: 
`silent` (the peer acknowledges nothing) and `partial` (the case above).
   - Two sim nodes on a bridge (sim:tcpblaster, 256 IOBs, `IOB_THROTTLE` 16, 
write buffers): an echo client on node A sends 256 KiB or 1 MiB to `tcpechod` 
on node B while it reads the echo. A capture on the TAP device of A, checked 
against the stream.
   
   | Test | master | this PR |
   |---|---|---|
   | `cmocka_net_tun`, knsh 1 CPU, 3 runs in one boot | FAIL 3 of 3: wrong data 
from offset 1456 (1457 bytes) | PASS 3 of 3 |
   | `cmocka_net_tun`, knsh 4 CPUs (`CONFIG_SMP`) | FAIL 3 of 3 | PASS 3 of 3 |
   | `cmocka_net_tun`, flat nsh | FAIL 3 of 3 | PASS 3 of 3 |
   | `wrbexp partial`, knsh (`CONFIG_DEBUG_NET_WARN`) | "Expiring wrb ... 
nrtx=8" at the 8th timeout; the data of offset 5840 at offset 1456; RST | 10 
rounds, the whole stream and the FIN, no RST |
   | `wrbexp silent` | 7 retransmissions, RST at 120 s, `recv()` gives 
`ETIMEDOUT` | 8 retransmissions, RST at 120 s, `ETIMEDOUT` |
   | sim, two nodes, 256 KiB and 1 MiB, 3 runs each | 6 of 6: wrong data from 
offset 46844 (the data of 49152), then a reset | 6 of 6: no wrong data; the 
server times out after 8 timeouts (`ENOTCONN` in `send()`), the client gets a 
reset |
   | `cmocka_net_tcp`, knsh 1 and 4 CPUs, flat nsh | 6 of 7 
(`test_tcp_connect_ipv4` fails) | the same |
   | `ostest`, knsh 1 and 4 CPUs, flat nsh | status 0 | status 0 |
   
   TCP over 127.0.0.1 in this configuration (96 IOBs), `tcplomin` (8 MiB from 
one thread to another): on master the write buffers of the sender hold the 
pool, the receiver drops the data below the IOB throttle, the sender frees 3 
write buffers ("Expiring wrb") at the eighth timeout, the connection is reset, 
and `send()` gives `ENOTCONN` after 46676 bytes. With this PR the sender keeps 
its data and retransmits; some retransmissions get through, so the connection 
does not time out, but it moves about one segment per timeout (71 timeouts in 
105 s; the shell stays responsive). The same with apache/nuttx#22078. With 
apache/nuttx#22089 (no drop of the data of a peer on this host) and this PR: 
`tcplomin` and `tcplobulk` 8 MiB PASS (79.7 MiB/s), `tcpthru` loopback 20 MiB 
PASS, `cmocka_net_tun` PASS, `ostest` status 0; #22089 alone gives the same 
numbers, but `cmocka_net_tun` fails. `tcplobulk` with a slow reader fails the 
time limit with #22089, with and without this PR.
   
   In the two-node runs both nodes drop the data of the other below the IOB 
throttle, so the server's connection makes no progress and times out on master 
and with this PR; this PR removes only the wrong data. The 8 timeouts come in 
less than one second because a segment with data that repeats the ACK number 
makes the fast retransmission code run `tcp_timer()`, which counts each call as 
a timeout; apache/nuttx#22078 removes that.
   
   Builds in the NuttX CI image (`EXTRAFLAGS="-Wno-cpp -Werror"`), master and 
this PR: all OK, 0 warnings. Text size (bytes):
   
   | Configuration | Build mode | master | this PR |
   |---|---|---|---|
   | qemu-armv8a:netnsh (no write buffers) | flat | 554188 | 554188 (+0) |
   | qemu-armv8a:netnsh + `NET_TCP_WRITE_BUFFERS`, `NET_SOLINGER` | flat | 
557164 | 557060 (-104) |
   | the same, CMake | flat | 557948 | 557844 (-104) |
   | qemu-armv8a:netnsh_smp + write buffers | flat, SMP | 532864 | 532760 
(-104) |
   | the same + out-of-order, SACK, NewReno, sendfile, urgent data | flat | 
561856 | 561740 (-116) |
   | the same without `NET_TCP_FAST_RETRANSMIT` | flat | 556908 | 556804 (-104) 
|
   | rv-virt:pnsh64 + TCP, write buffers | protected | 184672 | 184608 (-64) |
   | rv-virt:knetnsh64 | kernel | 229099 | 229099 (+0) |
   | rv-virt:knetnsh64 + write buffers, dynamic pools | kernel | 231479 | 
231397 (-82) |
   | lc823450-xgevk:krndis | protected | 193484 | 193484 (+0) |
   | sim:tcpblaster | sim | 997922 | 997778 (-144) |
   | lm3s6432-s2e:nsh | flat | 63496 | 63496 (+0) |
   | qemu-intel64:nsh_pci + TCP, write buffers | flat | 8760475 | 8760351 
(-124) |
   
   `tools/checkpatch.sh -c -u -m -g` passes. Documentation: 
`Documentation/os/networking/tcp_state_machine.rst` already says that the 
connection is closed with `TCP_TIMEDOUT` and a RST when the retransmissions 
exceed the limit; no change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] net/tcp: Do not free a w... casaroli (via GitHub) via commits

Reply via email to