casaroli opened a new pull request, #22029:
URL: https://github.com/apache/nuttx/pull/22029

   ## Summary
   
   When the peer resets an established TCP connection (a RST), NuttX did not 
report it. `recv()` gave end-of-file after the data that came before the RST, 
as for a FIN, so a program could not tell a stream that was cut from one that 
ended (a download cut by a reset looked complete). A `recv()` that waited at 
the reset gave `ENOTCONN`, `send()` gave `EPIPE` at once, and `poll()` set 
`SO_ERROR` to `ECONNREFUSED`. POSIX and Linux: the first receive or send fails 
with `ECONNRESET`, and `SO_ERROR` gives it until it is reported.
   
   1. **`fs/nfs: Reconnect also when a request fails with ECONNRESET.`** 
`nfs_request()` connects to the server again when a request fails with 
`ENOTCONN` or `EPIPE`. With commit 2 a reset of the NFS server over TCP gives 
`ECONNRESET` to the first receive or send: reconnect on it too, so that the 
reconnect continues to work (the same as 088075ce7fc did for `EPIPE`).
   2. **`net/tcp: Report a reset of the peer as ECONNRESET.`** `tcp_input()` 
marks the connection when a RST comes in `TCP_ESTABLISHED`, `TCP_FIN_WAIT_1` or 
`TCP_FIN_WAIT_2` (`TCP_RESET`, `TCP_RESETERR`, `SO_ERROR` = `ECONNRESET`). The 
first receive without data, send, `sendfile()` or `getsockopt(SO_ERROR)` that 
returns the error reports it, once (`tcp_reset_report()`). After that a receive 
gives end-of-file and a send `EPIPE` (and `SIGPIPE` without `MSG_NOSIGNAL`). 
The data that came before the RST is read first. A receive or send that returns 
a count keeps the error for the next call (`recv()` with `MSG_WAITALL` or with 
more than one buffer, a `send()` that queued a part). `poll()` after the reset 
gives `POLLHUP` and `POLLIN`, and `POLLERR` while the error is not reported. A 
RST in `TCP_SYN_SENT` still gives `ECONNREFUSED`; after the FIN of the peer a 
receive still gives end-of-file and a send `EPIPE`.
   3. **`Documentation/os/networking: Describe what a reset of the peer 
gives.`** The TCP state machine page.
   
   The change is in common network code: the same in flat, protected and kernel 
builds, on every architecture.
   
   ## Behaviour
   
   After a RST of the peer on an established connection (the peer closed with 
data unread), qemu-armv8a:knsh, measured with the cases of 
apache/nuttx-apps#MMMM:
   
   | Call | master | this PR (and Linux) |
   |---|---|---|
   | `recv()` with 5 bytes of the peer unread | 5 bytes, then 0 (end-of-file) | 
5 bytes, then -1 `ECONNRESET`, then 0 |
   | `recv()` that waits at the reset | -1 `ENOTCONN` | -1 `ECONNRESET`, then 0 
|
   | `recv(MSG_WAITALL)` that waits, 5 bytes came before the reset | 5, then 0 
| 5, then -1 `ECONNRESET`, then 0 |
   | `send()` | -1 `EPIPE` (and `SIGPIPE`) | -1 `ECONNRESET` (no `SIGPIPE`), 
then -1 `EPIPE` (and `SIGPIPE`) |
   | `getsockopt(SO_ERROR)` | 0, or `ECONNREFUSED` once a `poll()` ran | 
`ECONNRESET`, then 0 |
   | `poll(POLLIN \| POLLOUT)` with data unread | `POLLIN \| POLLERR \| 
POLLHUP`, `POLLERR` stays | `POLLIN \| POLLERR \| POLLHUP`; after `SO_ERROR`: 
`POLLIN \| POLLHUP` |
   | non-blocking `connect()` refused (RST in `TCP_SYN_SENT`) | `POLLERR`, 
`SO_ERROR` `ECONNREFUSED` | the same |
   | RST after the FIN of the peer (`TCP_CLOSE_WAIT`) | end-of-file, `EPIPE` | 
the same |
   
   POSIX: `recv()` and `send()` list `ECONNRESET` ("A connection was forcibly 
closed by a peer"); `send()` gives `EPIPE` and `SIGPIPE` when the socket "is no 
longer connected"; `SO_ERROR` "reports information about error status and 
clears it"; `POLLHUP` and `POLLOUT` exclude each other. Linux `tcp_reset()` 
sets `ECONNREFUSED` in `TCP_SYN_SENT`, `EPIPE` in `TCP_CLOSE_WAIT` and 
`ECONNRESET` in the other states; `tcp_recvmsg()` gives the queued data first, 
then the error once, then end-of-file. Linux also sets `POLLOUT` with 
`POLLHUP`; `poll_notify()` of NuttX clears `POLLOUT` when `POLLHUP` is set, as 
POSIX says. In `TCP_CLOSE_WAIT` NuttX does not set `SO_ERROR` to `EPIPE` as 
Linux does: that is out of the scope of this PR (master behaviour kept).
   
   ## Impact
   
   TCP only, plus one condition in `fs/nfs`. No API, ABI or Kconfig change. The 
errors that programs see after a reset of the peer change as in the table. 
In-tree users (nuttx and nuttx-apps, grep of `ECONNRESET`, `ENOTCONN`, `EPIPE`, 
`ECONNREFUSED` and of the in-kernel `psock_*()` users):
   
   - `fs/nfs/nfs_util.c`: reconnected only on `ENOTCONN` and `EPIPE`; commit 1 
adds `ECONNRESET`. Without it a request to an NFS server over TCP that reset 
the connection would fail once instead of reconnecting. (`rpcclnt_receive()` 
also loops on a `recv()` that returns 0; with this PR it gets `ECONNRESET` 
first.)
   - Handle `ECONNRESET` already, as a closed connection: 
`apps/examples/tcpecho` ("client aborted connection"), 
`apps/system/telnet/telnet_chatd.c`, `apps/examples/mqttc/mqttc_sub.c` (`EPIPE 
|| ECONNRESET`), `apps/industry/nxmodbus` (any error closes).
   - Have no check of one error after `recv()`/`send()` (a negative result ends 
the connection): `apps/netutils/ftpd`, `telnetd`, `webserver`, `thttpd` 
(ignores `SIGPIPE`), `dropbear` (ignores `SIGPIPE`), `netcat`, `iperf`, 
`apps/testing/nettest` (the loopback server), `drivers/net/telnet.c`, 
`drivers/serial/netdev_console.c`, `drivers/video/vnc`, 
`drivers/tee/optee_socket.c`, `fs/v9fs/socket_9p.c`.
   - `drivers/usrsock/usrsock_rpmsg_server.c` passes the error to the remote 
side as it is; `POLLHUP` still gives `USRSOCK_EVENT_REMOTE_CLOSED`.
   - Only `ECONNREFUSED` of `connect()`: 
`apps/testing/testsuites/kernel/syscall/cases/connect_test.c`; a RST in 
`TCP_SYN_SENT` still gives it.
   - No in-tree user waits for `ENOTCONN` from `recv()` or for end-of-file 
after a reset.
   
   Build modes and options: common code, the same in flat, protected and kernel 
builds and on every architecture. With `CONFIG_NET_SOCKOPTS` off there is no 
`SO_ERROR`: `TCP_RESETERR` alone holds the pending error. With and without 
`CONFIG_NET_TCP_WRITE_BUFFERS` the send paths (`tcp_send_buffered.c`, 
`tcp_send_unbuffered.c`, `tcp_sendfile.c`) report the reset where they return 
the error. `CONFIG_NET_SOLINGER`: an abortive close of the peer is a RST like 
any other. SMP: `tcp_input()` marks the connection under the device lock; 
`tcp_reset_report()` clears the mark under the device and connection locks 
(recursive mutexes; it is called with and without them held).
   
   Size: see the build table below (144 to 448 bytes of text on arm64 and 
riscv64, about 850 on sim).
   
   ## Testing
   
   qemu-armv8a under HVF (Apple M-series), TCP over the loopback device: 
`NET_IPv4`, `NET_TCP`, `NET_TCPBACKLOG`, `NET_LOOPBACK`, `NET_SOCKOPTS`, 
`NET_SOLINGER`, `NET_TCP_WRITE_BUFFERS`, `NET_TCP_KEEPALIVE`, `NET_STATISTICS`, 
64 IOBs of 1534 bytes; the default pools (8 connections, 16 devif callbacks) 
unless "dynamic" (`NET_TCP_ALLOC_CONNS=1`, `NET_ALLOC_DEVIF_CALLBACKS=1`). 
knsh: kernel build, the programs from `/system/bin`; nsh: flat build. Master is 
fc8d354f9a1 (the tests ran on it; this PR is rebased on the latest master 
before it opened).
   
   Tests:
   
   - `cmocka_net_tcp` with the 6 new cases of apache/nuttx-apps#MMMM 
(`testing/nettest/tcp/test_tcp_reset.c`), 2 runs in one boot (3 on 4 CPUs). The 
peer makes the reset by a close with data unread (RFC 1122, 4.2.2.13), which 
sends a RST on master too.
   - `tcpshutdata` (a standalone program from the work on apache/nuttx#22025): 
its 5 reset checks (`SO_LINGER` {1, 0} with data queued, a `recv()` that waits 
at the reset, `send()` twice and with `SIGPIPE`, `poll()` and `SO_ERROR`, 
`shutdown()`/`close()` after the reset). Its other checks test 
apache/nuttx#22025 (`shutdown(SHUT_WR)` with data queued): they fail on master 
and with this PR alone, and pass with both.
   
   | Configuration | master | this PR |
   |---|---|---|
   | qemu-armv8a:knsh, 1 CPU: the 6 new cmocka cases | 5 fail (`refused` 
passes) | 6 pass, in each run |
   | qemu-armv8a:knsh, 1 CPU: tcpshutdata reset checks | 4 of 5 fail | 5 of 5 
pass |
   | qemu-armv8a:knsh, 4 CPUs (`CONFIG_SMP`), dynamic pools: cmocka x3, 
tcpshutdata, ostest | not run | 6 pass in each run; 5 of 5 reset checks; ostest 
status 0 |
   | qemu-armv8a:nsh (flat): the 6 new cmocka cases | 5 fail | 6 pass |
   | qemu-armv8a:nsh without `NET_TCP_WRITE_BUFFERS` | not run | 6 pass |
   | qemu-armv8a:nsh without `NET_SOCKOPTS` (5 cases: no `refused`, `poll` 
checks `recv()`) | the 5 fail | 5 pass |
   
   `test_tcp_connect_ipv4` (an older case) fails in every run, on master too: 
its loopback server refuses the connection, and the group teardown then fails.
   
   Without `NET_SOCKOPTS` `cmocka_net_tcp` does not link on apps master 
(`test_tcp_connect_ipv4.c` and `nettest_tcpserver.c` call `setsockopt()`); that 
run used a local change that puts those two calls under `CONFIG_NET_SOCKOPTS` 
(not in any PR).
   
   With apache/nuttx#22025: the 3 commits of this PR apply on it without a 
conflict (`git merge-tree`; the two PRs change different lines of 
`net/tcp/tcp.h`, and different places of `tcp_state_machine.rst`). On #22025 + 
this PR, qemu-armv8a:knsh 1 CPU: tcpshutdata 12 of 12 (on #22025 alone the 4 
reset checks fail), cmocka 6 of 6 new cases. This PR does not need #22025: the 
cases make the reset in a way that master handles.
   
   Builds in the NuttX CI image (`EXTRAFLAGS="-Wno-cpp -Werror"`), master and 
this PR: all OK, 0 warnings. Text size (bytes), master -> this PR (master 
fc8d354f9a1; the master values of the rows marked * come from the same image 
and master in the work on apache/nuttx#22025, the others were built again here):
   
   | Configuration | Build mode | master | this PR |
   |---|---|---|---|
   | qemu-armv8a:netnsh (no write buffers) | flat | 547544 | 547992 (+448) |
   | qemu-armv8a:netnsh + `NET_TCP_WRITE_BUFFERS`, `NET_SOLINGER` | flat | 
550560 | 550980 (+420) |
   | the same, CMake | flat | 551344 | 551764 (+420) |
   | qemu-armv8a:netnsh_smp + `NET_TCP_WRITE_BUFFERS`, `NET_SOLINGER` * | flat, 
SMP | 526056 | 526476 (+420) |
   | rv-virt:pnsh64 + TCP, write buffers, `NET_SOLINGER` * | protected | 183712 
| 183856 (+144) |
   | rv-virt:pnsh64 + TCP, no write buffers * | protected | 181264 | 181680 
(+416) |
   | rv-virt:pnsh64 + TCP, write buffers, no `NET_SOCKOPTS` * | protected | 
181680 | 182064 (+384) |
   | rv-virt:knetnsh64 * | kernel | 227879 | 228229 (+350) |
   | rv-virt:knetnsh64 + write buffers, `NET_SOLINGER`, dynamic pools * | 
kernel | 230299 | 230619 (+320) |
   | sim:tcpblaster * | sim | 993650 | 994490 (+840) |
   | sim:tcpblaster without write buffers and `NET_SOLINGER` * | sim | 984922 | 
985770 (+848) |
   | lm3s6432-s2e:nsh (96 KiB flash) * | flat | 63344 | 63716 (+372), flash 
65.72 % |
   
   In qemu-armv8a:netnsh `NET_SOCKOPTS` cannot be turned off (an enabled 
program selects it), so the builds without it are rv-virt:pnsh64 (CI image) and 
qemu-armv8a:nsh (the runtime test above). The test of apache/nuttx-apps#MMMM 
also builds with `-Werror` in sim:tcpblaster (CI image) and in qemu-armv8a:knsh 
and :nsh (macOS).
   
   `tools/checkpatch.sh -c -u -m -g` passes. Documentation: commit 3.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to