casaroli opened a new pull request, #22029:
URL: https://github.com/apache/nuttx/pull/22029
## Summary
When the peer resets an established TCP connection (a RST), NuttX did not
report it. `recv()` gave end-of-file after the data that came before the RST,
as for a FIN, so a program could not tell a stream that was cut from one that
ended (a download cut by a reset looked complete). A `recv()` that waited at
the reset gave `ENOTCONN`, `send()` gave `EPIPE` at once, and `poll()` set
`SO_ERROR` to `ECONNREFUSED`. POSIX and Linux: the first receive or send fails
with `ECONNRESET`, and `SO_ERROR` gives it until it is reported.
1. **`fs/nfs: Reconnect also when a request fails with ECONNRESET.`**
`nfs_request()` connects to the server again when a request fails with
`ENOTCONN` or `EPIPE`. With commit 2 a reset of the NFS server over TCP gives
`ECONNRESET` to the first receive or send: reconnect on it too, so that the
reconnect continues to work (the same as 088075ce7fc did for `EPIPE`).
2. **`net/tcp: Report a reset of the peer as ECONNRESET.`** `tcp_input()`
marks the connection when a RST comes in `TCP_ESTABLISHED`, `TCP_FIN_WAIT_1` or
`TCP_FIN_WAIT_2` (`TCP_RESET`, `TCP_RESETERR`, `SO_ERROR` = `ECONNRESET`). The
first receive without data, send, `sendfile()` or `getsockopt(SO_ERROR)` that
returns the error reports it, once (`tcp_reset_report()`). After that a receive
gives end-of-file and a send `EPIPE` (and `SIGPIPE` without `MSG_NOSIGNAL`).
The data that came before the RST is read first. A receive or send that returns
a count keeps the error for the next call (`recv()` with `MSG_WAITALL` or with
more than one buffer, a `send()` that queued a part). `poll()` after the reset
gives `POLLHUP` and `POLLIN`, and `POLLERR` while the error is not reported. A
RST in `TCP_SYN_SENT` still gives `ECONNREFUSED`; after the FIN of the peer a
receive still gives end-of-file and a send `EPIPE`.
3. **`Documentation/os/networking: Describe what a reset of the peer
gives.`** The TCP state machine page.
The change is in common network code: the same in flat, protected and kernel
builds, on every architecture.
## Behaviour
After a RST of the peer on an established connection (the peer closed with
data unread), qemu-armv8a:knsh, measured with the cases of
apache/nuttx-apps#MMMM:
| Call | master | this PR (and Linux) |
|---|---|---|
| `recv()` with 5 bytes of the peer unread | 5 bytes, then 0 (end-of-file) |
5 bytes, then -1 `ECONNRESET`, then 0 |
| `recv()` that waits at the reset | -1 `ENOTCONN` | -1 `ECONNRESET`, then 0
|
| `recv(MSG_WAITALL)` that waits, 5 bytes came before the reset | 5, then 0
| 5, then -1 `ECONNRESET`, then 0 |
| `send()` | -1 `EPIPE` (and `SIGPIPE`) | -1 `ECONNRESET` (no `SIGPIPE`),
then -1 `EPIPE` (and `SIGPIPE`) |
| `getsockopt(SO_ERROR)` | 0, or `ECONNREFUSED` once a `poll()` ran |
`ECONNRESET`, then 0 |
| `poll(POLLIN \| POLLOUT)` with data unread | `POLLIN \| POLLERR \|
POLLHUP`, `POLLERR` stays | `POLLIN \| POLLERR \| POLLHUP`; after `SO_ERROR`:
`POLLIN \| POLLHUP` |
| non-blocking `connect()` refused (RST in `TCP_SYN_SENT`) | `POLLERR`,
`SO_ERROR` `ECONNREFUSED` | the same |
| RST after the FIN of the peer (`TCP_CLOSE_WAIT`) | end-of-file, `EPIPE` |
the same |
POSIX: `recv()` and `send()` list `ECONNRESET` ("A connection was forcibly
closed by a peer"); `send()` gives `EPIPE` and `SIGPIPE` when the socket "is no
longer connected"; `SO_ERROR` "reports information about error status and
clears it"; `POLLHUP` and `POLLOUT` exclude each other. Linux `tcp_reset()`
sets `ECONNREFUSED` in `TCP_SYN_SENT`, `EPIPE` in `TCP_CLOSE_WAIT` and
`ECONNRESET` in the other states; `tcp_recvmsg()` gives the queued data first,
then the error once, then end-of-file. Linux also sets `POLLOUT` with
`POLLHUP`; `poll_notify()` of NuttX clears `POLLOUT` when `POLLHUP` is set, as
POSIX says. In `TCP_CLOSE_WAIT` NuttX does not set `SO_ERROR` to `EPIPE` as
Linux does: that is out of the scope of this PR (master behaviour kept).
## Impact
TCP only, plus one condition in `fs/nfs`. No API, ABI or Kconfig change. The
errors that programs see after a reset of the peer change as in the table.
In-tree users (nuttx and nuttx-apps, grep of `ECONNRESET`, `ENOTCONN`, `EPIPE`,
`ECONNREFUSED` and of the in-kernel `psock_*()` users):
- `fs/nfs/nfs_util.c`: reconnected only on `ENOTCONN` and `EPIPE`; commit 1
adds `ECONNRESET`. Without it a request to an NFS server over TCP that reset
the connection would fail once instead of reconnecting. (`rpcclnt_receive()`
also loops on a `recv()` that returns 0; with this PR it gets `ECONNRESET`
first.)
- Handle `ECONNRESET` already, as a closed connection:
`apps/examples/tcpecho` ("client aborted connection"),
`apps/system/telnet/telnet_chatd.c`, `apps/examples/mqttc/mqttc_sub.c` (`EPIPE
|| ECONNRESET`), `apps/industry/nxmodbus` (any error closes).
- Have no check of one error after `recv()`/`send()` (a negative result ends
the connection): `apps/netutils/ftpd`, `telnetd`, `webserver`, `thttpd`
(ignores `SIGPIPE`), `dropbear` (ignores `SIGPIPE`), `netcat`, `iperf`,
`apps/testing/nettest` (the loopback server), `drivers/net/telnet.c`,
`drivers/serial/netdev_console.c`, `drivers/video/vnc`,
`drivers/tee/optee_socket.c`, `fs/v9fs/socket_9p.c`.
- `drivers/usrsock/usrsock_rpmsg_server.c` passes the error to the remote
side as it is; `POLLHUP` still gives `USRSOCK_EVENT_REMOTE_CLOSED`.
- Only `ECONNREFUSED` of `connect()`:
`apps/testing/testsuites/kernel/syscall/cases/connect_test.c`; a RST in
`TCP_SYN_SENT` still gives it.
- No in-tree user waits for `ENOTCONN` from `recv()` or for end-of-file
after a reset.
Build modes and options: common code, the same in flat, protected and kernel
builds and on every architecture. With `CONFIG_NET_SOCKOPTS` off there is no
`SO_ERROR`: `TCP_RESETERR` alone holds the pending error. With and without
`CONFIG_NET_TCP_WRITE_BUFFERS` the send paths (`tcp_send_buffered.c`,
`tcp_send_unbuffered.c`, `tcp_sendfile.c`) report the reset where they return
the error. `CONFIG_NET_SOLINGER`: an abortive close of the peer is a RST like
any other. SMP: `tcp_input()` marks the connection under the device lock;
`tcp_reset_report()` clears the mark under the device and connection locks
(recursive mutexes; it is called with and without them held).
Size: see the build table below (144 to 448 bytes of text on arm64 and
riscv64, about 850 on sim).
## Testing
qemu-armv8a under HVF (Apple M-series), TCP over the loopback device:
`NET_IPv4`, `NET_TCP`, `NET_TCPBACKLOG`, `NET_LOOPBACK`, `NET_SOCKOPTS`,
`NET_SOLINGER`, `NET_TCP_WRITE_BUFFERS`, `NET_TCP_KEEPALIVE`, `NET_STATISTICS`,
64 IOBs of 1534 bytes; the default pools (8 connections, 16 devif callbacks)
unless "dynamic" (`NET_TCP_ALLOC_CONNS=1`, `NET_ALLOC_DEVIF_CALLBACKS=1`).
knsh: kernel build, the programs from `/system/bin`; nsh: flat build. Master is
fc8d354f9a1 (the tests ran on it; this PR is rebased on the latest master
before it opened).
Tests:
- `cmocka_net_tcp` with the 6 new cases of apache/nuttx-apps#MMMM
(`testing/nettest/tcp/test_tcp_reset.c`), 2 runs in one boot (3 on 4 CPUs). The
peer makes the reset by a close with data unread (RFC 1122, 4.2.2.13), which
sends a RST on master too.
- `tcpshutdata` (a standalone program from the work on apache/nuttx#22025):
its 5 reset checks (`SO_LINGER` {1, 0} with data queued, a `recv()` that waits
at the reset, `send()` twice and with `SIGPIPE`, `poll()` and `SO_ERROR`,
`shutdown()`/`close()` after the reset). Its other checks test
apache/nuttx#22025 (`shutdown(SHUT_WR)` with data queued): they fail on master
and with this PR alone, and pass with both.
| Configuration | master | this PR |
|---|---|---|
| qemu-armv8a:knsh, 1 CPU: the 6 new cmocka cases | 5 fail (`refused`
passes) | 6 pass, in each run |
| qemu-armv8a:knsh, 1 CPU: tcpshutdata reset checks | 4 of 5 fail | 5 of 5
pass |
| qemu-armv8a:knsh, 4 CPUs (`CONFIG_SMP`), dynamic pools: cmocka x3,
tcpshutdata, ostest | not run | 6 pass in each run; 5 of 5 reset checks; ostest
status 0 |
| qemu-armv8a:nsh (flat): the 6 new cmocka cases | 5 fail | 6 pass |
| qemu-armv8a:nsh without `NET_TCP_WRITE_BUFFERS` | not run | 6 pass |
| qemu-armv8a:nsh without `NET_SOCKOPTS` (5 cases: no `refused`, `poll`
checks `recv()`) | the 5 fail | 5 pass |
`test_tcp_connect_ipv4` (an older case) fails in every run, on master too:
its loopback server refuses the connection, and the group teardown then fails.
Without `NET_SOCKOPTS` `cmocka_net_tcp` does not link on apps master
(`test_tcp_connect_ipv4.c` and `nettest_tcpserver.c` call `setsockopt()`); that
run used a local change that puts those two calls under `CONFIG_NET_SOCKOPTS`
(not in any PR).
With apache/nuttx#22025: the 3 commits of this PR apply on it without a
conflict (`git merge-tree`; the two PRs change different lines of
`net/tcp/tcp.h`, and different places of `tcp_state_machine.rst`). On #22025 +
this PR, qemu-armv8a:knsh 1 CPU: tcpshutdata 12 of 12 (on #22025 alone the 4
reset checks fail), cmocka 6 of 6 new cases. This PR does not need #22025: the
cases make the reset in a way that master handles.
Builds in the NuttX CI image (`EXTRAFLAGS="-Wno-cpp -Werror"`), master and
this PR: all OK, 0 warnings. Text size (bytes), master -> this PR (master
fc8d354f9a1; the master values of the rows marked * come from the same image
and master in the work on apache/nuttx#22025, the others were built again here):
| Configuration | Build mode | master | this PR |
|---|---|---|---|
| qemu-armv8a:netnsh (no write buffers) | flat | 547544 | 547992 (+448) |
| qemu-armv8a:netnsh + `NET_TCP_WRITE_BUFFERS`, `NET_SOLINGER` | flat |
550560 | 550980 (+420) |
| the same, CMake | flat | 551344 | 551764 (+420) |
| qemu-armv8a:netnsh_smp + `NET_TCP_WRITE_BUFFERS`, `NET_SOLINGER` * | flat,
SMP | 526056 | 526476 (+420) |
| rv-virt:pnsh64 + TCP, write buffers, `NET_SOLINGER` * | protected | 183712
| 183856 (+144) |
| rv-virt:pnsh64 + TCP, no write buffers * | protected | 181264 | 181680
(+416) |
| rv-virt:pnsh64 + TCP, write buffers, no `NET_SOCKOPTS` * | protected |
181680 | 182064 (+384) |
| rv-virt:knetnsh64 * | kernel | 227879 | 228229 (+350) |
| rv-virt:knetnsh64 + write buffers, `NET_SOLINGER`, dynamic pools * |
kernel | 230299 | 230619 (+320) |
| sim:tcpblaster * | sim | 993650 | 994490 (+840) |
| sim:tcpblaster without write buffers and `NET_SOLINGER` * | sim | 984922 |
985770 (+848) |
| lm3s6432-s2e:nsh (96 KiB flash) * | flat | 63344 | 63716 (+372), flash
65.72 % |
In qemu-armv8a:netnsh `NET_SOCKOPTS` cannot be turned off (an enabled
program selects it), so the builds without it are rv-virt:pnsh64 (CI image) and
qemu-armv8a:nsh (the runtime test above). The test of apache/nuttx-apps#MMMM
also builds with `-Werror` in sim:tcpblaster (CI image) and in qemu-armv8a:knsh
and :nsh (macOS).
`tools/checkpatch.sh -c -u -m -g` passes. Documentation: commit 3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]