casaroli opened a new issue, #22088:
URL: https://github.com/apache/nuttx/issues/22088

   ### Description
   
   With `CONFIG_NET_TCP_WRITE_BUFFERS` and the default IOB pool (96 IOBs of 196 
bytes, `CONFIG_IOB_THROTTLE` 8), a large TCP transfer between two sockets on 
127.0.0.1 stops after 13 to 20 KB. This also occurs between two threads of one 
process. The sender retransmits, and after about 13.5 s the connection is 
reset: `send()` fails with `ENOTCONN` and `recv()` with `ECONNRESET`. On 
ea5d2c9ae49 alone (without the later commits of #22000) the transfer does not 
stop but goes on at about 4.5 KB/s.
   
   ### Reproducer
   
   qemu-armv8a:knsh (also a flat build), `CONFIG_NET_LOOPBACK`, 
`CONFIG_NET_TCP_WRITE_BUFFERS`, `CONFIG_IOB_NBUFFERS=96`, the other options at 
their defaults:
   
   ```c
   #include <netinet/in.h>
   #include <pthread.h>
   #include <stdio.h>
   #include <string.h>
   #include <sys/socket.h>
   #include <unistd.h>
   
   #define TOTAL (8 * 1024 * 1024)
   
   static struct sockaddr_in g_addr;
   static int g_listen;
   static char g_rbuf[65536];
   static char g_sbuf[65536];
   
   static void *receiver(void *arg)
   {
     long total = 0;
     ssize_t n;
     int s = accept(g_listen, NULL, NULL);
   
     while ((n = recv(s, g_rbuf, sizeof(g_rbuf), 0)) > 0)
       {
         total += n;
       }
   
     printf("received %ld of %d bytes\n", total, TOTAL);
     close(s);
     return NULL;
   }
   
   int main(void)
   {
     socklen_t len = sizeof(g_addr);
     pthread_t tid;
     long sent = 0;
     int s;
   
     g_addr.sin_family = AF_INET;
     g_addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
     g_listen = socket(AF_INET, SOCK_STREAM, 0);
     bind(g_listen, (struct sockaddr *)&g_addr, sizeof(g_addr));
     listen(g_listen, 1);
     getsockname(g_listen, (struct sockaddr *)&g_addr, &len);
     pthread_create(&tid, NULL, receiver, NULL);
   
     s = socket(AF_INET, SOCK_STREAM, 0);
     connect(s, (struct sockaddr *)&g_addr, sizeof(g_addr));
     while (sent < TOTAL)
       {
         ssize_t n = send(s, g_sbuf, sizeof(g_sbuf), 0);
   
         if (n <= 0)
           {
             perror("send");
             break;
           }
   
         sent += n;
       }
   
     printf("sent %ld bytes\n", sent);
     close(s);
     pthread_join(tid, NULL);
     return 0;
   }
   ```
   
   qemu-armv8a:knsh under HVF, 1 CPU:
   
   ```
   master ed122374690:
   send: Transport endpoint is not connected
   sent 46676 bytes
   received 14674 of 8388608 bytes
   
   9df78aa1e69 (before #22000):
   sent 8388608 bytes
   received 8388608 of 8388608 bytes
   ```
   
   The same with 4 CPUs (master: 223106 bytes received), in a flat build 
(qemu-armv8a:nsh: 23330 of 1 MiB, then `ECONNRESET`), and with `tcpthru` (20 
MiB between two threads: `send()` fails with `ENOTCONN` after 13.5 s). Without 
`CONFIG_NET_TCP_WRITE_BUFFERS` the transfer completes.
   
   ### Bisect
   
   9df78aa1e69 (the parent of ea5d2c9ae49): the transfer completes (8 MiB in 
0.1 s). ea5d2c9ae49 "net/tcp: keep the throttled IOBs out of the read-ahead 
buffer." (#22000): it does not complete (about 4.5 KB/s, a retransmission 
timeout for each segment that gets through).
   
   ### Cause
   
   Since ea5d2c9ae49, `tcp_datahandler()` drops the received data while 
`iob_navail(false) < CONFIG_IOB_THROTTLE`. The write buffers of the sender can 
use all IOBs above the throttle (88 of 96), and the received segment holds some 
of the other 8. The receiver drops the segment, so no ACK frees the write 
buffers, and the next segment is dropped again. A log in `tcp_datahandler()` at 
each drop shows the pool at that time:
   
   ```
   free 1, sender write queue 87 IOBs, receiver read-ahead 0 IOBs, received 
segment 8 IOBs (1478 bytes), caller tcp_data_event()
   ```
   
   The read-ahead buffer of the receiver is empty, and `tcp_get_recvwindow()` 
offers one MSS to an empty read-ahead buffer when no throttled IOB is free (the 
comment there names the fallback of `tcp_datahandler()`). Until 34d2cde8a8f, 
`tcp_datahandler()` had that fallback: an empty read-ahead buffer took the data 
from the unthrottled IOBs (bf210560017, "net/tcp: fallback to unthrottle pool 
to avoid deadlock"). ea5d2c9ae49 brought the throttle back without the fallback.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to