As another "me too" situation, I'm seeing the same phenomenon, though on Rocky 9 rather than Ubuntu and on older kernels (5.14). Reporting details here on the off chance this provides some insight.
Hardware: Ampere Altra Max 128 cores (aarch64), ConnectX6-DX NICs (2 x dual 100G port) Kernel versions tested: 5.15 (Rocky 9 native kernel) and 6.8 (elrepo kernel), both configured with 64 KB pages OS: Rocky Linux 9 Software: nginx serving ~90k HTTPS clients at ~350 GBit/s (a synthetic load test) Bare-metal (no virtualisation). In my test environment, ~90k HTTPS connections are opened (and reused via keepalive) and used to stream ~350 GBit/s of traffic to a cluster of load generators. In this scenario, TCP memory gradually creeps up until reaching the memory pressure threshold in /proc/sys/net/ipv4/tcp_mem (243890 pages, or 15.6 GB in this system). At this point memory usage growth actually increases slightly (and increased CPU load and response times are also observed). The system eventually reaches the ultimate limit (365832 pages, or 23.4 GB) at which point most connections fail and all requests receive very slow responses. Closing all connections or restarting nginx does not free up the memory, only a reboot resolves the situation -- as reported above already. Leaked memory appears to persist even if all connections are closed prior to hitting any of the above limits. Unfortunately I don't yet have any ideas on how to fix this but would be glad to hear (and will share) any insights about what might be going on here! ** Attachment added: "TCP memory consumption" https://bugs.launchpad.net/ubuntu/+source/linux-signed-aws-6.2/+bug/2045560/+attachment/5763728/+files/boom.png -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2045560 Title: TCP memory leak, slow network (arm64) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-signed-aws-6.2/+bug/2045560/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
