GitHub user michaelrommel added a comment to the discussion: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3
OK - here we go: Thank you so much for pointing out the error with the Bytes crate and showing me the correct way to make the allocated memory go down. I could reproduce that clearly in my test program. Unfortunately that will not help me with the actual problem that I was battling: I was setting up an FTP server using `libunftp` and the OpenDAL `unftp-sbe-opendal` crate and I encountered the problem when I was sending small packets in longer intervals. My server ran out of memory and crashed. It was an AWS Fargate container with 24GB RAM and I had 1000 connections each sending a 30MB file in 4kB chunks (the default), just 20ms apart. That's when I started the investigation. So that's why I thought the network is the issue. But it seems that alone is not the case, as your playground code indicated and when I put it in the receiver it worked fine as expected. Now the next thing I did was to alter the receiver to use a similar code as in the `unftp-sbe-opendal` crate, namely to use `tokio::io::copy()` to hand over the bytes. Unfortunately it is not easily possible to look at the allocations in regular intervals, as the `copy()` function is just one line of code. I tried to get the mem size from the outside running shell scripts in parallel, but I do not trust macOS on ARM enough to give me the right readings. Tomorrow I can test that again on an Fargate container... In the meantime I wrote a small ProgressReader, that does the instrumentation on each call, but the reported values will certainly a bit higher, since another buffer is involved. But the underlying issue can still be seen. Here is the new graph: <img width="1430" height="844" alt="image" src="https://github.com/user-attachments/assets/360681fd-4016-46f0-a95c-eba0ea7a39de"; /> The 1k, 4k, 32k lines all go down, the larger the packets are that have been sent, the less problematic is the queue. The two red-ish lines 2nd and 3rd from top show that when the sender sends the standard 4k size as fast as possible, the receiver is sometimes not able to get each and every packet from the OS (tokiocopy-std-nosleep), so the OS hands over larger packets to the app on the `socket.read()` call, hence the mem consumption is slightly lower than the tokiocopy-std-4ms line, where the sender waits 4ms between each send. If both receiver and sender wait 4ms between sends and reads, the problem is still visible. Only, if I artificially slow down the receiver, by introducing a sleep between polls/reads, then the memory consumption drops to the expected level. But still I have no clue, why the 1k line is so high... Since `unftp-sbe-opendal` is part of the OpenDAL repo, I wanted to share this finding. I may be wrong, but if an FTP server is based on that, there is a theoretical possibility that this could be used as an attack vector (which I stumbled on unintentionally)... Of course rate limiting etc. are other mitigations, but in my case I cannot use that, unfortunately... Maybe you have some thoughts on the new code in `src/receiver-tokiocopy`. Thanks a lot, will share more test tomorrow, when I can access the Fargate infrastructure... Michael. GitHub link: https://github.com/apache/opendal/discussions/7200#discussioncomment-15891814 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
