GitHub user michaelrommel added a comment to the discussion: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3
Thank you for the quick answer! I will give that a try, but I should mention, that the problem originally occurred, when I was using tokio::io::copy() between the streams. On the sending side, there was a client from `suppaftp` talking to an FTP backend using `libunftp` with an OpenDAL S3 backend. I originally thought, it is a problem with libunftp, but then I tried to eliminate possible causes. First remove libunftp, then remove the `tokio::io::copy` etc. I started out also with a smaller buffersize like 4k. Only in the latest version I bumped up the buffer to 128k in order to test with larger chunks from the sender. As far as I understand, `tokio::io::copy` uses a 2k buffer internally and there the same issue occurs... But I will first give your example a spin and try to understand the issue. GitHub link: https://github.com/apache/opendal/discussions/7200#discussioncomment-15890028 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
