rickyma commented on PR #1759: URL: https://github.com/apache/incubator-uniffle/pull/1759#issuecomment-2144469049
> Currently we have put the shuffle data into the off-heap memory in shuffle server . But I found it still occupancy a lot of heap memory. From the above results, we can see that the main reason for high memory usage is that there are too many blocks. And the reason why there are so many blocks is because the blocks are very small. It can solve the problem of high heap memory usage caused by too many small blocks as you mentioned in the issue in another way. If you still want to implement something like MSLAB, maybe we can make it a feature and consider to implement it later on. Currently, I think this PR will be enough for most small blocks cases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
