rickyma opened a new pull request, #1759: URL: https://github.com/apache/incubator-uniffle/pull/1759
### What changes were proposed in this pull request? Introduce blocks number threshold when flushing a single buffer to reduce gc. ### Why are the changes needed? Fix: https://github.com/apache/incubator-uniffle/issues/1727. In a production environment, the Uniffle server may run jobs with various unreasonable configurations. These jobs might have a large number of partitions (tens of thousands, hundreds of thousands, or even millions), or they might have manually been configured with a very small spill size, or some other reasons. This can ultimately bring a large number of blocks to the server, and the server has no choice but to maintain them in the heap memory for a long time, simply because their data size does not meet the conditions for flushing. This can cause severe garbage collection issues on the server side, and in extreme cases, it can even lead to out-of-heap-memory errors. In Netty mode, we use off-heap memory to store shuffle data. However, when facing jobs with extremely unreasonable configurations, the total size of the reference objects of the blocks maintained in the heap memory by the server may even exceed the size of the data stored off-heap. This can bring great instability to the server. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing UTs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
