rickyma commented on PR #1759: URL: https://github.com/apache/incubator-uniffle/pull/1759#issuecomment-2147856361
> And it seems that the larger the spill value is, the more off-heap memory will be used. That's why the executors are killed by yarn and i don't have time to find out more details currently. I can try to explain this to you. When the size of the spill is larger, it means that more data is accumulated at the client side each time, so the amount of data sent each time is also larger. Therefore, more server responses are received in the same time period. After the netty mode is enabled, this part of the responses will occupy off-heap memory, which means that the off-heap memory may be used more during this time. If the off-heap memory configured for your executor is not large enough, it may cause OOM. This is why I have been recommending you to use the `spark.rss.client.memory.spill.ratio` parameter. If you enable this parameter and set it to less than 1.0, it means that you only send part of the spill data each time, which makes the client's requests smoother overall. When the value of the ratio is reached, a part is sent, instead of waiting until the end to send all at once. If you set up an off-heap memory monitoring for the client side, you will find that if the rati o is enabled, the used off-heap memory curve will be smoother. If the ratio is 1.0 (not enabled), the curve will be jagged, and the memory used at the peak may be very large, which may cause OOM. Below is a comparison graph for different memory spill ratios:  So again, `spark.rss.client.memory.spill.ratio` should help you mitigate executors being killed by yarn problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
