xianjingfeng commented on PR #1759: URL: https://github.com/apache/incubator-uniffle/pull/1759#issuecomment-2144748703
> emmm. This might be due to the relatively small memory configuration of the executor, which is actually an issue with the task resource configurations. How many reduce partitions do you have in your downstream stage? There must be a lot, right? This can also be optimized from the perspective of task configurations. How much memory is configured for your executors? What is the spill size value? How many reduce partitions are there in the downstream stage? Have you tried `rss.client.memory.spill.ratio` mentioned in [#1727 (comment)](https://github.com/apache/incubator-uniffle/issues/1727#issuecomment-2121933432). It will help you reduce small blocks. We can't limit how users use, so it is normal that there are many reduce partitions. And it seems that the larger the spill value is, the more off-heap memory will be used. That's why the executors are killed by yarn and i don't have time to find out more details currently. So we dare not modify any memory-related configuration uniformly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
