yangzhg opened a new issue #6978: URL: https://github.com/apache/incubator-doris/issues/6978
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description Currently, the main reason that causes doris to exit abnormally is that it occupies too much memory and OOM is killed by the system. Although doris has a mem_limit configuration, the memory usage limit is not strict. As a result, some memory is not recorded and consumed in doris. There are two main parts of the rough: 1. The memory that can be tracked by memtracker is mainly occupied by various mempools 2. Memory that cannot be tracked through memtracker, mainly memory requested through new/malloc The current problem is as follows: 1. The memory that can be tracked by memtracker does not consider the total memory occupied by the current system when consuming, but only considers the mem limit of the current tracker. This situation is particularly prominent on the memtracker without memory display. 2. There are currently no restrictions on the memory requested through new/malloc ### Use case In order to solve the above two problems, what we need to do is as follows: 1. For the memory that can be tracked through the mem tracker, when you consume, check whether the used memory exceeds the system memory limit. If it exceeds the system memory limit, the application will fail directly. 2. For memory that cannot be tracked by the system, a hook for memory allocation needs to be set. Establish a mapping relationship between memory application and query. If the memory allocation exceeds the limit, the corresponding query will be cancelled. 3. Organize internal allocation relationships to reduce unnecessary memory allocation ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
