matanper opened a new issue, #70721: URL: https://github.com/apache/doris/issues/70721
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4. master (`13cff7ff484`) has the same code. ### What's Wrong? On cgroup v2 with workload group CPU control enabled, the BE stops seeing its cgroup memory limit and sizes its memory from the host's `MemTotal`. In a container with a memory limit, `mem_limit` and the soft limit then sit at or above the limit the kernel enforces. The BE is OOM-killed by the kernel before its own memory GC or query cancellation acts. From a BE in a container limited to 144 GiB on a 192 GiB host: ``` sys physical memory 185.30 GB. process memory used 120.59 GB ..., limit 166.77 GB, soft limit 150.10 GB. ``` BEs in the same deployment without workload group CPU control report their container limit. For example, a BE in a 26 GiB container logs `sys physical memory 26.00 GB ... limit 23.40 GB, soft limit 21.06 GB`. ### Why 1. With cgroup v2 CPU control, `CgroupCpuCtl::init_cgroup_v2_query_path_public_file` (`be/src/agent/cgroup_cpu_ctl.cpp:148-154` on master) writes the BE's pid into `<doris_cgroup_cpu_path>/query/cgroup.procs`. The whole BE process now lives in the workload-group query cgroup. 2. The memory limit lookup (`get_cgroups_path` in `be/src/common/cgroup_memory_ctl.cpp`) first calls `cgroupsv2_memory_controller_enabled`. That reads only the process's own `cgroup.controllers`, on the assumption that "it is sufficient to check the bottom-most nested `cgroup.controllers` file". 3. The query cgroup does not list `memory`: - If the CPU subtree is threaded, only threaded controllers (`cpu`, `cpuset`, `pids`, ...) can be enabled there, so `memory` can never appear. - In a domain subtree, `CgroupCpuCtl` enables only `+cpu` in `subtree_control`, so `memory` does not appear there either. 4. `cgroupsv2_memory_controller_enabled` returns false, the v1 lookup fails, and `MemInfo::refresh_proc_meminfo` keeps `_s_cgroup_mem_limit` at `INT64_MAX`. `physical_mem` stays at `MemTotal`. The BE's memory is still charged to the nearest ancestor that has the memory controller (the container's cgroup), and that cgroup's `memory.max` is what the kernel enforces. `CGroupUtil::get_cgroupsv2_path` already walks up to find `memory.max`, but the controller check before it rejects the process first. ### What You Expected? The BE finds the memory limit of the cgroup its memory is charged to, the nearest ancestor with the memory controller, whether or not it has moved itself into a workload group cgroup. It then derives `mem_limit` and the soft limit from that limit. ### How to Reproduce? 1. Run a BE on cgroup v2 in a container with a memory limit below the host's memory (for example `docker run --memory=8g ...` on a 32 GB host), with workload group CPU control enabled (`doris_cgroup_cpu_path` set to a subtree under the container's cgroup, with `cpu` delegated). 2. Check `cat /proc/$(pgrep doris_be)/cgroup`: it ends in `/query`. 3. Check be.INFO: `sys physical memory` reports the host's memory, not 8 GB, and `limit`/`soft limit` are derived from it. The lookup can also be seen without a BE, against a cgroup layout like: ``` <container cgroup> cgroup.controllers: cpuset cpu io memory pids, memory.max: 8589934592 <container cgroup>/doris cgroup.type: threaded <container cgroup>/doris/query cgroup.type: threaded, cgroup.controllers: cpuset cpu pids ``` with the process in `doris/query`: `cgroupsv2_memory_controller_enabled` reports false. ### Anything Else? Proposed fix: replace the bottom-most check with a walk from the process's cgroup up to the cgroup mount. Stop at the first cgroup that lists `memory` in `cgroup.controllers` and has `memory.max`, `memory.current` and `memory.stat`, and use that directory for both the limit and the usage. When the process's own cgroup has the memory controller, including a private cgroup namespace (`0::/`), the result is the same as today. We have a patch with unit tests and can open a PR. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
