matanper opened a new issue, #70721:
URL: https://github.com/apache/doris/issues/70721

   ### Search before asking
   
   - [x] I had searched in the 
[issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no 
similar issues.
   
   ### Version
   
   4.1.4. master (`13cff7ff484`) has the same code.
   
   ### What's Wrong?
   
   On cgroup v2 with workload group CPU control enabled, the BE stops seeing 
its cgroup memory limit and sizes its memory from the host's `MemTotal`. In a 
container with a memory limit, `mem_limit` and the soft limit then sit at or 
above the limit the kernel enforces. The BE is OOM-killed by the kernel before 
its own memory GC or query cancellation acts.
   
   From a BE in a container limited to 144 GiB on a 192 GiB host:
   
   ```
   sys physical memory 185.30 GB. process memory used 120.59 GB ..., limit 
166.77 GB, soft limit 150.10 GB.
   ```
   
   BEs in the same deployment without workload group CPU control report their 
container limit. For example, a BE in a 26 GiB container logs `sys physical 
memory 26.00 GB ... limit 23.40 GB, soft limit 21.06 GB`.
   
   ### Why
   
   1. With cgroup v2 CPU control, 
`CgroupCpuCtl::init_cgroup_v2_query_path_public_file` 
(`be/src/agent/cgroup_cpu_ctl.cpp:148-154` on master) writes the BE's pid into 
`<doris_cgroup_cpu_path>/query/cgroup.procs`. The whole BE process now lives in 
the workload-group query cgroup.
   2. The memory limit lookup (`get_cgroups_path` in 
`be/src/common/cgroup_memory_ctl.cpp`) first calls 
`cgroupsv2_memory_controller_enabled`. That reads only the process's own 
`cgroup.controllers`, on the assumption that "it is sufficient to check the 
bottom-most nested `cgroup.controllers` file".
   3. The query cgroup does not list `memory`:
      - If the CPU subtree is threaded, only threaded controllers (`cpu`, 
`cpuset`, `pids`, ...) can be enabled there, so `memory` can never appear.
      - In a domain subtree, `CgroupCpuCtl` enables only `+cpu` in 
`subtree_control`, so `memory` does not appear there either.
   4. `cgroupsv2_memory_controller_enabled` returns false, the v1 lookup fails, 
and `MemInfo::refresh_proc_meminfo` keeps `_s_cgroup_mem_limit` at `INT64_MAX`. 
`physical_mem` stays at `MemTotal`.
   
   The BE's memory is still charged to the nearest ancestor that has the memory 
controller (the container's cgroup), and that cgroup's `memory.max` is what the 
kernel enforces. `CGroupUtil::get_cgroupsv2_path` already walks up to find 
`memory.max`, but the controller check before it rejects the process first.
   
   ### What You Expected?
   
   The BE finds the memory limit of the cgroup its memory is charged to, the 
nearest ancestor with the memory controller, whether or not it has moved itself 
into a workload group cgroup. It then derives `mem_limit` and the soft limit 
from that limit.
   
   ### How to Reproduce?
   
   1. Run a BE on cgroup v2 in a container with a memory limit below the host's 
memory (for example `docker run --memory=8g ...` on a 32 GB host), with 
workload group CPU control enabled (`doris_cgroup_cpu_path` set to a subtree 
under the container's cgroup, with `cpu` delegated).
   2. Check `cat /proc/$(pgrep doris_be)/cgroup`: it ends in `/query`.
   3. Check be.INFO: `sys physical memory` reports the host's memory, not 8 GB, 
and `limit`/`soft limit` are derived from it.
   
   The lookup can also be seen without a BE, against a cgroup layout like:
   
   ```
   <container cgroup>            cgroup.controllers: cpuset cpu io memory pids, 
memory.max: 8589934592
   <container cgroup>/doris       cgroup.type: threaded
   <container cgroup>/doris/query cgroup.type: threaded, cgroup.controllers: 
cpuset cpu pids
   ```
   
   with the process in `doris/query`: `cgroupsv2_memory_controller_enabled` 
reports false.
   
   ### Anything Else?
   
   Proposed fix: replace the bottom-most check with a walk from the process's 
cgroup up to the cgroup mount. Stop at the first cgroup that lists `memory` in 
`cgroup.controllers` and has `memory.max`, `memory.current` and `memory.stat`, 
and use that directory for both the limit and the usage. When the process's own 
cgroup has the memory controller, including a private cgroup namespace 
(`0::/`), the result is the same as today.
   
   We have a patch with unit tests and can open a PR.
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to