Doris-Breakwater commented on issue #70721:
URL: https://github.com/apache/doris/issues/70721#issuecomment-6975119819

   Breakwater-GitHub-Analysis-Slot: slot_a165d07036bf
   
   The reported controller-discovery defect is supported by the source. I would 
triage this as a high-impact BE memory-detection bug affecting cgroup v2 
deployments where workload-group CPU initialization moves the BE below the 
cgroup exposing its memory interfaces. The kernel memory limit remains 
enforced; Doris loses visibility of it. The issue currently has no labels.
   
   **Verified from source**
   
   I checked the local `4.1.4` tag (`ad35a140c7fd`) and the cited master commit 
`13cff7ff484596f29aa1381750e906db3c8bb704`. Their `cgroup_memory_ctl.cpp`, 
`cgroup_util.cpp`, and `mem_info.cpp` are identical; the relevant CPU 
initialization behavior is also unchanged.
   
   - 
[`CgroupCpuCtl::init_cgroup_v2_query_path_public_file`](https://github.com/apache/doris/blob/13cff7ff484596f29aa1381750e906db3c8bb704/be/src/agent/cgroup_cpu_ctl.cpp#L129-L155)
 enables `+cpu` and writes `getpid()` into `query/cgroup.procs`. It does not 
propagate the memory controller. Whether `memory` is absent depends on the 
existing hierarchy; enabling CPU control alone does not prove its absence.
   - [`cgroupsv2_memory_controller_enabled` and 
`get_cgroups_path`](https://github.com/apache/doris/blob/13cff7ff484596f29aa1381750e906db3c8bb704/be/src/common/cgroup_memory_ctl.cpp#L34-L171)
 check only the process's current `cgroup.controllers`. If it lacks `memory`, 
the v2 path is rejected before the existing ancestor lookup runs. With no v1 
memory membership, reader creation fails.
   - 
[`CGroupUtil::get_cgroupsv2_path`](https://github.com/apache/doris/blob/13cff7ff484596f29aa1381750e906db3c8bb704/be/src/util/cgroup_util.cpp#L197-L219)
 already searches ancestors, including the visible mount root. The premature 
controller gate prevents that behavior in the reported layout.
   - 
[`MemInfo::refresh_proc_meminfo`](https://github.com/apache/doris/blob/13cff7ff484596f29aa1381750e906db3c8bb704/be/src/util/mem_info.cpp#L97-L206)
 explicitly resets the cgroup limit to `INT64_MAX` after lookup failure and 
schedules a retry after approximately 300 seconds. Without a successful cgroup 
refresh, both physical-memory sizing and available-memory reporting fall back 
to `/proc/meminfo`. Percentage `mem_limit` is recalculated from that physical 
memory, and the soft limit is derived from it. This explains the supplied 
host-sized thresholds. A successful limit read alone is insufficient: usage 
refresh must succeed too.
   
   The [kernel cgroup v2 
documentation](https://docs.kernel.org/admin-guide/cgroup-v2.html#threads) 
confirms that threaded subtrees support only threaded controllers, excluding 
`memory`; their domain resource consumption belongs to the threaded domain. 
Thus, absence of `memory` at the CPU leaf does not establish absence of 
ancestor memory enforcement.
   
   **Suggested fix and validation**
   
   Please open the proposed PR. Resolving an ancestor directory with the memory 
controller and the required memory interfaces, then using that directory 
consistently for limit and usage, addresses the demonstrated discovery failure. 
Keep the search bounded by the visible cgroup mount, including 
private-namespace membership `0::/`, and preserve read-error diagnostics and v1 
fallback.
   
   One scope caveat: the nearest directory with memory interfaces is not 
necessarily the tightest enforcing ancestor. A selected child can have 
`memory.max=max` or a larger limit while a higher ancestor is capped. The 
current reader also stops at the nearest interface, so this is an existing 
limitation, not evidence against the proposed fix. Test/document that case 
separately; simply mixing an ancestor limit with a child's usage would 
misrepresent shared ancestor pressure.
   
   Requested regression coverage:
   
   - CPU-only descendants, both threaded and domain layouts, resolving the 
ancestor's limit **and** usage.
   - Existing leaf memory-controller behavior, private namespace root `0::/`, 
and mount-boundary termination.
   - Unlimited `memory.max`, absent/unreadable interfaces, and v1 behavior.
   - An integration check after CPU initialization and subsequent memory 
refresh: with an 8 GiB container and default `mem_limit=90%`, 
`soft_mem_limit_frac=0.9`, expect approximately 8 GiB physical memory, 7.2 GiB 
process limit, and 6.48 GiB soft limit when the selected memory cgroup is the 
capped container cgroup.
   
   **Evidence still needed for the observed OOM**
   
   The code path is verified; I have not run a BE/container reproduction or the 
proposed patch's tests. Please attach:
   
   - Kernel/container-runtime versions, cgroup namespace mode, 
`/proc/<BE_PID>/cgroup`, and relevant cgroup2 mount entries from the BE's mount 
namespace.
   - From the BE-visible leaf through the container ancestor: `cgroup.type`, 
`cgroup.controllers`, `cgroup.subtree_control`, `memory.max`, `memory.current`, 
and `memory.stat`, noting missing files.
   - Effective `doris_cgroup_cpu_path`, `enable_use_cgroup_memory_info`, 
`mem_limit`, and `soft_mem_limit_frac`; timestamped CPU initialization, 
`Refresh cgroup memory limit failed`, and memory-summary logs.
   - Kernel OOM victim records and changes in `memory.events` (`max`, `oom`, 
`oom_kill`) at the enforcing cgroup, to distinguish this sizing defect from 
other causes of termination.
   
   Pending a fix, an absolute `mem_limit` below the container's enforced limit, 
with headroom for other charged memory, can restore lower process thresholds 
after a BE restart. The checked code supports absolute units; this mitigation 
does not restore cgroup usage visibility or guarantee protection from kernel 
OOM.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to