matanper opened a new issue, #70575: URL: https://github.com/apache/doris/issues/70575
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4 (`branch-4.1`). master (`e82d2650e38`) has the same code. ### What's Wrong? With the official Docker images on a cgroup v2 host, `ENABLE_WORKLOAD_GROUP=true` does not give workload groups any CPU control. The BE starts normally, and nothing at WARN or ERROR level says that CPU isolation is off. cgroup v2 is the default on current Kubernetes node images (EKS AL2023, recent Ubuntu, COS), so with the official images, workload group CPU limits do not apply on most Kubernetes clusters. Both BE entrypoints hard-code the cgroup v1 layout: - `docker/runtime/be/resource/be_disaggregated_entrypoint.sh:52` and `:525-538` - `docker/runtime/be/resource/be_entrypoint.sh:59` and `:462-475` ```bash WORKLOAD_GROUP_PATH="/sys/fs/cgroup/cpu/doris" ... mkdir -p /sys/fs/cgroup/cpu/doris chmod 770 /sys/fs/cgroup/cpu/doris chown -R root:root /sys/fs/cgroup/cpu/doris if [[ -f "/sys/fs/cgroup/cgroup.controllers" ]]; then chmod a+w /sys/fs/cgroup/cgroup.procs ``` and write `doris_cgroup_cpu_path = /sys/fs/cgroup/cpu/doris` into be.conf. ### Why On cgroup v2 there is no `cpu` controller mount. `/sys/fs/cgroup/cpu` is just an ordinary child cgroup named `cpu`, created by that `mkdir`. The `cpu` controller is never enabled in its parent's `cgroup.subtree_control`, so `cpu/` and `cpu/doris` have no controllers at all. `CgroupCpuCtl::init_cgroup_v2_query_path_public_file` (`be/src/agent/cgroup_cpu_ctl.cpp:129-144`) then writes `+cpu` into `<doris_cgroup_cpu_path>/cgroup.subtree_control`. The write fails because the controller is not available there. The failure only lands in the `[cgroup_init_path]` INFO line (`:111-123`, "write cgroup v2 file failed"), and the BE keeps running. No workload group gets a `cpu.max`. Placing the subtree at a fixed path under the cgroupfs root also misses the container's own cgroup. With a host cgroup namespace (or a runtime that exposes the host hierarchy), the subtree would sit outside the pod's `cpu.max`. ### What You Expected? On cgroup v2, the entrypoint sets up the workload group subtree under the container's own cgroup, with the `cpu` controller delegated, so that each workload group's `cpu.max` applies inside the pod's CPU limit. If that is impossible (read-only cgroupfs, `cpu` not delegated), the BE says so clearly instead of starting without isolation. ### How to Reproduce? On a cgroup v2 host (here: Amazon Linux 2023 arm64, kernel 6.18, Docker): ```bash docker run --rm --privileged --cgroupns=private --cpus=2 amazonlinux:2023 bash -c ' mkdir -p /sys/fs/cgroup/cpu/doris && chmod 770 /sys/fs/cgroup/cpu/doris && chmod a+w /sys/fs/cgroup/cgroup.procs; echo "entrypoint setup rc=$?" echo "controllers in cpu/: [$(cat /sys/fs/cgroup/cpu/cgroup.controllers)]" echo "controllers in cpu/doris: [$(cat /sys/fs/cgroup/cpu/doris/cgroup.controllers)]" echo +cpu > /sys/fs/cgroup/cpu/doris/cgroup.subtree_control || echo "the BE +cpu write fails"' ``` Output: ``` entrypoint setup rc=0 controllers in cpu/: [] controllers in cpu/doris: [] the BE +cpu write fails ``` With the official BE image: run it with `ENABLE_WORKLOAD_GROUP=true` on a cgroup v2 node, create a workload group with a CPU limit, and grep be.INFO for `[cgroup_init_path]`. It reports `write cgroup v2 file failed`, and no `cpu.max` exists under `/sys/fs/cgroup/cpu/doris`. ### Anything Else? Proposed fix (both entrypoints): 1. On cgroup v2, find the container's own cgroup from the `0::` line of `/proc/self/cgroup`. This works for both a private and a host cgroup namespace. 2. Check that it has `cpu.max`, that the entrypoint is in its `cgroup.procs`, and that `cpu` is in its `cgroup.controllers`. Log which check failed. 3. Create `<container cgroup>/doris` and make it `threaded` before writing `+cpu` into the container cgroup's `cgroup.subtree_control`. `cpu` is a threaded controller, so the kernel accepts this on a cgroup that already has processes. Then pre-create `doris/query` as `threaded`, the layout `CgroupCpuCtl` expects. Set `doris_cgroup_cpu_path` to `<container cgroup>/doris`. 4. If any of this fails, exit with a clear error rather than starting the BE without CPU isolation. 5. Keep the v1 path as it is, guarded by a check that `/sys/fs/cgroup/cpu/cpu.cfs_quota_us` exists. We tested this on the same host. In a privileged container with a private or a host cgroup namespace, setup succeeds. A threaded workload group under `doris/query` with `cpu.max 20000 100000` then throttles a busy thread to about 20% (1.0 s of CPU in 5 s). `docker exec` still works after setup. In an unprivileged container with a read-only cgroupfs, setup fails with a clear error. Separately, the BE logging the failed `+cpu` write at INFO hides the problem. A WARN there would have made this visible. We have a patch and can open a PR. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
