eskabetxe opened a new pull request, #1220: URL: https://github.com/apache/flink-kubernetes-operator/pull/1220
## What is the purpose of the change Fixes [FLINK-40944](https://issues.apache.org/jira/browse/FLINK-40944). `kubernetes.{jobmanager,taskmanager}.cpu.limit-factor` raises the container CPU limit above the request. The container-aware JVM derives `Runtime.availableProcessors()` from the cgroup CPU quota, i.e. from the inflated **limit**, while the memory model (including `MaxDirectMemorySize`) stays **request**-sized. Thread pools and direct-buffer allocators sized from the processor count (e.g. the Netty pooled allocator arenas, Beam portability pools in PyFlink) then grow with the limit factor against an unchanged direct-memory ceiling, producing `OutOfMemoryError: Direct buffer memory` in production. This PR pins `-XX:ActiveProcessorCount` to the CPU request whenever the effective CPU limit factor is > 1, so the JVM stays sized to the request regardless of the inflated cgroup quota (precedent: Spark's SPARK-31028). It is a no-op when limit == request. ## Brief change log - `FlinkConfigBuilder`: new `applyActiveProcessorCountForCpuLimitFactor()` step wired into `buildFrom()`. When the effective `kubernetes.{jobmanager,taskmanager}.cpu.limit-factor` is > 1 (covers both the new `spec.*.resources` field and the raw-config path), it appends `-XX:ActiveProcessorCount=<max(1, ceil(cpu request))>` to `env.java.opts.{jobmanager,taskmanager}` in the effective config. - The TaskManager CPU request is resolved the same way Flink resolves the container CPU (`TaskExecutorProcessUtils.getCpuCoresWithFallback`): `taskmanager.cpu-cores` first, else `kubernetes.taskmanager.cpu`, falling back to `taskmanager.numberOfTaskSlots` when unset (-1). - An explicit user-set `-XX:ActiveProcessorCount` always wins: detected in `env.java.opts` / `env.java.opts.all` / `env.java.opts.{jobmanager,taskmanager}` / `env.java.default-opts.*` and in the `FLINK_ENV_JAVA_OPTS`, `FLINK_ENV_JAVA_OPTS_JM`/`_TM` and `JVM_ARGS` pod-template env vars (which `config.sh` prefers over config-map values). This also makes the step idempotent. - Non-finite / non-positive / unrepresentable CPU values skip the injection with a WARN instead of emitting a bogus flag. - Docs: new "CPU Limit Factor and the JVM Active Processor Count" section in `docs/content/docs/deployment/configuration.md`. ## Verifying this change This change added tests and can be verified as follows: - 12 new unit tests in `FlinkConfigBuilderTest`: injection via the new `resources` field (incl. ceil rounding both ways) and via raw config through the full `buildFrom()` chain; no-op at the factor == 1 / < 1 / malformed boundaries; user override wins per component (incl. legacy `env.java.opts`, `env.java.opts.all`, `env.java.default-opts.*`, pod-template env vars); TM/JM independence; slot-count fallback when `kubernetes.taskmanager.cpu` is unset; `taskmanager.cpu-cores` precedence; idempotency; whitespace-only existing opts. - Red-green: removing the `buildFrom()` wiring fails the wiring test with `expected: <-XX:ActiveProcessorCount=10> but was: <>`. - Full `flink-kubernetes-operator` module suite passes (2279 tests), `mvn spotless:check` and `checkstyle:check` clean. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changes to the `CustomResourceDescriptors`: no - Core observer or reconciler logic that is regularly executed: yes (the effective config now carries the injected `-XX:ActiveProcessorCount` JVM arg, but only when a CPU limit factor > 1 is configured; config generation itself runs on cache miss, not per reconcile) ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? docs (new section in `deployment/configuration.md`) --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (opencode 1.18.34, plus model-assisted review passes; all changes manually reviewed and verified by the author) Generated-by: opencode 1.18.34 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
