[
https://issues.apache.org/jira/browse/FLINK-40944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
João Boto updated FLINK-40944:
------------------------------
Description:
h3. Description
CPU limit-factor inflates the JVM's availableProcessors() to the CPU limit,
risking OutOfMemoryError: Direct buffer memory — operator should pin
-XX:ActiveProcessorCount to the CPU request
h3. Problem
* kubernetes.\{jobmanager,taskmanager}.cpu.limit-factor (FLINK-15648) raises
the container CPU limit above the request (e.g. request 10, factor 4 → limit
40) to provide burst headroom. Nothing else in the memory model accounts for
this:
The container-aware JVM derives Runtime.availableProcessors() from the cgroup
CPU quota, which reflects the limit, not the request — so the JVM sees 40 cores
instead of 10.
* Thread pools sized off that count scale up accordingly — e.g. Netty's pooled
direct-memory allocator (PooledByteBufAllocator arenas/threads in Flink's
network stack), Beam portability/bundle-factory pools in PyFlink jobs.
* But MaxDirectMemorySize is computed from the fixed, request-sized
taskmanager.memory.* model and never changes with the CPU factor.
Net effect: ~factor× more thread-driven direct-buffer demand against an
unchanged direct-memory ceiling → java.lang.OutOfMemoryError: Direct buffer
memory.
h3. Real-world impact
Production TaskManager with cpu: 10, cpu.limit-factor: 4 (limit 40),
taskmanager.memory.process.size: 20G repeatedly failed with OutOfMemoryError:
Direct buffer memory inside a Kafka source fetcher thread, right at the JVM's
~1.6 GiB direct-memory ceiling. Reproducible and directly tied to the inflated
processor count. Neither FLINK-15648, the operator's ResourceRequirements
rework, nor the fractional-CPU discussions on the user list mention this
interaction.
h3. Proposed solution
When the effective CPU limit-factor > 1, the operator should append
-XX:ActiveProcessorCount=<max(1, ceil(cpu request))> to
env.java.opts.jobmanager/env.java.opts.taskmanager in the effective
configuration (config map), so the JVM stays sized to the request regardless of
the inflated cgroup quota. This is a no-op when limit == request, and covers
both JM and TM in native and standalone modes since JVM opts flow from the
shared config. An explicit -XX:ActiveProcessorCount already present in the
user's env.java.opts.* must win (skip injection).
FlinkConfigBuilder is the natural place: it already computes the effective
limit factor (setResourceRequirements) and reads the deprecated-path values
from kubernetes.*.cpu.limit-factor (same options
FlinkUtils.calculateClusterCpuUsage consumes), so a single post-build step
covers both the new resources field and the raw-config path.
Precedent: Flink core already injects JVM options into env.java.opts for k8s
(KubernetesUtils.addGCLogsToJavaOpts for GC logging); Spark resolved the
identical issue by setting ActiveProcessorCount on driver/executor JVMs
(SPARK-31028).
h4. Alternatives considered:
* Fix in flink-kubernetes core where the limit-factor is applied to the
container spec — broader (covers non-operator deployments too) but the JVM
command is assembled at entrypoint/process-launch time, making config-side
injection on the operator the cleaner first step.
* Documentation-only recommendation. Weaker: the failure mode is a silent prod
OOM that appears only when the feature is used as designed.
h4. Workaround today (what we run)
Users can render -XX:ActiveProcessorCount=<ceil(cpu request)> via
env.java.opts.\{jobmanager,taskmanager} themselves whenever a limit-factor is
set — which is what our platform wrapper does; but this shouldn't be required
to use a first-class operator/feature safely.
was:
h3. Description
CPU limit-factor inflates the JVM's availableProcessors() to the CPU limit,
risking OutOfMemoryError: Direct buffer memory — operator should pin
-XX:ActiveProcessorCount to the CPU request
h3.
Problem
* kubernetes.\{jobmanager,taskmanager}.cpu.limit-factor (FLINK-15648) raises
the container CPU limit above the request (e.g. request 10, factor 4 → limit
40) to provide burst headroom. Nothing else in the memory model accounts for
this:
The container-aware JVM derives Runtime.availableProcessors() from the cgroup
CPU quota, which reflects the limit, not the request — so the JVM sees 40 cores
instead of 10.
* Thread pools sized off that count scale up accordingly — e.g. Netty's pooled
direct-memory allocator (PooledByteBufAllocator arenas/threads in Flink's
network stack), Beam portability/bundle-factory pools in PyFlink jobs.
* But MaxDirectMemorySize is computed from the fixed, request-sized
taskmanager.memory.* model and never changes with the CPU factor.
Net effect: ~factor× more thread-driven direct-buffer demand against an
unchanged direct-memory ceiling → java.lang.OutOfMemoryError: Direct buffer
memory.
h3.
Real-world impact
Production TaskManager with cpu: 10, cpu.limit-factor: 4 (limit 40),
taskmanager.memory.process.size: 20G repeatedly failed with OutOfMemoryError:
Direct buffer memory inside a Kafka source fetcher thread, right at the JVM's
~1.6 GiB direct-memory ceiling. Reproducible and directly tied to the inflated
processor count. Neither FLINK-15648, the operator's ResourceRequirements
rework, nor the fractional-CPU discussions on the user list mention this
interaction.
h3. Proposed solution
When the effective CPU limit-factor > 1, the operator should append
-XX:ActiveProcessorCount=<max(1, ceil(cpu request))> to
env.java.opts.jobmanager/env.java.opts.taskmanager in the effective
configuration (config map), so the JVM stays sized to the request regardless of
the inflated cgroup quota. This is a no-op when limit == request, and covers
both JM and TM in native and standalone modes since JVM opts flow from the
shared config. An explicit -XX:ActiveProcessorCount already present in the
user's env.java.opts.* must win (skip injection).
FlinkConfigBuilder is the natural place: it already computes the effective
limit factor (setResourceRequirements) and reads the deprecated-path values
from kubernetes.*.cpu.limit-factor (same options
FlinkUtils.calculateClusterCpuUsage consumes), so a single post-build step
covers both the new resources field and the raw-config path.
Precedent: Flink core already injects JVM options into env.java.opts for k8s
(KubernetesUtils.addGCLogsToJavaOpts for GC logging); Spark resolved the
identical issue by setting ActiveProcessorCount on driver/executor JVMs
(SPARK-31028).
Alternatives considered:
* Fix in flink-kubernetes core where the limit-factor is applied to the
container spec — broader (covers non-operator deployments too) but the JVM
command is assembled at entrypoint/process-launch time, making config-side
injection on the operator the cleaner first step.
* Documentation-only recommendation. Weaker: the failure mode is a silent prod
OOM that appears only when the feature is used as designed.
h3.
Workaround today (what we run)
Users can render -XX:ActiveProcessorCount=<ceil(cpu request)> via
env.java.opts.\{jobmanager,taskmanager} themselves whenever a limit-factor is
set — which is what our platform wrapper does; but this shouldn't be required
to use a first-class operator/feature safely.
> CPU limit-factor inflates the JVM's availableProcessors() to the CPU limit
> ---------------------------------------------------------------------------
>
> Key: FLINK-40944
> URL: https://issues.apache.org/jira/browse/FLINK-40944
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Reporter: João Boto
> Priority: Major
> Labels: pull-request-available
>
> h3. Description
> CPU limit-factor inflates the JVM's availableProcessors() to the CPU limit,
> risking OutOfMemoryError: Direct buffer memory — operator should pin
> -XX:ActiveProcessorCount to the CPU request
> h3. Problem
> * kubernetes.\{jobmanager,taskmanager}.cpu.limit-factor (FLINK-15648) raises
> the container CPU limit above the request (e.g. request 10, factor 4 → limit
> 40) to provide burst headroom. Nothing else in the memory model accounts for
> this:
> The container-aware JVM derives Runtime.availableProcessors() from the cgroup
> CPU quota, which reflects the limit, not the request — so the JVM sees 40
> cores instead of 10.
> * Thread pools sized off that count scale up accordingly — e.g. Netty's
> pooled direct-memory allocator (PooledByteBufAllocator arenas/threads in
> Flink's network stack), Beam portability/bundle-factory pools in PyFlink jobs.
> * But MaxDirectMemorySize is computed from the fixed, request-sized
> taskmanager.memory.* model and never changes with the CPU factor.
> Net effect: ~factor× more thread-driven direct-buffer demand against an
> unchanged direct-memory ceiling → java.lang.OutOfMemoryError: Direct buffer
> memory.
> h3. Real-world impact
> Production TaskManager with cpu: 10, cpu.limit-factor: 4 (limit 40),
> taskmanager.memory.process.size: 20G repeatedly failed with OutOfMemoryError:
> Direct buffer memory inside a Kafka source fetcher thread, right at the JVM's
> ~1.6 GiB direct-memory ceiling. Reproducible and directly tied to the
> inflated processor count. Neither FLINK-15648, the operator's
> ResourceRequirements rework, nor the fractional-CPU discussions on the user
> list mention this interaction.
> h3. Proposed solution
> When the effective CPU limit-factor > 1, the operator should append
> -XX:ActiveProcessorCount=<max(1, ceil(cpu request))> to
> env.java.opts.jobmanager/env.java.opts.taskmanager in the effective
> configuration (config map), so the JVM stays sized to the request regardless
> of the inflated cgroup quota. This is a no-op when limit == request, and
> covers both JM and TM in native and standalone modes since JVM opts flow from
> the shared config. An explicit -XX:ActiveProcessorCount already present in
> the user's env.java.opts.* must win (skip injection).
> FlinkConfigBuilder is the natural place: it already computes the effective
> limit factor (setResourceRequirements) and reads the deprecated-path values
> from kubernetes.*.cpu.limit-factor (same options
> FlinkUtils.calculateClusterCpuUsage consumes), so a single post-build step
> covers both the new resources field and the raw-config path.
> Precedent: Flink core already injects JVM options into env.java.opts for k8s
> (KubernetesUtils.addGCLogsToJavaOpts for GC logging); Spark resolved the
> identical issue by setting ActiveProcessorCount on driver/executor JVMs
> (SPARK-31028).
> h4. Alternatives considered:
> * Fix in flink-kubernetes core where the limit-factor is applied to the
> container spec — broader (covers non-operator deployments too) but the JVM
> command is assembled at entrypoint/process-launch time, making config-side
> injection on the operator the cleaner first step.
> * Documentation-only recommendation. Weaker: the failure mode is a silent
> prod OOM that appears only when the feature is used as designed.
> h4. Workaround today (what we run)
> Users can render -XX:ActiveProcessorCount=<ceil(cpu request)> via
> env.java.opts.\{jobmanager,taskmanager} themselves whenever a limit-factor is
> set — which is what our platform wrapper does; but this shouldn't be required
> to use a first-class operator/feature safely.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
