[
https://issues.apache.org/jira/browse/FLINK-21036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270123#comment-17270123
]
Chesnay Schepler commented on FLINK-21036:
------------------------------------------
[~trohrmann] Do I understand you correctly that agree with removing
{{TASK_MANAGER_NUMBER_OF_TASK_SLOTS}} but you want to introduce a new flag such
that, when enabled, this option is set the number of CPU cores?
If we want to keep this auto configuration then we should look into an
alternative that better respects the docker configuration. (And I'd only do
that if we figure this out)
There seem to be 2 different ways to configure CPU usage:
{{--cpu}}: Limits the CPU utilization to some fraction of the available CPU
cores.
{{--cpuset-cpus}}: Locks the container to specific CPU cores. I'd say this
would be too cumbersome to use for larger deployments though.
{{grep -c ^processor /proc/cpuinfo}} returns in both cases the total number of
CPU cores, which isn't what we want. {{nproc}} properly handles the second
setting; in the first setting it still returns all cores, which makes sense
since it may still use them after all.
I have yet to find a way to figure out from within the container what {{--cpu}}
was set to.
> Consider removing automatic configuration fo number of slots from docker
> ------------------------------------------------------------------------
>
> Key: FLINK-21036
> URL: https://issues.apache.org/jira/browse/FLINK-21036
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Scripts
> Reporter: Chesnay Schepler
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> The {{docker-entrypoint.sh}} supports setting the number of task slots via
> the {{TASK_MANAGER_NUMBER_OF_TASK_SLOTS}} environment variable, which
> defaults to the number of cpu cores via {{$(grep -c ^processor
> /proc/cpuinfo)}}.
> The environment variable itself is redundant nowadays since we introduced
> {{FLINK_PROPERTIES}}, and is no longer documented.
> Defaulting to the number of CPU cores can be considered convenience, but it
> seems odd to have this specific to docker while the distribution defaults to
> {{1}}.
> The bigger issue in my mind though is that this creates a configuration
> mismatch between the Job- and TaskManager processes; the ResourceManager
> specifically needs to know how many slots a worker has to make decisions
> about redundancy and allocating resources.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
