[
https://issues.apache.org/jira/browse/FLINK-41025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guotao Ding updated FLINK-41025:
--------------------------------
Description:
h3. Motivation
Kubernetes enables {{enableServiceLinks}} in the pod spec by default. With it
enabled, the kubelet injects several environment variables
({{<SVC>_SERVICE_HOST}}, {{<SVC>_SERVICE_PORT}}, {{<SVC>_PORT_*}}, ...) into
every container for each Service in the same namespace.
Every Flink native Kubernetes cluster creates its own rest and internal
Services, so the number of injected environment variables grows with the number
of Flink clusters in the namespace. In our production environment, a single
JobManager/TaskManager pod ended up with more than 20,000 environment
variables. In this case, submitting a Flink job on Kubernetes becomes very
slow: the JobManager and TaskManager pods take a long time to start, since
every shell invocation in the pod has to deal with the huge number of
environment variables.
We have disabled service links for all Flink pods in our production
environment. Since then, the number of environment variables in a pod has
dropped to less than 100, the startup time is back to normal, and no functional
impact on Flink has been observed.
Flink does not read these Service environment variables: components are reached
via Service DNS names and ConfigMaps. {{KUBERNETES_SERVICE_HOST}} and
{{KUBERNETES_SERVICE_PORT}}, which the Kubernetes client relies on, are
injected regardless of {{enableServiceLinks}}.
h3. Proposed Change
* Add a config option {{kubernetes.pod.service-links.enabled}} (Boolean, no
default value).
* If it is configured, {{InitJobManagerDecorator}} and
{{InitTaskManagerDecorator}} set {{enableServiceLinks}} of the JobManager and
TaskManager pod spec to the configured value, overwriting the value from the
pod template.
* If it is not configured, the field is left untouched, i.e. the value from the
pod template is kept, or the Kubernetes default ({{true}}) applies when the pod
template does not define it either. There is no behavior change for existing
deployments.
* Document the option and add {{enableServiceLinks}} to the "Fields Overwritten
by Flink" table of the native Kubernetes documentation.
h3. Alternatives
Setting {{enableServiceLinks: false}} in a pod template
({{kubernetes.pod-template-file.*}}) already works. A dedicated option is still
proposed because:
* The number of injected variables grows with the Services that Flink itself
creates, so the issue shows up specifically in namespaces with many Flink
clusters.
* The symptom (slow pod startup) does not point to the environment variables,
so the pod template workaround is hard to find. A documented option makes the
cause and the fix discoverable.
* Kubernetes is not going to change the default of {{enableServiceLinks}} since
it would be a breaking change (see
[kubernetes#121787|https://github.com/kubernetes/kubernetes/issues/121787]), so
it has to be disabled by the application that creates the pods.
* Other pod spec fields that can be set via pod template also have dedicated
options, e.g. {{kubernetes.hostnetwork.enabled}},
{{kubernetes.service-account}}, {{kubernetes.jobmanager.node-selector}} /
{{kubernetes.taskmanager.node-selector}}, {{kubernetes.jobmanager.tolerations}}
/ {{kubernetes.taskmanager.tolerations}}.
h3. How to reproduce
# Submit a large number of Flink jobs in native Kubernetes mode into one
namespace.
# Start a new Flink job in that namespace, and check {{env | wc -l}} and {{time
bash -c true}} inside its JobManager/TaskManager pod.
# Set {{enableServiceLinks: false}} via a pod template and repeat step 2.
h3. References
* [kubernetes#92615: 1000+ services in the same namespace will degrade pod
start time & eventually prevent pods from
starting|https://github.com/kubernetes/kubernetes/issues/92615]
* [kubernetes#121787: Set enableServiceLinks to false as
default|https://github.com/kubernetes/kubernetes/issues/121787]
* [太多的 service 信息环境变量可能会导致容器中执行 bash
命令特别的慢|https://mozillazg.com/2020/06/kubernetes-k8s-too-many-service-environment-variables-cause-pod-container-start-bash-too-slow.html]
* [任务执行龟速，原因竟然是......|https://cloud.tencent.com/developer/article/1916662]
was:
h3. Motivation
Kubernetes enables {{enableServiceLinks}} in the pod spec by default. With it
enabled, the kubelet injects several environment variables
({{<SVC>_SERVICE_HOST}}, {{<SVC>_SERVICE_PORT}}, {{<SVC>_PORT_*}}, ...) into
every container for each Service in the same namespace.
Every Flink native Kubernetes cluster creates its own rest and internal
Services, so the number of injected environment variables grows with the number
of Flink clusters in the namespace. In our production environment, a single
JobManager/TaskManager pod ended up with more than 20,000 environment
variables. In this case, submitting a Flink job on Kubernetes becomes very
slow: the JobManager and TaskManager pods take a long time to start, since
every shell invocation in the pod has to deal with the huge number of
environment variables.
We have disabled service links for all Flink pods in our production
environment. Since then, the number of environment variables in a pod has
dropped to less than 100, the startup time is back to normal, and no functional
impact on Flink has been observed.
Flink does not read these Service environment variables: components are reached
via Service DNS names and ConfigMaps. {{KUBERNETES_SERVICE_HOST}} and
{{KUBERNETES_SERVICE_PORT}}, which the Kubernetes client relies on, are
injected regardless of {{enableServiceLinks}}.
h3. Proposed Change
* Add a config option {{kubernetes.pod.service-links.enabled}} (Boolean, no
default value).
* If it is configured, {{InitJobManagerDecorator}} and
{{InitTaskManagerDecorator}} set {{enableServiceLinks}} of the JobManager and
TaskManager pod spec to the configured value, overwriting the value from the
pod template.
* If it is not configured, the field is left untouched, i.e. the value from the
pod template is kept, or the Kubernetes default ({{true}}) applies when the pod
template does not define it either. There is no behavior change for existing
deployments.
* Document the option and add {{enableServiceLinks}} to the "Fields Overwritten
by Flink" table of the native Kubernetes documentation.
h3. Alternatives
Setting {{enableServiceLinks: false}} in a pod template
({{kubernetes.pod-template-file.*}}) already works. A dedicated option is still
proposed because:
* The number of injected variables grows with the Services that Flink itself
creates, so the issue shows up specifically in namespaces with many Flink
clusters.
* The symptom (slow pod startup) does not point to the environment variables,
so the pod template workaround is hard to find. A documented option makes the
cause and the fix discoverable.
* Kubernetes is not going to change the default of {{enableServiceLinks}} since
it would be a breaking change (see
[kubernetes#121787|https://github.com/kubernetes/kubernetes/issues/121787]), so
it has to be disabled by the application that creates the pods.
* Other pod spec fields that can be set via pod template also have dedicated
options, e.g. {{kubernetes.hostnetwork.enabled}},
{{kubernetes.service-account}}, {{kubernetes.jobmanager.node-selector}} /
{{kubernetes.taskmanager.node-selector}}, {{kubernetes.jobmanager.tolerations}}
/ {{kubernetes.taskmanager.tolerations}}.
h3. How to reproduce
# Create a large number of dummy Services in a namespace.
# Start a pod in that namespace with and without {{enableServiceLinks: false}}.
# Compare {{env | wc -l}} and {{time bash -c true}} inside the pod.
h3. References
* [kubernetes#92615: 1000+ services in the same namespace will degrade pod
start time & eventually prevent pods from
starting|https://github.com/kubernetes/kubernetes/issues/92615]
* [kubernetes#121787: Set enableServiceLinks to false as
default|https://github.com/kubernetes/kubernetes/issues/121787]
* [太多的 service 信息环境变量可能会导致容器中执行 bash
命令特别的慢|https://mozillazg.com/2020/06/kubernetes-k8s-too-many-service-environment-variables-cause-pod-container-start-bash-too-slow.html]
* [任务执行龟速，原因竟然是......|https://cloud.tencent.com/developer/article/1916662]
> Add config option to control enableServiceLinks of JobManager and TaskManager
> pods
> ----------------------------------------------------------------------------------
>
> Key: FLINK-41025
> URL: https://issues.apache.org/jira/browse/FLINK-41025
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes
> Reporter: Guotao Ding
> Priority: Major
>
> h3. Motivation
> Kubernetes enables {{enableServiceLinks}} in the pod spec by default. With it
> enabled, the kubelet injects several environment variables
> ({{<SVC>_SERVICE_HOST}}, {{<SVC>_SERVICE_PORT}}, {{<SVC>_PORT_*}}, ...) into
> every container for each Service in the same namespace.
> Every Flink native Kubernetes cluster creates its own rest and internal
> Services, so the number of injected environment variables grows with the
> number of Flink clusters in the namespace. In our production environment, a
> single JobManager/TaskManager pod ended up with more than 20,000 environment
> variables. In this case, submitting a Flink job on Kubernetes becomes very
> slow: the JobManager and TaskManager pods take a long time to start, since
> every shell invocation in the pod has to deal with the huge number of
> environment variables.
> We have disabled service links for all Flink pods in our production
> environment. Since then, the number of environment variables in a pod has
> dropped to less than 100, the startup time is back to normal, and no
> functional impact on Flink has been observed.
> Flink does not read these Service environment variables: components are
> reached via Service DNS names and ConfigMaps. {{KUBERNETES_SERVICE_HOST}} and
> {{KUBERNETES_SERVICE_PORT}}, which the Kubernetes client relies on, are
> injected regardless of {{enableServiceLinks}}.
> h3. Proposed Change
> * Add a config option {{kubernetes.pod.service-links.enabled}} (Boolean, no
> default value).
> * If it is configured, {{InitJobManagerDecorator}} and
> {{InitTaskManagerDecorator}} set {{enableServiceLinks}} of the JobManager and
> TaskManager pod spec to the configured value, overwriting the value from the
> pod template.
> * If it is not configured, the field is left untouched, i.e. the value from
> the pod template is kept, or the Kubernetes default ({{true}}) applies when
> the pod template does not define it either. There is no behavior change for
> existing deployments.
> * Document the option and add {{enableServiceLinks}} to the "Fields
> Overwritten by Flink" table of the native Kubernetes documentation.
> h3. Alternatives
> Setting {{enableServiceLinks: false}} in a pod template
> ({{kubernetes.pod-template-file.*}}) already works. A dedicated option is
> still proposed because:
> * The number of injected variables grows with the Services that Flink itself
> creates, so the issue shows up specifically in namespaces with many Flink
> clusters.
> * The symptom (slow pod startup) does not point to the environment variables,
> so the pod template workaround is hard to find. A documented option makes the
> cause and the fix discoverable.
> * Kubernetes is not going to change the default of {{enableServiceLinks}}
> since it would be a breaking change (see
> [kubernetes#121787|https://github.com/kubernetes/kubernetes/issues/121787]),
> so it has to be disabled by the application that creates the pods.
> * Other pod spec fields that can be set via pod template also have dedicated
> options, e.g. {{kubernetes.hostnetwork.enabled}},
> {{kubernetes.service-account}}, {{kubernetes.jobmanager.node-selector}} /
> {{kubernetes.taskmanager.node-selector}},
> {{kubernetes.jobmanager.tolerations}} /
> {{kubernetes.taskmanager.tolerations}}.
> h3. How to reproduce
> # Submit a large number of Flink jobs in native Kubernetes mode into one
> namespace.
> # Start a new Flink job in that namespace, and check {{env | wc -l}} and
> {{time bash -c true}} inside its JobManager/TaskManager pod.
> # Set {{enableServiceLinks: false}} via a pod template and repeat step 2.
> h3. References
> * [kubernetes#92615: 1000+ services in the same namespace will degrade pod
> start time & eventually prevent pods from
> starting|https://github.com/kubernetes/kubernetes/issues/92615]
> * [kubernetes#121787: Set enableServiceLinks to false as
> default|https://github.com/kubernetes/kubernetes/issues/121787]
> * [太多的 service 信息环境变量可能会导致容器中执行 bash
> 命令特别的慢|https://mozillazg.com/2020/06/kubernetes-k8s-too-many-service-environment-variables-cause-pod-container-start-bash-too-slow.html]
> * [任务执行龟速，原因竟然是......|https://cloud.tencent.com/developer/article/1916662]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
