[
https://issues.apache.org/jira/browse/FLINK-41052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gerk Elznik updated FLINK-41052:
--------------------------------
Description:
h2. Problem
The chart renders the operator configuration into the {{flink-operator-config}}
ConfigMap, but the pod template does not change when that ConfigMap does, so a
configuration-only change applied through Helm, Argo CD or any other tool that
renders the chart never restarts the operator. The dynamic config watcher
covers the reconcile-time options, but the options documented under the [System
Configuration
Startup|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#system-startup]
and [Advanced System Configuration
Startup|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#advanced-startup]
sections, plus the [metrics
reporters|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#system-metrics-configuration]
and the logging config (unless {{monitorInterval}} is enabled), {*}keep the
value read at startup until someone restarts the pod by hand{*}. The [Dynamic
Operator
Configuration|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#dynamic-operator-configuration]
section already says so, so the gap is known; the chart just offers no way to
close it.
Measured current behavior in a kind cluster on main (chart 1.17-SNAPSHOT,
webhook enabled): a {{helm upgrade}} that adds
{{kubernetes.operator.leader-election.enabled: true}} updates the ConfigMap,
the pod is not recreated, and the operator keeps logging {{{}Operator leader
election is disabled{}}}.
The usual Helm answer is a checksum of the rendered ConfigMap on the pod
template, so the Deployment rolls exactly when the configuration changes,
whichever tool applies the chart. Many charts hardcode that.
Here my current thinking is to *make it opt-in* because the dynamic config
feature is still there to avoid restarts and a recreation under the default
{{Recreate}} strategy is a short outage during which the admission webhook
({{{}failurePolicy: Fail{}}}) rejects Flink resource changes, and a
configuration the operator cannot load would crash-loop the new pod, whereas
today the watcher logs the error and keeps the old configuration, so we'll want
users to be well aware of what they're signing up for when they opt-in.
h2. Proposal
# A new value {{operatorPod.restartOnConfigChange}} (default {{{}false{}}})
that adds {{checksum/config: <sha256 of the rendered
controller/configmap.yaml>}} to the pod template annotations, only when the
chart manages that ConfigMap ({{{}defaultConfiguration.create{}}}). Direct
edits of the live ConfigMap are not tracked and keep going through the dynamic
reload. Default off keeps existing installs byte-identical. If you would rather
have the conventional always-on behavior, that is a one-line change and I am
happy to make it. One property worth knowing: the ConfigMap render carries the
chart labels, so a chart version bump also rolls the pod once; that usually
coincides with an image change anyway, and hashing only the {{data}} block
would avoid it if you prefer.
# As a smaller companion, a new value {{operatorDeployment.annotations}}
rendered on the Deployment object's own metadata. {{operatorPod.labels}}
already lands on both the pod template and the Deployment (FLINK-35402), but
annotations only reach the pod template. Deployment-level annotations are what
controllers such as Stakater Reloader or Argo CD read, and they are also where
ownership and alert-routing metadata usually lives. It is also the complement
of the first item for people who bring their own ConfigMap: the chart cannot
checksum what it does not render, so they need an external trigger, and
Reloader reads its annotation from the Deployment. For clarity, FLINK-29391 and
FLINK-37421 ask for labels on the JobManager and TaskManager Deployments that
the operator creates; this ticket is only about the operator's own Deployment
from the chart.
Both parts are additive and change nothing with default values.
h2. Verification so far (prototype, not yet pushed)
helm-unittest (71 pass on Helm 4.3, also with the 0.8.1 plugin version CI
pins), ct lint (Helm 3.16 image, which also renders the toggle with the same
checksum as Helm 4), and a kind cluster with the webhook enabled through
cert-manager, operator image built from the branch:
* toggle off, helm adds leader election to the config: ConfigMap updated, pod
uid unchanged, operator still reports leader election disabled
* toggle on, same values: pod recreated, {{{}Operator leader election is
enabled{}}}, checksum present
* toggle on, config changed again: pod recreated, checksum changed
* toggle on, identical values re-applied: no pod churn, checksum unchanged
* toggle on with {{{}defaultConfiguration.create: false{}}}: no checksum
annotation rendered
* {{operatorDeployment.annotations}} appears on the Deployment and not on the
pod template
* two running FlinkDeployments (v2_2) stayed RUNNING across the operator
recreation, JobManager pods untouched, no operator errors after the restart
h2. Docs
Rows in the Helm installation table for both values (EN and ZH) and a short
note in the [Dynamic Operator
Configuration|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#dynamic-operator-configuration]
section: what the toggle tracks, that live edits and bring-your-own ConfigMaps
are not covered, and the outage trade-off under {{{}Recreate{}}}.
was:
h2. Problem
The chart renders the operator configuration into the {{flink-operator-config}}
ConfigMap, but the pod template does not change when that ConfigMap does, so a
configuration-only change applied through Helm, Argo CD or any other tool that
renders the chart never restarts the operator. The dynamic config watcher
covers the reconcile-time options, but the options documented under the [System
Configuration
Startup|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#system-startup]
and [Advanced System Configuration
Startup|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#advanced-startup]
sections, plus the [metrics
reporters|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#system-metrics-configuration]
and the logging config (unless {{monitorInterval}} is enabled), {*}keep the
value read at startup until someone restarts the pod by hand{*}. The [Dynamic
Operator
Configuration|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#dynamic-operator-configuration]
section already says so, so the gap is known; the chart just offers no way to
close it.
Measured current behavior in a kind cluster on main (chart 1.17-SNAPSHOT,
webhook enabled): a {{helm upgrade}} that adds
{{kubernetes.operator.leader-election.enabled: true}} updates the ConfigMap,
the pod is not recreated, and the operator keeps logging {{{}Operator leader
election is disabled{}}}.
The usual Helm answer is a checksum of the rendered ConfigMap on the pod
template, so the Deployment rolls exactly when the configuration changes,
whichever tool applies the chart. Many charts hardcode that.
Here my current thinking is to *make it opt-in* because the dynamic config
feature is still there to avoid restarts and a recreation under the default
{{Recreate}} strategy is a short outage during which the admission webhook
({{{}failurePolicy: Fail{}}}) rejects Flink resource changes, and a
configuration the operator cannot load would crash-loop the new pod, whereas
today the watcher logs the error and keeps the old configuration, so we'll want
users to be well aware of what they're signing up for when they opt-in.
h2. Proposal
# A new value {{operatorPod.restartOnConfigChange}} (default {{{}false{}}})
that adds {{checksum/config: <sha256 of the rendered
controller/configmap.yaml>}} to the pod template annotations, only when the
chart manages that ConfigMap ({{{}defaultConfiguration.create{}}}). Direct
edits of the live ConfigMap are not tracked and keep going through the dynamic
reload. Default off keeps existing installs byte-identical. If you would rather
have the conventional always-on behavior, that is a one-line change and I am
happy to make it. One property worth knowing: the ConfigMap render carries the
chart labels, so a chart version bump also rolls the pod once; that usually
coincides with an image change anyway, and hashing only the {{data}} block
would avoid it if you prefer.
# As a smaller companion, a new value {{operatorDeployment.annotations}}
rendered on the Deployment object's own metadata. {{operatorPod.labels}}
already lands on both the pod template and the Deployment (FLINK-35402), but
annotations only reach the pod template. Deployment-level annotations are what
controllers such as Stakater Reloader or Argo CD read, and they are also where
ownership and alert-routing metadata usually lives. It is also the complement
of the first item for people who bring their own ConfigMap: the chart cannot
checksum what it does not render, so they need an external trigger, and
Reloader reads its annotation from the Deployment. For clarity, FLINK-29391 and
FLINK-37421 ask for labels on the JobManager and TaskManager Deployments that
the operator creates; this ticket is only about the operator's own Deployment
from the chart.
Both parts are additive and change nothing with default values.
h2. Verification so far (prototype, not yet pushed)
helm-unittest (71 pass on Helm 4.3, also with the 0.8.1 plugin version CI
pins), ct lint (Helm 3.16 image, which also renders the toggle with the same
checksum as Helm 4), and a kind cluster with the webhook enabled through
cert-manager, operator image built from the branch:
* toggle off, helm adds leader election to the config: ConfigMap updated, pod
uid unchanged, operator still reports leader election disabled
* toggle on, same values: pod recreated, {{{}Operator leader election is
enabled{}}}, checksum present
* toggle on, config changed again: pod recreated, checksum changed
* toggle on, identical values re-applied: no pod churn, checksum unchanged
* toggle on with {{{}defaultConfiguration.create: false{}}}: no checksum
annotation rendered
* {{operatorDeployment.annotations}} appears on the Deployment and not on the
pod template
* two running FlinkDeployments (v2_2) stayed RUNNING across the operator
recreation, JobManager pods untouched, no operator errors after the restart
h2. Docs
Rows in the Helm installation table for both values (EN and ZH) and a short
note in the [Dynamic Operator
Configuration|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#dynamic-operator-configuration]
section: what the toggle tracks, that live edits and bring-your-own ConfigMaps
are not covered, and the outage trade-off under {{{}Recreate{}}}.
#
> Helm chart: recreate the operator pod on ConfigMap changes and support
> annotations on the operator Deployment
> -------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-41052
> URL: https://issues.apache.org/jira/browse/FLINK-41052
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
> Reporter: Gerk Elznik
> Priority: Minor
> Labels: helm, operator
>
> h2. Problem
> The chart renders the operator configuration into the
> {{flink-operator-config}} ConfigMap, but the pod template does not change
> when that ConfigMap does, so a configuration-only change applied through
> Helm, Argo CD or any other tool that renders the chart never restarts the
> operator. The dynamic config watcher covers the reconcile-time options, but
> the options documented under the [System Configuration
> Startup|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#system-startup]
> and [Advanced System Configuration
> Startup|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#advanced-startup]
> sections, plus the [metrics
> reporters|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#system-metrics-configuration]
> and the logging config (unless {{monitorInterval}} is enabled), {*}keep the
> value read at startup until someone restarts the pod by hand{*}. The [Dynamic
> Operator
> Configuration|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#dynamic-operator-configuration]
> section already says so, so the gap is known; the chart just offers no way
> to close it.
> Measured current behavior in a kind cluster on main (chart 1.17-SNAPSHOT,
> webhook enabled): a {{helm upgrade}} that adds
> {{kubernetes.operator.leader-election.enabled: true}} updates the ConfigMap,
> the pod is not recreated, and the operator keeps logging {{{}Operator leader
> election is disabled{}}}.
> The usual Helm answer is a checksum of the rendered ConfigMap on the pod
> template, so the Deployment rolls exactly when the configuration changes,
> whichever tool applies the chart. Many charts hardcode that.
> Here my current thinking is to *make it opt-in* because the dynamic config
> feature is still there to avoid restarts and a recreation under the default
> {{Recreate}} strategy is a short outage during which the admission webhook
> ({{{}failurePolicy: Fail{}}}) rejects Flink resource changes, and a
> configuration the operator cannot load would crash-loop the new pod, whereas
> today the watcher logs the error and keeps the old configuration, so we'll
> want users to be well aware of what they're signing up for when they opt-in.
> h2. Proposal
> # A new value {{operatorPod.restartOnConfigChange}} (default {{{}false{}}})
> that adds {{checksum/config: <sha256 of the rendered
> controller/configmap.yaml>}} to the pod template annotations, only when the
> chart manages that ConfigMap ({{{}defaultConfiguration.create{}}}). Direct
> edits of the live ConfigMap are not tracked and keep going through the
> dynamic reload. Default off keeps existing installs byte-identical. If you
> would rather have the conventional always-on behavior, that is a one-line
> change and I am happy to make it. One property worth knowing: the ConfigMap
> render carries the chart labels, so a chart version bump also rolls the pod
> once; that usually coincides with an image change anyway, and hashing only
> the {{data}} block would avoid it if you prefer.
> # As a smaller companion, a new value {{operatorDeployment.annotations}}
> rendered on the Deployment object's own metadata. {{operatorPod.labels}}
> already lands on both the pod template and the Deployment (FLINK-35402), but
> annotations only reach the pod template. Deployment-level annotations are
> what controllers such as Stakater Reloader or Argo CD read, and they are also
> where ownership and alert-routing metadata usually lives. It is also the
> complement of the first item for people who bring their own ConfigMap: the
> chart cannot checksum what it does not render, so they need an external
> trigger, and Reloader reads its annotation from the Deployment. For clarity,
> FLINK-29391 and FLINK-37421 ask for labels on the JobManager and TaskManager
> Deployments that the operator creates; this ticket is only about the
> operator's own Deployment from the chart.
> Both parts are additive and change nothing with default values.
> h2. Verification so far (prototype, not yet pushed)
> helm-unittest (71 pass on Helm 4.3, also with the 0.8.1 plugin version CI
> pins), ct lint (Helm 3.16 image, which also renders the toggle with the same
> checksum as Helm 4), and a kind cluster with the webhook enabled through
> cert-manager, operator image built from the branch:
> * toggle off, helm adds leader election to the config: ConfigMap updated,
> pod uid unchanged, operator still reports leader election disabled
> * toggle on, same values: pod recreated, {{{}Operator leader election is
> enabled{}}}, checksum present
> * toggle on, config changed again: pod recreated, checksum changed
> * toggle on, identical values re-applied: no pod churn, checksum unchanged
> * toggle on with {{{}defaultConfiguration.create: false{}}}: no checksum
> annotation rendered
> * {{operatorDeployment.annotations}} appears on the Deployment and not on
> the pod template
> * two running FlinkDeployments (v2_2) stayed RUNNING across the operator
> recreation, JobManager pods untouched, no operator errors after the restart
> h2. Docs
> Rows in the Helm installation table for both values (EN and ZH) and a short
> note in the [Dynamic Operator
> Configuration|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#dynamic-operator-configuration]
> section: what the toggle tracks, that live edits and bring-your-own
> ConfigMaps are not covered, and the outage trade-off under {{{}Recreate{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
