This is an automated email from the ASF dual-hosted git repository.
gyfora pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new a8407c47 [FLINK-39791][helm] Honor defaultConfiguration.config.yaml
and always mount config.yaml (#1126)
a8407c47 is described below
commit a8407c47a78c29f15a7dd9cc399b93b83d944223
Author: Gerk Elznik <[email protected]>
AuthorDate: Tue Aug 11 04:10:07 2026 -0600
[FLINK-39791][helm] Honor defaultConfiguration.config.yaml and always mount
config.yaml (#1126)
---
.github/workflows/e2e.yaml | 4 +-
docs/content.zh/docs/deployment/compatibility.md | 2 +-
docs/content.zh/docs/deployment/configuration.md | 24 +++++++--
.../docs/deployment/helm/installation.md | 4 +-
docs/content.zh/docs/internals/startup.md | 8 +--
docs/content/docs/deployment/compatibility.md | 2 +-
docs/content/docs/deployment/configuration.md | 24 +++++++--
docs/content/docs/deployment/helm/installation.md | 4 +-
docs/content/docs/internals/startup.md | 8 +--
e2e-tests/test_dynamic_flink_conf.sh | 6 ++-
.../operator/config/FlinkConfigManagerTest.java | 39 ++++++++++++++
.../conf/{flink-conf.yaml => config.yaml} | 4 --
.../templates/controller/configmap.yaml | 24 +++------
.../templates/controller/deployment.yaml | 5 --
.../tests/controller/configmap_test.yaml | 61 ++++++++++++++++++++++
.../tests/controller/deployment_test.yaml | 13 +++++
helm/flink-kubernetes-operator/values.yaml | 11 +++-
17 files changed, 192 insertions(+), 51 deletions(-)
diff --git a/.github/workflows/e2e.yaml b/.github/workflows/e2e.yaml
index 197ea985..5d01ceab 100644
--- a/.github/workflows/e2e.yaml
+++ b/.github/workflows/e2e.yaml
@@ -84,8 +84,8 @@ jobs:
- name: Start the operator
run: |
if [[ "${{ inputs.test }}" == "test_flink_operator_ha.sh" ]]; then
- sed -i "s/# kubernetes.operator.leader-election.enabled:
false/kubernetes.operator.leader-election.enabled: true/"
helm/flink-kubernetes-operator/conf/flink-conf.yaml
- sed -i "s/# kubernetes.operator.leader-election.lease-name:
flink-operator-lease/kubernetes.operator.leader-election.lease-name:
flink-operator-lease/" helm/flink-kubernetes-operator/conf/flink-conf.yaml
+ sed -i "s/# kubernetes.operator.leader-election.enabled:
false/kubernetes.operator.leader-election.enabled: true/"
helm/flink-kubernetes-operator/conf/config.yaml
+ sed -i "s/# kubernetes.operator.leader-election.lease-name:
flink-operator-lease/kubernetes.operator.leader-election.lease-name:
flink-operator-lease/" helm/flink-kubernetes-operator/conf/config.yaml
sed -i "s/replicas: 1/replicas: 2/"
helm/flink-kubernetes-operator/values.yaml
fi
if [[ "${{ inputs.test }}" == "test_dynamic_config.sh" ]]; then
diff --git a/docs/content.zh/docs/deployment/compatibility.md
b/docs/content.zh/docs/deployment/compatibility.md
index fe627034..fce7d41d 100644
--- a/docs/content.zh/docs/deployment/compatibility.md
+++ b/docs/content.zh/docs/deployment/compatibility.md
@@ -97,7 +97,7 @@ The operator JVM links against a specific Flink release for
its own runtime need
This bundled runtime is used by the operator process for purposes such as:
- Pluggable filesystem access (`org.apache.flink.core.fs.FileSystem`), used to
fetch `FlinkSessionJob` artifacts from non-HTTP schemes such as `s3://`,
`gs://`, `hdfs://`.
-- Configuration parsing (`org.apache.flink.configuration.GlobalConfiguration`)
that reads the operator's own `flink-conf.yaml`.
+- Configuration parsing (`org.apache.flink.configuration.GlobalConfiguration`)
that reads the operator's own `config.yaml`.
- Plugin discovery (`org.apache.flink.core.plugin.PluginManager`) that loads
custom validators, mutators, listeners, and filesystem factories.
## Kubernetes
diff --git a/docs/content.zh/docs/deployment/configuration.md
b/docs/content.zh/docs/deployment/configuration.md
index 6d08bc9b..3c66a0c8 100644
--- a/docs/content.zh/docs/deployment/configuration.md
+++ b/docs/content.zh/docs/deployment/configuration.md
@@ -51,10 +51,28 @@ defaultConfiguration:
### YAML Configuration File Format
-Operator configuration is delivered through the
`defaultConfiguration.flink-conf.yaml` block in the Helm `values.yaml`. The
operator's runtime loads that file with Flink's legacy configuration parser,
which accepts only flat key-value pairs written with dot notation and does not
support nested YAML maps.
+Operator configuration can be delivered in either of Flink's two configuration
formats, by defining the matching key under `defaultConfiguration` in the Helm
`values.yaml`:
+
+- `flink-conf.yaml` is the legacy format, which accepts only flat key-value
pairs written with dot notation and does not support nested YAML maps. This is
the format the chart ships by default.
+- `config.yaml` is the YAML 1.2 format defined from Flink 1.19 onward and the
only format Flink 2.0 accepts. It supports nested keys (see the [Flink
Configuration
File](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#flink-configuration-file)
documentation):
+
+```yaml
+defaultConfiguration:
+ create: true
+ append: true
+ config.yaml: |+
+ kubernetes.operator:
+ metrics.reporter.slf4j:
+ factory.class: org.apache.flink.metrics.slf4j.Slf4jReporterFactory
+ interval: 5 MINUTE
+ reconcile.interval: 15 s
+ observer.progress-check.interval: 5 s
+```
+
+When both keys are set, `config.yaml` takes precedence. Either way the
resolved configuration is mounted as `config.yaml`, the file name Flink 2.0
requires.
{{< hint info >}}
-Flink 2.0 onward also defines a `config.yaml` format with YAML 1.2 syntax that
supports nested keys (see the [Flink Configuration
File](https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/config/#flink-configuration-file)
documentation). The Helm chart's `values.yaml` shows a `config.yaml` example
block, but with the current chart it is not effectively loadable: the ConfigMap
template unconditionally renders both `flink-conf.yaml` and `config.yaml` keys,
and Flink's [...]
+Prefer the nested form shown above. A flat, dotted-key override of a key that
the chart's default `conf/config.yaml` already ships would be a duplicate key
under Flink's strict YAML parser and fail at operator startup.
{{< /hint >}}
### Flink Version and Namespace Specific Defaults
@@ -101,7 +119,7 @@ Whether dynamic configuration updates are enabled can be
verified in the `deploy
o.a.f.k.o.c.FlinkConfigManager [INFO ] Enabled dynamic config updates,
checking config changes every PT5M
```
-To change config values dynamically, the ConfigMap can be edited directly with
`kubectl patch` or `kubectl edit`. For example, the reconcile interval is
changed by overriding `kubernetes.operator.reconcile.interval`.
+To change config values dynamically, the ConfigMap can be edited directly with
`kubectl patch` or `kubectl edit`. Edits belong to the `config.yaml` key, which
is the file the operator mounts. For example, the reconcile interval is changed
by overriding `kubernetes.operator.reconcile.interval`.
The update, here to 30 seconds, is confirmed in the operator log:
diff --git a/docs/content.zh/docs/deployment/helm/installation.md
b/docs/content.zh/docs/deployment/helm/installation.md
index 0ed010b3..46c6d9ff 100644
--- a/docs/content.zh/docs/deployment/helm/installation.md
+++ b/docs/content.zh/docs/deployment/helm/installation.md
@@ -75,9 +75,9 @@ The configurable parameters of the Helm chart and their
default values are detai
| Parameters | Description
| Default Value
[...]
|------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
| defaultConfiguration.append | Whether to append
configuration files with configs.
| true
[...]
-| defaultConfiguration.config.yaml | The newer configuration
file format for Flink that will be enforced in Flink 2.0. Note this was
introduced in Flink 1.19. |
kubernetes.operator.metrics.reporter.slf4j.factory.class:
org.apache.flink.metrics.slf4j.Slf4jReporterFactory<br/>kubernetes.operator.metrics.reporter.slf4j.interval:
5 MINUTE<br/>kubernetes.operator.reconcile.interval: 15
s<br/>kubernetes.operator.observer.progress-check.interva [...]
+| defaultConfiguration.config.yaml | Modern YAML 1.2 config
format (introduced in Flink 1.19, required by Flink 2.0). Takes precedence over
flink-conf.yaml when set; the resolved config is always mounted as config.yaml.
|
kubernetes.operator.metrics.reporter.slf4j.factory.class:
org.apache.flink.metrics.slf4j.Slf4jReporterFactory<br/>kubernetes.operator.metrics.reporter.slf4j.interval:
5 MINUTE<br/>kubernetes.operator.reconcile.interv [...]
| defaultConfiguration.create | Whether to enable default
configuration to create for flink-kubernetes-operator.
| true
[...]
-| defaultConfiguration.flink-conf.yaml | The default configuration
of flink-conf.yaml.
|
kubernetes.operator.metrics.reporter.slf4j.factory.class:
org.apache.flink.metrics.slf4j.Slf4jReporterFactory<br/>kubernetes.operator.metrics.reporter.slf4j.interval:
5 MINUTE<br/>kubernetes.operator.reconcile.interval: 15
s<br/>kubernetes.operator.observer.progress-check.interva [...]
+| defaultConfiguration.flink-conf.yaml | Legacy flat config format.
Used when config.yaml is not set; the resolved config is always mounted as
config.yaml.
|
kubernetes.operator.metrics.reporter.slf4j.factory.class:
org.apache.flink.metrics.slf4j.Slf4jReporterFactory<br/>kubernetes.operator.metrics.reporter.slf4j.interval:
5 MINUTE<br/>kubernetes.operator.reconcile.i [...]
| defaultConfiguration.log4j-console.properties | The default configuration
of log4j-console.properties.
|
[...]
| defaultConfiguration.log4j-operator.properties | The default configuration
of log4j-operator.properties.
|
[...]
| defaultConfiguration.logback-operator.xml | The default configuration
of logback-operator.xml. Used when `logging.framework` is set to `logback`.
|
[...]
diff --git a/docs/content.zh/docs/internals/startup.md
b/docs/content.zh/docs/internals/startup.md
index f37d4058..2f5f1b1b 100644
--- a/docs/content.zh/docs/internals/startup.md
+++ b/docs/content.zh/docs/internals/startup.md
@@ -91,7 +91,7 @@ helm install -f custom-values.yaml flink-kubernetes-operator
helm/flink-kubern
│
│ flink-kubernetes-operator Deployment is
created with:
│ ENV: LOG_CONFIG=..., JVM_ARGS=...,
OPERATOR_NAMESPACE, ...
- │ Volume items: config.yaml (default)
or flink-conf.yaml,
+ │ Volume items: config.yaml,
│
log4j-operator.properties or logback-operator.xml,
│
log4j-console.properties or logback-console.xml
▼
@@ -118,10 +118,10 @@ The following notes elaborate on the chart's CRD
handling, helper-driven labels,
- `crds/` uses Helm's special CRD handling: applied on `helm install` only,
untouched by `helm upgrade` and `helm uninstall`. Based on this, Helm CRD
upgrades and deletions are out-of-band today, and they need to be manually
executed via `kubectl` commands (`kubectl apply -f crds/`, `kubectl delete crd
...`). See [Upgrading the Operator → Upgrading the CRDs]({{< ref
"docs/operations/upgrade#2-upgrading-the-crds" >}}) for the supported procedure.
- `_helpers.tpl` defines a common label set applied to every chart-rendered
resource: `app.kubernetes.io/name`, `app.kubernetes.io/version`,
`app.kubernetes.io/managed-by`, and `helm.sh/chart`. The same
`app.kubernetes.io/name` label is the operator Deployment's
`selector.matchLabels`, so a single `kubectl get all -l
app.kubernetes.io/name=<release-name>` reaches every chart-created object.
- - The `flink-operator-config` ConfigMap is created only when
`defaultConfiguration.create` is set to `true`, and it carries six data keys
(`config.yaml`, `flink-conf.yaml`, `log4j-operator.properties`,
`log4j-console.properties`, `logback-operator.xml`, `logback-console.xml`)
populated with two different strategies:
- - Append: chart's `conf/flink-conf.yaml`, `conf/log4j-operator.properties`,
and `conf/log4j-console.properties` baselines (when
`defaultConfiguration.append=true`), followed by user-supplied
`defaultConfiguration."config.yaml"` / `"flink-conf.yaml"` /
`"log4j-operator.properties"` / `"log4j-console.properties"` overrides
concatenated on top.
+ - The `flink-operator-config` ConfigMap is created only when
`defaultConfiguration.create` is set to `true`, and it carries five data keys
(`config.yaml`, `log4j-operator.properties`, `log4j-console.properties`,
`logback-operator.xml`, `logback-console.xml`) populated with two different
strategies:
+ - Append: chart's `conf/config.yaml`, `conf/log4j-operator.properties`, and
`conf/log4j-console.properties` baselines (when
`defaultConfiguration.append=true`), followed by user-supplied
`defaultConfiguration."config.yaml"` / `"log4j-operator.properties"` /
`"log4j-console.properties"` overrides concatenated on top. The single
`config.yaml` entry resolves `defaultConfiguration."config.yaml"` when set,
otherwise `defaultConfiguration."flink-conf.yaml"`.
- Replace: user-supplied `defaultConfiguration."logback-operator.xml"` /
`"logback-console.xml"` if set, otherwise the chart's `conf/logback-*.xml`
baseline when `defaultConfiguration.append=true`. The two sources are mutually
exclusive, not concatenated.
- - For the two top-level YAML keys (`config.yaml`, `flink-conf.yaml`),
`kubernetes.operator.watched.namespaces` and
`kubernetes.operator.health.probe.*` are auto-injected when the matching
`values.yaml` fields are set.
+ - In the `config.yaml` entry, `kubernetes.operator.watched.namespaces` and
`kubernetes.operator.health.probe.*` are auto-injected when the matching
`values.yaml` fields are set.
- The operator `Deployment` is a standard Pod-spec passthrough configured via
`operatorPod.*` and related keys in `values.yaml`:
- container `name`, `image`, `command`, `ports` (metrics, health-probe),
`volumeMounts` (config, artifacts, TLS cert), `env`, `envFrom`,
`livenessProbe`, `startupProbe`, `lifecycle.postStart`, and `securityContext`.
- pod-level scheduling fields `nodeSelector`, `affinity`, `tolerations`,
`topologySpreadConstraints`, `priorityClassName`, `serviceAccountName`,
`imagePullSecrets`, and optional `initContainers` / `sidecarContainers`.
diff --git a/docs/content/docs/deployment/compatibility.md
b/docs/content/docs/deployment/compatibility.md
index fe627034..fce7d41d 100644
--- a/docs/content/docs/deployment/compatibility.md
+++ b/docs/content/docs/deployment/compatibility.md
@@ -97,7 +97,7 @@ The operator JVM links against a specific Flink release for
its own runtime need
This bundled runtime is used by the operator process for purposes such as:
- Pluggable filesystem access (`org.apache.flink.core.fs.FileSystem`), used to
fetch `FlinkSessionJob` artifacts from non-HTTP schemes such as `s3://`,
`gs://`, `hdfs://`.
-- Configuration parsing (`org.apache.flink.configuration.GlobalConfiguration`)
that reads the operator's own `flink-conf.yaml`.
+- Configuration parsing (`org.apache.flink.configuration.GlobalConfiguration`)
that reads the operator's own `config.yaml`.
- Plugin discovery (`org.apache.flink.core.plugin.PluginManager`) that loads
custom validators, mutators, listeners, and filesystem factories.
## Kubernetes
diff --git a/docs/content/docs/deployment/configuration.md
b/docs/content/docs/deployment/configuration.md
index 6d08bc9b..3c66a0c8 100644
--- a/docs/content/docs/deployment/configuration.md
+++ b/docs/content/docs/deployment/configuration.md
@@ -51,10 +51,28 @@ defaultConfiguration:
### YAML Configuration File Format
-Operator configuration is delivered through the
`defaultConfiguration.flink-conf.yaml` block in the Helm `values.yaml`. The
operator's runtime loads that file with Flink's legacy configuration parser,
which accepts only flat key-value pairs written with dot notation and does not
support nested YAML maps.
+Operator configuration can be delivered in either of Flink's two configuration
formats, by defining the matching key under `defaultConfiguration` in the Helm
`values.yaml`:
+
+- `flink-conf.yaml` is the legacy format, which accepts only flat key-value
pairs written with dot notation and does not support nested YAML maps. This is
the format the chart ships by default.
+- `config.yaml` is the YAML 1.2 format defined from Flink 1.19 onward and the
only format Flink 2.0 accepts. It supports nested keys (see the [Flink
Configuration
File](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#flink-configuration-file)
documentation):
+
+```yaml
+defaultConfiguration:
+ create: true
+ append: true
+ config.yaml: |+
+ kubernetes.operator:
+ metrics.reporter.slf4j:
+ factory.class: org.apache.flink.metrics.slf4j.Slf4jReporterFactory
+ interval: 5 MINUTE
+ reconcile.interval: 15 s
+ observer.progress-check.interval: 5 s
+```
+
+When both keys are set, `config.yaml` takes precedence. Either way the
resolved configuration is mounted as `config.yaml`, the file name Flink 2.0
requires.
{{< hint info >}}
-Flink 2.0 onward also defines a `config.yaml` format with YAML 1.2 syntax that
supports nested keys (see the [Flink Configuration
File](https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/config/#flink-configuration-file)
documentation). The Helm chart's `values.yaml` shows a `config.yaml` example
block, but with the current chart it is not effectively loadable: the ConfigMap
template unconditionally renders both `flink-conf.yaml` and `config.yaml` keys,
and Flink's [...]
+Prefer the nested form shown above. A flat, dotted-key override of a key that
the chart's default `conf/config.yaml` already ships would be a duplicate key
under Flink's strict YAML parser and fail at operator startup.
{{< /hint >}}
### Flink Version and Namespace Specific Defaults
@@ -101,7 +119,7 @@ Whether dynamic configuration updates are enabled can be
verified in the `deploy
o.a.f.k.o.c.FlinkConfigManager [INFO ] Enabled dynamic config updates,
checking config changes every PT5M
```
-To change config values dynamically, the ConfigMap can be edited directly with
`kubectl patch` or `kubectl edit`. For example, the reconcile interval is
changed by overriding `kubernetes.operator.reconcile.interval`.
+To change config values dynamically, the ConfigMap can be edited directly with
`kubectl patch` or `kubectl edit`. Edits belong to the `config.yaml` key, which
is the file the operator mounts. For example, the reconcile interval is changed
by overriding `kubernetes.operator.reconcile.interval`.
The update, here to 30 seconds, is confirmed in the operator log:
diff --git a/docs/content/docs/deployment/helm/installation.md
b/docs/content/docs/deployment/helm/installation.md
index 0ed010b3..46c6d9ff 100644
--- a/docs/content/docs/deployment/helm/installation.md
+++ b/docs/content/docs/deployment/helm/installation.md
@@ -75,9 +75,9 @@ The configurable parameters of the Helm chart and their
default values are detai
| Parameters | Description
| Default Value
[...]
|------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
| defaultConfiguration.append | Whether to append
configuration files with configs.
| true
[...]
-| defaultConfiguration.config.yaml | The newer configuration
file format for Flink that will be enforced in Flink 2.0. Note this was
introduced in Flink 1.19. |
kubernetes.operator.metrics.reporter.slf4j.factory.class:
org.apache.flink.metrics.slf4j.Slf4jReporterFactory<br/>kubernetes.operator.metrics.reporter.slf4j.interval:
5 MINUTE<br/>kubernetes.operator.reconcile.interval: 15
s<br/>kubernetes.operator.observer.progress-check.interva [...]
+| defaultConfiguration.config.yaml | Modern YAML 1.2 config
format (introduced in Flink 1.19, required by Flink 2.0). Takes precedence over
flink-conf.yaml when set; the resolved config is always mounted as config.yaml.
|
kubernetes.operator.metrics.reporter.slf4j.factory.class:
org.apache.flink.metrics.slf4j.Slf4jReporterFactory<br/>kubernetes.operator.metrics.reporter.slf4j.interval:
5 MINUTE<br/>kubernetes.operator.reconcile.interv [...]
| defaultConfiguration.create | Whether to enable default
configuration to create for flink-kubernetes-operator.
| true
[...]
-| defaultConfiguration.flink-conf.yaml | The default configuration
of flink-conf.yaml.
|
kubernetes.operator.metrics.reporter.slf4j.factory.class:
org.apache.flink.metrics.slf4j.Slf4jReporterFactory<br/>kubernetes.operator.metrics.reporter.slf4j.interval:
5 MINUTE<br/>kubernetes.operator.reconcile.interval: 15
s<br/>kubernetes.operator.observer.progress-check.interva [...]
+| defaultConfiguration.flink-conf.yaml | Legacy flat config format.
Used when config.yaml is not set; the resolved config is always mounted as
config.yaml.
|
kubernetes.operator.metrics.reporter.slf4j.factory.class:
org.apache.flink.metrics.slf4j.Slf4jReporterFactory<br/>kubernetes.operator.metrics.reporter.slf4j.interval:
5 MINUTE<br/>kubernetes.operator.reconcile.i [...]
| defaultConfiguration.log4j-console.properties | The default configuration
of log4j-console.properties.
|
[...]
| defaultConfiguration.log4j-operator.properties | The default configuration
of log4j-operator.properties.
|
[...]
| defaultConfiguration.logback-operator.xml | The default configuration
of logback-operator.xml. Used when `logging.framework` is set to `logback`.
|
[...]
diff --git a/docs/content/docs/internals/startup.md
b/docs/content/docs/internals/startup.md
index f37d4058..2f5f1b1b 100644
--- a/docs/content/docs/internals/startup.md
+++ b/docs/content/docs/internals/startup.md
@@ -91,7 +91,7 @@ helm install -f custom-values.yaml flink-kubernetes-operator
helm/flink-kubern
│
│ flink-kubernetes-operator Deployment is
created with:
│ ENV: LOG_CONFIG=..., JVM_ARGS=...,
OPERATOR_NAMESPACE, ...
- │ Volume items: config.yaml (default)
or flink-conf.yaml,
+ │ Volume items: config.yaml,
│
log4j-operator.properties or logback-operator.xml,
│
log4j-console.properties or logback-console.xml
▼
@@ -118,10 +118,10 @@ The following notes elaborate on the chart's CRD
handling, helper-driven labels,
- `crds/` uses Helm's special CRD handling: applied on `helm install` only,
untouched by `helm upgrade` and `helm uninstall`. Based on this, Helm CRD
upgrades and deletions are out-of-band today, and they need to be manually
executed via `kubectl` commands (`kubectl apply -f crds/`, `kubectl delete crd
...`). See [Upgrading the Operator → Upgrading the CRDs]({{< ref
"docs/operations/upgrade#2-upgrading-the-crds" >}}) for the supported procedure.
- `_helpers.tpl` defines a common label set applied to every chart-rendered
resource: `app.kubernetes.io/name`, `app.kubernetes.io/version`,
`app.kubernetes.io/managed-by`, and `helm.sh/chart`. The same
`app.kubernetes.io/name` label is the operator Deployment's
`selector.matchLabels`, so a single `kubectl get all -l
app.kubernetes.io/name=<release-name>` reaches every chart-created object.
- - The `flink-operator-config` ConfigMap is created only when
`defaultConfiguration.create` is set to `true`, and it carries six data keys
(`config.yaml`, `flink-conf.yaml`, `log4j-operator.properties`,
`log4j-console.properties`, `logback-operator.xml`, `logback-console.xml`)
populated with two different strategies:
- - Append: chart's `conf/flink-conf.yaml`, `conf/log4j-operator.properties`,
and `conf/log4j-console.properties` baselines (when
`defaultConfiguration.append=true`), followed by user-supplied
`defaultConfiguration."config.yaml"` / `"flink-conf.yaml"` /
`"log4j-operator.properties"` / `"log4j-console.properties"` overrides
concatenated on top.
+ - The `flink-operator-config` ConfigMap is created only when
`defaultConfiguration.create` is set to `true`, and it carries five data keys
(`config.yaml`, `log4j-operator.properties`, `log4j-console.properties`,
`logback-operator.xml`, `logback-console.xml`) populated with two different
strategies:
+ - Append: chart's `conf/config.yaml`, `conf/log4j-operator.properties`, and
`conf/log4j-console.properties` baselines (when
`defaultConfiguration.append=true`), followed by user-supplied
`defaultConfiguration."config.yaml"` / `"log4j-operator.properties"` /
`"log4j-console.properties"` overrides concatenated on top. The single
`config.yaml` entry resolves `defaultConfiguration."config.yaml"` when set,
otherwise `defaultConfiguration."flink-conf.yaml"`.
- Replace: user-supplied `defaultConfiguration."logback-operator.xml"` /
`"logback-console.xml"` if set, otherwise the chart's `conf/logback-*.xml`
baseline when `defaultConfiguration.append=true`. The two sources are mutually
exclusive, not concatenated.
- - For the two top-level YAML keys (`config.yaml`, `flink-conf.yaml`),
`kubernetes.operator.watched.namespaces` and
`kubernetes.operator.health.probe.*` are auto-injected when the matching
`values.yaml` fields are set.
+ - In the `config.yaml` entry, `kubernetes.operator.watched.namespaces` and
`kubernetes.operator.health.probe.*` are auto-injected when the matching
`values.yaml` fields are set.
- The operator `Deployment` is a standard Pod-spec passthrough configured via
`operatorPod.*` and related keys in `values.yaml`:
- container `name`, `image`, `command`, `ports` (metrics, health-probe),
`volumeMounts` (config, artifacts, TLS cert), `env`, `envFrom`,
`livenessProbe`, `startupProbe`, `lifecycle.postStart`, and `securityContext`.
- pod-level scheduling fields `nodeSelector`, `affinity`, `tolerations`,
`topologySpreadConstraints`, `priorityClassName`, `serviceAccountName`,
`imagePullSecrets`, and optional `initContainers` / `sidecarContainers`.
diff --git a/e2e-tests/test_dynamic_flink_conf.sh
b/e2e-tests/test_dynamic_flink_conf.sh
index 3569d2c3..1552804f 100755
--- a/e2e-tests/test_dynamic_flink_conf.sh
+++ b/e2e-tests/test_dynamic_flink_conf.sh
@@ -17,7 +17,9 @@
# limitations under the License.
################################################################################
-# This script tests the operator dynamic config on watched namespace:
+# This script tests the operator dynamic config on watched namespace, with the
chart configured
+# through the legacy defaultConfiguration.flink-conf.yaml key, which the chart
resolves into the
+# config.yaml the operator mounts:
# 1. Create a new namespace
# 2. Change the watched namespaces by patching on the flink-config-override
# 3. Monitor the operator log to find the watched namespace changed info
@@ -34,7 +36,7 @@ echo "Current operator pod is ${operator_pod}"
create_namespace dynamic
kubectl config set-context --current --namespace="${operator_namespace}"
-patch_flink_config '{"data": {"flink-conf.yaml":
"kubernetes.operator.watched.namespaces: default,flink,dynamic"}}'
+patch_flink_config '{"data": {"config.yaml":
"kubernetes.operator.watched.namespaces: default,flink,dynamic"}}'
wait_for_operator_logs "${operator_pod}" "Setting default configuration to
{kubernetes.operator.watched.namespaces=default,flink,dynamic}" ${TIMEOUT} ||
exit 1
echo "Successfully run the dynamic property test"
diff --git
a/flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/config/FlinkConfigManagerTest.java
b/flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/config/FlinkConfigManagerTest.java
index 39414af4..715c072b 100644
---
a/flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/config/FlinkConfigManagerTest.java
+++
b/flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/config/FlinkConfigManagerTest.java
@@ -18,6 +18,7 @@
package org.apache.flink.kubernetes.operator.config;
import org.apache.flink.autoscaler.config.AutoScalerOptions;
+import org.apache.flink.configuration.ConfigConstants;
import org.apache.flink.configuration.ConfigOption;
import org.apache.flink.configuration.ConfigOptions;
import org.apache.flink.configuration.Configuration;
@@ -210,6 +211,44 @@ public class FlinkConfigManagerTest {
assertEquals(Map.of("foo", "1", "bar", "2"), conf.toMap());
}
+ /**
+ * The operator configuration uses the config.yaml format while an
external override directory
+ * may still be mounted in the legacy flink-conf.yaml format.
+ */
+ @Test
+ public void testLegacyFormatConfigOverrides(
+ @TempDir Path confDir, @TempDir Path confOverrideDir) throws
IOException {
+ Map<String, String> originalEnv = System.getenv();
+ try {
+ Files.write(
+ confDir.resolve(GlobalConfiguration.FLINK_CONF_FILENAME),
+ List.of("kubernetes.operator.reconcile.interval: 15 s",
"foo: 1"));
+ Files.write(
+
confOverrideDir.resolve(GlobalConfiguration.LEGACY_FLINK_CONF_FILENAME),
+ List.of("kubernetes.operator.reconcile.interval: 30 s",
"bar: 2"));
+
+ Map<String, String> systemEnv = new HashMap<>(originalEnv);
+ systemEnv.put(ConfigConstants.ENV_FLINK_CONF_DIR,
confDir.toString());
+ TestUtils.setEnv(systemEnv);
+
+ var conf =
+ FlinkConfigManager.loadGlobalConfiguration(
+ Optional.of(confOverrideDir.toString()));
+
+ // Overrides in the legacy format are loaded and take precedence
+ assertEquals(
+ Duration.ofSeconds(30),
+
conf.get(KubernetesOperatorConfigOptions.OPERATOR_RECONCILE_INTERVAL));
+ assertEquals("2", conf.toMap().get("bar"));
+ // Settings only present in the operator configuration are kept
+ assertEquals("1", conf.toMap().get("foo"));
+ // The operator configuration is loaded last, so its format
decides how values parse
+ assertTrue(GlobalConfiguration.isStandardYaml());
+ } finally {
+ TestUtils.setEnv(originalEnv);
+ }
+ }
+
@Test
public void testWatchNamespaceOverride() {
Map<String, String> originalEnv = System.getenv();
diff --git a/helm/flink-kubernetes-operator/conf/flink-conf.yaml
b/helm/flink-kubernetes-operator/conf/config.yaml
similarity index 98%
rename from helm/flink-kubernetes-operator/conf/flink-conf.yaml
rename to helm/flink-kubernetes-operator/conf/config.yaml
index 0f6cd746..a04a8237 100644
--- a/helm/flink-kubernetes-operator/conf/flink-conf.yaml
+++ b/helm/flink-kubernetes-operator/conf/config.yaml
@@ -16,10 +16,6 @@
# limitations under the License.
################################################################################
-# Flink job/cluster related configs
-taskmanager.numberOfTaskSlots: 1
-parallelism.default: 1
-
# These parameters are required for Java 17 support.
# Flink 1.18 uses env.java.opts.all, if a user supplies their own version of
these opts in their FlinkDeployment the options below will be overridden.
# env.java.default-opts.all is used for 1.19 onwards so users can supply their
own opts.all in their Job deployments and have these appended.
diff --git a/helm/flink-kubernetes-operator/templates/controller/configmap.yaml
b/helm/flink-kubernetes-operator/templates/controller/configmap.yaml
index 5ab29cd8..18f61e9b 100644
--- a/helm/flink-kubernetes-operator/templates/controller/configmap.yaml
+++ b/helm/flink-kubernetes-operator/templates/controller/configmap.yaml
@@ -25,24 +25,14 @@ metadata:
data:
config.yaml: |+
{{- if .Values.defaultConfiguration.append }}
- {{- $.Files.Get "conf/flink-conf.yaml" | nindent 4 -}}
+ {{- $.Files.Get "conf/config.yaml" | nindent 4 -}}
{{- end }}
-{{- if hasKey (.Values.defaultConfiguration) "config.yaml" }}
- {{- index (.Values.defaultConfiguration) "config.yaml" | nindent 4 -}}
-{{- end }}
-{{- if .Values.watchNamespaces }}
- kubernetes.operator.watched.namespaces: {{ join ","
.Values.watchNamespaces }}
-{{- end }}
-{{- if index .Values "operatorHealth" }}
- kubernetes.operator.health.probe.enabled: true
- kubernetes.operator.health.probe.port: {{ .Values.operatorHealth.port }}
-{{- end }}
- flink-conf.yaml: |+
-{{- if .Values.defaultConfiguration.append }}
- {{- $.Files.Get "conf/flink-conf.yaml" | nindent 4 -}}
-{{- end }}
-{{- if hasKey (.Values.defaultConfiguration) "flink-conf.yaml" }}
- {{- index (.Values.defaultConfiguration) "flink-conf.yaml" | nindent 4 -}}
+{{- with (index .Values.defaultConfiguration "config.yaml") }}
+ {{- . | nindent 4 -}}
+{{- else }}
+ {{- with (index .Values.defaultConfiguration "flink-conf.yaml") }}
+ {{- . | nindent 4 -}}
+ {{- end }}
{{- end }}
{{- if .Values.watchNamespaces }}
kubernetes.operator.watched.namespaces: {{ join ","
.Values.watchNamespaces }}
diff --git
a/helm/flink-kubernetes-operator/templates/controller/deployment.yaml
b/helm/flink-kubernetes-operator/templates/controller/deployment.yaml
index a31175c9..036e2716 100644
--- a/helm/flink-kubernetes-operator/templates/controller/deployment.yaml
+++ b/helm/flink-kubernetes-operator/templates/controller/deployment.yaml
@@ -244,13 +244,8 @@ spec:
configMap:
name: flink-operator-config
items:
- {{- if hasKey .Values.defaultConfiguration "flink-conf.yaml" }}
- - key: flink-conf.yaml
- path: flink-conf.yaml
- {{- else }}
- key: config.yaml
path: config.yaml
- {{- end }}
{{- if eq .Values.logging.framework "logback" }}
- key: logback-operator.xml
path: logback-operator.xml
diff --git
a/helm/flink-kubernetes-operator/tests/controller/configmap_test.yaml
b/helm/flink-kubernetes-operator/tests/controller/configmap_test.yaml
index 90476695..74ec6111 100644
--- a/helm/flink-kubernetes-operator/tests/controller/configmap_test.yaml
+++ b/helm/flink-kubernetes-operator/tests/controller/configmap_test.yaml
@@ -68,3 +68,64 @@ tests:
asserts:
- isNotNull:
path: data["log4j-console.properties"]
+
+ - it: Emits a single config.yaml key with today's effective defaults and no
redundant seed keys
+ asserts:
+ - isNotNull:
+ path: data["config.yaml"]
+ - notExists:
+ path: data["flink-conf.yaml"]
+ - matchRegex:
+ path: data["config.yaml"]
+ pattern:
"(?m)^\\s*kubernetes.operator.metrics.reporter.slf4j.factory.class:"
+ - matchRegex:
+ path: data["config.yaml"]
+ pattern: "(?m)^\\s*kubernetes.operator.reconcile.interval: 15 s"
+ - matchRegex:
+ path: data["config.yaml"]
+ pattern: "(?m)^\\s*kubernetes.operator.health.probe.enabled: true"
+ - notMatchRegex:
+ path: data["config.yaml"]
+ pattern: "(?m)^\\s*taskmanager.numberOfTaskSlots:"
+ - notMatchRegex:
+ path: data["config.yaml"]
+ pattern: "(?m)^\\s*parallelism.default:"
+
+ - it: A user flink-conf.yaml override flows into config.yaml when
config.yaml is unset
+ set:
+ defaultConfiguration:
+ flink-conf.yaml: |+
+ kubernetes.operator.reconcile.interval: 99 s
+ asserts:
+ - matchRegex:
+ path: data["config.yaml"]
+ pattern: "reconcile.interval: 99 s"
+
+ - it: When both are set, config.yaml wins and flink-conf.yaml is ignored
+ set:
+ defaultConfiguration:
+ config.yaml: |+
+ kubernetes.operator:
+ reconcile.interval: 30 s
+ flink-conf.yaml: |+
+ kubernetes.operator.reconcile.interval: 99 s
+ asserts:
+ - matchRegex:
+ path: data["config.yaml"]
+ pattern: "reconcile.interval: 30 s"
+ - notMatchRegex:
+ path: data["config.yaml"]
+ pattern: "reconcile.interval: 99 s"
+
+ - it: A flat config.yaml override of a former seed key is not duplicated
(relies on dropping it from the seed)
+ set:
+ defaultConfiguration:
+ config.yaml: |+
+ taskmanager.numberOfTaskSlots: 2
+ asserts:
+ - matchRegex:
+ path: data["config.yaml"]
+ pattern: "taskmanager.numberOfTaskSlots: 2"
+ - notMatchRegex:
+ path: data["config.yaml"]
+ pattern: "taskmanager.numberOfTaskSlots: 1"
diff --git
a/helm/flink-kubernetes-operator/tests/controller/deployment_test.yaml
b/helm/flink-kubernetes-operator/tests/controller/deployment_test.yaml
index e6333d76..08dda10d 100644
--- a/helm/flink-kubernetes-operator/tests/controller/deployment_test.yaml
+++ b/helm/flink-kubernetes-operator/tests/controller/deployment_test.yaml
@@ -383,3 +383,16 @@ tests:
key: log4j-operator.properties
path: log4j-operator.properties
+- it: Should mount the operator config as config.yaml by default
+ asserts:
+ - contains:
+ path: spec.template.spec.volumes[0].configMap.items
+ content:
+ key: config.yaml
+ path: config.yaml
+ - notContains:
+ path: spec.template.spec.volumes[0].configMap.items
+ content:
+ key: flink-conf.yaml
+ path: flink-conf.yaml
+
diff --git a/helm/flink-kubernetes-operator/values.yaml
b/helm/flink-kubernetes-operator/values.yaml
index 8eaa936e..aecf1d01 100644
--- a/helm/flink-kubernetes-operator/values.yaml
+++ b/helm/flink-kubernetes-operator/values.yaml
@@ -174,7 +174,16 @@ defaultConfiguration:
# This option has not effect, if create is equal to false.
append: true
-# Uncomment to use the new config.yaml format, but also comment out the
flink-config.yaml key.
+# Operator configuration overrides go under ONE of the two keys below. If both
are set,
+# config.yaml wins; either way the operator mounts the result as config.yaml.
+#
+# config.yaml - modern YAML 1.2 format (Flink 1.19+, required by Flink
2.0). Recommended.
+# flink-conf.yaml - legacy flat "key: value" format (shipped as the default
below).
+#
+# Prefer the nested config.yaml form: a flat, dotted-key override of a key the
chart already
+# ships (e.g. the Java 17 opts in conf/config.yaml) would be a duplicate key
under Flink's
+# strict YAML parser and fail at operator startup. Example, equivalent to the
default below:
+#
# config.yaml: |+
# kubernetes.operator:
# metrics.reporter.slf4j:
