This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 6be1a39b [SPARK-59991] Remove unused `operatorConfiguration.append`
and fix broken `conf/` links
6be1a39b is described below
commit 6be1a39b85df6527387275f6ae998c0a71308c45
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Oct 6 06:25:30 2026 -0700
[SPARK-59991] Remove unused `operatorConfiguration.append` and fix broken
`conf/` links
### What changes were proposed in this pull request?
This PR aims to remove the unused `operatorConfiguration.append` from the
Helm chart and fix the broken `conf/` links in the documentation.
### Why are the changes needed?
The Helm chart has never had a `conf/` directory, so
`operatorConfiguration.append` has no effect in any release, e.g.
[1.0.0](https://github.com/apache/spark-kubernetes-operator/releases/tag/1.0.0)
(2026-07-23). However, `values.yaml` and the documentation still describe it
and link to the non-existent conf files.
### Does this PR introduce _any_ user-facing change?
Yes, `operatorConfiguration.append` is removed, but the rendered manifests
are identical. The migration guide is updated.
### How was this patch tested?
Pass the CIs. I also verified manually that `helm template` renders
identical manifests before and after this PR.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #937 from dongjoon-hyun/SPARK-59991.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../helm/spark-kubernetes-operator/templates/spark-operator.yaml | 6 ------
build-tools/helm/spark-kubernetes-operator/values.schema.json | 7 -------
build-tools/helm/spark-kubernetes-operator/values.yaml | 3 ---
docs/configuration.md | 5 ++---
docs/migration_guide.md | 6 ++++++
docs/operations.md | 3 +--
tests/e2e/helm/helm-test-values/configmap-metadata/values.yaml | 1 -
7 files changed, 9 insertions(+), 22 deletions(-)
diff --git
a/build-tools/helm/spark-kubernetes-operator/templates/spark-operator.yaml
b/build-tools/helm/spark-kubernetes-operator/templates/spark-operator.yaml
index a49a88a8..bee73ba8 100644
--- a/build-tools/helm/spark-kubernetes-operator/templates/spark-operator.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/templates/spark-operator.yaml
@@ -218,15 +218,9 @@ metadata:
{{- end }}
data:
log4j2.properties: |
-{{- if .Values.operatorConfiguration.append }}
-{{- with $.Files.Get "conf/log4j2.properties" }}{{ . | nindent 4 }}{{ end }}
-{{- end }}
{{- with index .Values.operatorConfiguration "log4j2.properties" }}{{ . |
nindent 4 }}{{ end }}
spark-operator.properties: |
{{- include "spark-operator.defaultPropertyOverrides" . | nindent 4 }}
-{{- if .Values.operatorConfiguration.append }}
-{{- with $.Files.Get "conf/spark-operator.properties" }}{{ . | nindent 4
}}{{ end }}
-{{- end }}
{{- with index .Values.operatorConfiguration "spark-operator.properties" }}{{
. | nindent 4 }}{{ end }}
metrics.properties: |
{{- with index .Values.operatorConfiguration "metrics.properties" }}{{ . |
nindent 4 }}{{ end }}
diff --git a/build-tools/helm/spark-kubernetes-operator/values.schema.json
b/build-tools/helm/spark-kubernetes-operator/values.schema.json
index 082da38e..5d0dc51a 100644
--- a/build-tools/helm/spark-kubernetes-operator/values.schema.json
+++ b/build-tools/helm/spark-kubernetes-operator/values.schema.json
@@ -792,14 +792,7 @@
"operatorConfiguration": {
"type": "object",
"description": "Operator configuration",
- "required": [
- "append"
- ],
"properties": {
- "append": {
- "type": "boolean",
- "description": "Append configuration instead of replacing"
- },
"configMap": {
"type": "object",
"description": "Configuration for the operator config map",
diff --git a/build-tools/helm/spark-kubernetes-operator/values.yaml
b/build-tools/helm/spark-kubernetes-operator/values.yaml
index 05077eea..a59cb0c1 100644
--- a/build-tools/helm/spark-kubernetes-operator/values.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/values.yaml
@@ -209,9 +209,6 @@ workloadResources:
"app.kubernetes.io/component": "spark-workload"
operatorConfiguration:
- # If set to true, below properties would be appended to default conf files
under conf/
- # Otherwise, below would override default conf files
- append: true
# Labels and annotations for the operatorConfiguration config map
configMap:
annotations:
diff --git a/docs/configuration.md b/docs/configuration.md
index 79628589..a67ee0fc 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -23,9 +23,8 @@ under the License.
Spark Operator supports different ways to configure the behavior:
-* **spark-operator.properties** provided when deploying the operator. In
addition to the
- [property
file](../build-tools/helm/spark-kubernetes-operator/conf/spark-operator.properties),
- it is also possible to override or append config properties in helm
+* **spark-operator.properties** provided when deploying the operator. It is
possible to override
+ the [default values](./config_properties.md) of config properties in helm
[Values files](../build-tools/helm/spark-kubernetes-operator/values.yaml).
* **System Properties** : when provided as system properties (e.g. via -D
options to the
operator JVM), they apply only to the properties which are not provided in
the property file,
diff --git a/docs/migration_guide.md b/docs/migration_guide.md
index a1852739..7175caa0 100644
--- a/docs/migration_guide.md
+++ b/docs/migration_guide.md
@@ -93,6 +93,12 @@ affect building the project, CI, tests or examples.
take precedence over `JAVA_TOOL_OPTIONS` and `JDK_JAVA_OPTIONS` in
`operatorContainer.env`, and
cannot contain a space, since the value is split on spaces. To restore the
behavior before
1.1.0, set `jvmArgs` to `-Dfile.encoding=UTF8`
([SPARK-58426](https://issues.apache.org/jira/browse/SPARK-58426)).
+- Since 1.1.0, `operatorConfiguration.append` is removed, and the chart no
longer reads
+ `conf/log4j2.properties` and `conf/spark-operator.properties` in the chart
directory, which it
+ has never included. A values file which still sets `append` keeps working,
since the key is
+ ignored. If you added such files to the chart, prepend their content to
+ `operatorConfiguration.log4j2.properties` and
`operatorConfiguration.spark-operator.properties`
+ respectively to restore the behavior before 1.1.0.
### Operator
diff --git a/docs/operations.md b/docs/operations.md
index b75d82e4..1e09c7a2 100644
--- a/docs/operations.md
+++ b/docs/operations.md
@@ -162,10 +162,9 @@ following table:
| workloadResources.sparkClusterSentinel.create | If
enabled, sentinel resources will be created for operator to watch and reconcile
for the health probe purpose.
| false
|
| workloadResources.sparkClusterSentinel.name | Name for
sentinel resources.
| "spark-cluster-sentinel"
|
| workloadResources.sparkClusterSentinel.sentinelNamespaces.data | A list of
namespaces where sentinel resources will be created in. Note that these
namespaces have to be a subset of `workloadResources.namespaces.data`.
|
|
-| operatorConfiguration.append | If set to
true, below conf file & properties would be appended to default conf.
Otherwise, they would override default properties.
| true
|
| operatorConfiguration.configMap.annotations |
Annotations to be applied for the operator configuration config map.
|
|
| operatorConfiguration.configMap.labels | Labels to
be applied for the operator configuration config map.
|
|
-| operatorConfiguration.log4j2.properties | The
default log4j2 configuration.
| Refer default
[log4j2.properties](../build-tools/helm/spark-kubernetes-operator/conf/log4j2.properties)
|
+| operatorConfiguration.log4j2.properties | The
default log4j2 configuration.
| Refer default
[values.yaml](../build-tools/helm/spark-kubernetes-operator/values.yaml)
|
| operatorConfiguration.spark-operator.properties | The
default operator configuration.
|
|
| operatorConfiguration.metrics.properties | The
default operator metrics (sink) configuration.
|
|
| operatorConfiguration.dynamicConfig.create | If set to
true, a config map would be created & watched by operator as source of truth
for hot properties loading.
| false
|
diff --git a/tests/e2e/helm/helm-test-values/configmap-metadata/values.yaml
b/tests/e2e/helm/helm-test-values/configmap-metadata/values.yaml
index b0636a14..cacec108 100644
--- a/tests/e2e/helm/helm-test-values/configmap-metadata/values.yaml
+++ b/tests/e2e/helm/helm-test-values/configmap-metadata/values.yaml
@@ -26,7 +26,6 @@
# and can be used to validate the Helm test for this functionality.
operatorConfiguration:
- append: true
configMap:
labels:
test.example.com/label: "test-value"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
