This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git


The following commit(s) were added to refs/heads/main by this push:
     new ff2e83a  [SPARK-58426] Render each JVM option as its own command 
element
ff2e83a is described below

commit ff2e83a9796e67f6ff83e50dd99f8d04bfc5cd12
Author: Nicholas Barton <[email protected]>
AuthorDate: Mon Aug 3 13:18:08 2026 -0700

    [SPARK-58426] Render each JVM option as its own command element
    
    ### What changes were proposed in this pull request?
    
    Each JVM option in 
`operatorDeployment.operatorPod.operatorContainer.jvmArgs` now renders as its 
own element of the operator container's `command`, instead of the whole string 
being passed through a single `"$(OPERATOR_JAVA_OPTS)"` element. The `with` 
guard drops empty strings, so leading, trailing, and repeated spaces produce no 
empty arguments.
    
    Also adds a helm-unittest suite covering the rendered `command`, runs it 
from the existing lint job, and anchors `/tests/` in `.helmignore` so the suite 
is not packaged into the released chart.
    
    ### Why are the changes needed?
    
    Kubernetes `$(VAR)` expansion substitutes into a single argv element and 
never splits on whitespace, so the entire `jvmArgs` string reaches the JVM as 
one argument. It begins with `-D`, so the JVM reads it as one system property 
and ignores every later option. Nothing errors or warns.
    
    None of the shipped defaults (`CrashOnOutOfMemoryError`, `ErrorFile`, 
`UseParallelGC`, `InitialRAMPercentage`, `MaxRAMPercentage`, `AlwaysPreTouch`, 
and `UseCompactObjectHeaders`) have any effect. E.g. the operator runs at the 
JVM default 25% `MaxRAMPercentage` instead of 80%.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, insofar as we fixed a bug: `jvmArgs` values now take effect as 
expected.
    
    ### How was this patch tested?
    
    Pass the CIs.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (claude-opus-5)
    
    Closes #778 from snicholasbarton/SPARK-58426.
    
    Authored-by: Nicholas Barton 
<[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .github/workflows/build_and_test.yml               |   1 +
 .../templates/spark-operator.yaml                  |  14 ++-
 .../templates/tests/test-operator-command.yaml     | 114 +++++++++++++++++++++
 .../helm-test-values/operator-command/values.yaml  |  35 +++++++
 4 files changed, 163 insertions(+), 1 deletion(-)

diff --git a/.github/workflows/build_and_test.yml 
b/.github/workflows/build_and_test.yml
index da24a03..0a92289 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -237,6 +237,7 @@ jobs:
         test-group:
           - configmap-metadata
           - network-policy
+          - operator-command
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
diff --git 
a/build-tools/helm/spark-kubernetes-operator/templates/spark-operator.yaml 
b/build-tools/helm/spark-kubernetes-operator/templates/spark-operator.yaml
index 8016275..046464f 100644
--- a/build-tools/helm/spark-kubernetes-operator/templates/spark-operator.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/templates/spark-operator.yaml
@@ -76,7 +76,19 @@ spec:
         - name: {{ .Chart.Name }}
           image: {{ include "spark-operator.imagePath" . }}
           imagePullPolicy: {{ .Values.image.pullPolicy }}
-          command: ["java", "-cp", "./spark-kubernetes-operator.jar", 
"$(LOG_CONFIG)", "$(OPERATOR_JAVA_OPTS)", 
"org.apache.spark.k8s.operator.SparkOperator"]
+          # Kubernetes $(VAR) expansion never splits on whitespace: passing 
jvmArgs as one
+          # "$(OPERATOR_JAVA_OPTS)" element makes the JVM silently drop all 
but the first option.
+          command:
+            - "java"
+            - "-cp"
+            - "./spark-kubernetes-operator.jar"
+            - "$(LOG_CONFIG)"
+            {{- range splitList " " (default "" 
.Values.operatorDeployment.operatorPod.operatorContainer.jvmArgs) }}
+            {{- with . }}
+            - {{ . | quote }}
+            {{- end }}
+            {{- end }}
+            - "org.apache.spark.k8s.operator.SparkOperator"
           ports:
             - containerPort: {{ include "spark-operator.probePort" . }}
               name: probe-port
diff --git 
a/build-tools/helm/spark-kubernetes-operator/templates/tests/test-operator-command.yaml
 
b/build-tools/helm/spark-kubernetes-operator/templates/tests/test-operator-command.yaml
new file mode 100644
index 0000000..a7c82e7
--- /dev/null
+++ 
b/build-tools/helm/spark-kubernetes-operator/templates/tests/test-operator-command.yaml
@@ -0,0 +1,114 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+apiVersion: v1
+kind: Pod
+metadata:
+  name: "{{ include "spark-operator.name" . }}-test-operator-command"
+  namespace: {{ .Release.Namespace }}
+  labels:
+    {{- include "spark-operator.commonLabels" . | nindent 4 }}
+  annotations:
+    "helm.sh/hook": test
+    "helm.sh/hook-delete-policy": before-hook-creation
+spec:
+  containers:
+    - name: kubectl
+      image: bitnamisecure/kubectl:latest
+      command: ['bash', '-c']
+      args:
+        - |
+          set -e
+          echo "Testing operator container command..."
+
+          # The operator pod is selected by the deployment selector labels 
rather than by
+          # spark-role, because commonLabels puts spark-role=operator on these 
test pods too.
+          SELECTOR="app.kubernetes.io/name={{ include "spark-operator.name" . 
}},app.kubernetes.io/component=operator-deployment"
+
+          # Kubernetes stores command as authored and expands $(VAR) only when 
the container
+          # starts, so waiting for readiness is what proves the JVM accepted 
the options it was
+          # actually handed. Whether a bundled element makes the JVM exit 
depends on the leading
+          # option, so this wait complements rather than replaces the argv 
assertions below.
+          echo "Waiting for the operator pod to become ready..."
+          kubectl wait --for=condition=Ready pod -l "$SELECTOR" -n {{ 
.Release.Namespace }} --timeout=180s
+
+          POD_JSON=$(kubectl get pods -l "$SELECTOR" -n {{ .Release.Namespace 
}} -o json)
+          CONTAINER=$(echo "$POD_JSON" | jq '.items[0].spec.containers[] | 
select(.name == "{{ .Chart.Name }}")')
+
+          # jq splits jvmArgs here instead of reusing the chart's splitList so 
that the
+          # expectation is derived independently of the template under test.
+          JVM_ARGS=$(echo "$CONTAINER" | jq -c --argjson jvmArgs '{{ toJson 
.Values.operatorDeployment.operatorPod.operatorContainer.jvmArgs }}' \
+            '[$jvmArgs | splits("[ \t]+") | select(length > 0)]')
+
+          echo "Validating the fixed leading arguments..."
+          HEAD=$(echo "$CONTAINER" | jq -c '.command[0:4]')
+          
EXPECTED_HEAD='["java","-cp","./spark-kubernetes-operator.jar","$(LOG_CONFIG)"]'
+          if [ "$HEAD" != "$EXPECTED_HEAD" ]; then
+            echo "ERROR: Expected command to start with $EXPECTED_HEAD, Got: 
$HEAD"
+            exit 1
+          fi
+
+          echo "Validating the main class..."
+          MAIN_CLASS=$(echo "$CONTAINER" | jq -r '.command[-1]')
+          if [ "$MAIN_CLASS" != "org.apache.spark.k8s.operator.SparkOperator" 
]; then
+            echo "ERROR: Expected the last element to be the operator main 
class, Got: $MAIN_CLASS"
+            exit 1
+          fi
+
+          # Covers splitting, ordering, whitespace collapse and metacharacter 
preservation:
+          # re-splitting jvmArgs must reproduce the argv slice exactly.
+          echo "Validating that each jvmArg is its own argv element..."
+          ACTUAL_JVM_ARGS=$(echo "$CONTAINER" | jq -c '.command[4:-1]')
+          if [ "$ACTUAL_JVM_ARGS" != "$JVM_ARGS" ]; then
+            echo "ERROR: Expected jvmArgs elements $JVM_ARGS, Got: 
$ACTUAL_JVM_ARGS"
+            exit 1
+          fi
+
+          echo "Validating that no element bundles multiple options..."
+          BUNDLED=$(echo "$CONTAINER" | jq -c '[.command[] | select(test("[ 
\t]"))]')
+          if [ "$BUNDLED" != "[]" ]; then
+            echo "ERROR: Expected no command element to contain whitespace, 
Got: $BUNDLED"
+            exit 1
+          fi
+
+          # An empty element would make the JVM fail to start, so spaces must 
not produce one.
+          echo "Validating that no element is empty..."
+          EMPTY_COUNT=$(echo "$CONTAINER" | jq '[.command[] | select(length == 
0)] | length')
+          if [ "$EMPTY_COUNT" != "0" ]; then
+            echo "ERROR: Expected no empty command element, Got: $EMPTY_COUNT"
+            exit 1
+          fi
+
+          echo "Validating that jvmArgs is not passed as an unsplit 
\$(OPERATOR_JAVA_OPTS)..."
+          PLACEHOLDER=$(echo "$CONTAINER" | jq '.command | 
index("$(OPERATOR_JAVA_OPTS)")')
+          if [ "$PLACEHOLDER" != "null" ]; then
+            echo "ERROR: Found \$(OPERATOR_JAVA_OPTS) in the command at index 
$PLACEHOLDER"
+            exit 1
+          fi
+
+          # Documented in docs/operations.md, so it stays exported though the 
command no longer
+          # uses it. Both sides are trimmed because the env value is rendered 
as a plain YAML
+          # scalar, which drops surrounding whitespace before the API server 
ever sees it.
+          echo "Validating that OPERATOR_JAVA_OPTS is still exported..."
+          EXPECTED_OPTS=$(jq -rn --argjson jvmArgs '{{ toJson 
.Values.operatorDeployment.operatorPod.operatorContainer.jvmArgs }}' \
+            '$jvmArgs | sub("^[ \t]+"; "") | sub("[ \t]+$"; "")')
+          ACTUAL_OPTS=$(echo "$CONTAINER" | jq -r '[.env[] | select(.name == 
"OPERATOR_JAVA_OPTS")] | if length == 1 then (.[0].value // "") else 
"<missing>" end')
+          if [ "$ACTUAL_OPTS" != "$EXPECTED_OPTS" ]; then
+            echo "ERROR: Expected OPERATOR_JAVA_OPTS to carry the full jvmArgs 
string '$EXPECTED_OPTS', Got: '$ACTUAL_OPTS'"
+            exit 1
+          fi
+
+          echo "All operator container command tests passed!"
+  restartPolicy: Never
+  serviceAccountName: {{ .Values.operatorRbac.serviceAccount.name }}
diff --git a/tests/e2e/helm/helm-test-values/operator-command/values.yaml 
b/tests/e2e/helm/helm-test-values/operator-command/values.yaml
new file mode 100644
index 0000000..12b5e19
--- /dev/null
+++ b/tests/e2e/helm/helm-test-values/operator-command/values.yaml
@@ -0,0 +1,35 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Test values for validating how jvmArgs is rendered into the operator 
container command
+#
+# Usage:
+#   helm install spark-operator . -f 
tests/e2e/helm/helm-test-values/operator-command/values.yaml
+#   helm test spark-operator
+#
+# jvmArgs is quoted and deliberately awkward: surrounding and repeated 
separators must not
+# produce empty argv elements, and the '#' / '{' / ':' metacharacters must 
survive rendering.
+# Every option stays valid on its own so that the operator pod still starts.
+#
+# Keep '-XX:+UseParallelGC' first. If the options were ever bundled back into 
a single argv
+# element, the JVM would read the whole element as one option and reject it, 
so the pod fails
+# to start and the readiness wait in the test fails too. A leading 
'-Dkey=value' would instead
+# be accepted as one oversized property definition and the operator would come 
up silently
+# misconfigured, leaving only the argv assertions to catch the regression.
+
+operatorDeployment:
+  operatorPod:
+    operatorContainer:
+      jvmArgs: "  -XX:+UseParallelGC   -Dfile.encoding=UTF8 -Dtest.comment=#no 
-Dtest.map={a:b} -Dtest.quoted=yes  "


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to