This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git


The following commit(s) were added to refs/heads/main by this push:
     new 2a90397  [SPARK-58495] Use 7077 for `spark` named container port in 
master StatefulSet
2a90397 is described below

commit 2a90397993b49bfb02b52837539bde997c80b6cc
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Aug 1 23:07:18 2026 -0700

    [SPARK-58495] Use 7077 for `spark` named container port in master 
StatefulSet
    
    ### What changes were proposed in this pull request?
    
    This PR aims to use `7077` for the `spark` named container port in the 
master StatefulSet of `SparkClusterResourceSpec`, instead of `7070`.
    
    ### Why are the changes needed?
    
    The `spark` named container port declaration was inconsistent with the 
actual master RPC port. The Spark standalone master listens on `7077` (the 
default of `SPARK_MASTER_PORT`), and all other references agree on `7077`:
    
    - The master Service exposes port `7077` with `targetPort: spark` (a named 
port reference).
    - Workers connect via `spark://<name>-master-svc:7077`.
    
    Currently, this mismatch is harmless because the master Service is headless 
(`ClusterIP: None`), so workers resolve the pod IP via DNS and connect directly 
to `7077` without any Service port translation, and `containerPort` itself is a 
non-enforcing declaration. However, the stale value would surface as a real 
failure if:
    
    1. A user overrides `masterSpec.serviceSpec` with a non-headless Service 
type, where `targetPort: spark` would resolve to `7070` and break connections.
    2. Any Endpoints/EndpointSlice consumer (e.g., Prometheus service 
discovery, service mesh) reads the resolved port.
    3. A NetworkPolicy referencing the named port is added on the master side.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No, for the default headless master Service. The generated master pod spec 
now declares the correct port, which only affects configurations that resolve 
the `spark` named port.
    
    ### How was this patch tested?
    
    Pass the CIs.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Fable 5
    
    Closes #775
    
    Closes #781 from dongjoon-hyun/SPARK-58495.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../java/org/apache/spark/k8s/operator/SparkClusterResourceSpec.java    | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/spark-submission-worker/src/main/java/org/apache/spark/k8s/operator/SparkClusterResourceSpec.java
 
b/spark-submission-worker/src/main/java/org/apache/spark/k8s/operator/SparkClusterResourceSpec.java
index 8cadf00..70eb6c4 100644
--- 
a/spark-submission-worker/src/main/java/org/apache/spark/k8s/operator/SparkClusterResourceSpec.java
+++ 
b/spark-submission-worker/src/main/java/org/apache/spark/k8s/operator/SparkClusterResourceSpec.java
@@ -260,7 +260,7 @@ public class SparkClusterResourceSpec {
         .endPort()
         .addNewPort()
         .withName("spark")
-        .withContainerPort(7070)
+        .withContainerPort(7077)
         .endPort()
         .addNewPort()
         .withName("rest")


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to