[
https://issues.apache.org/jira/browse/SPARK-59606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-59606:
----------------------------------
Fix Version/s: kubernetes-operator-1.1.0
(was: kubernetes-operator-2.0.0)
> Map `SparkCluster` pods by `spark.operator/spark-cluster-name` in
> `SparkClusterReconciler`
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-59606
> URL: https://issues.apache.org/jira/browse/SPARK-59606
> Project: Spark
> Issue Type: Sub-task
> Components: Kubernetes
> Affects Versions: kubernetes-operator-1.1.0
> Reporter: Peter Toth
> Assignee: Peter Toth
> Priority: Major
> Labels: pull-request-available
> Fix For: kubernetes-operator-1.1.0
>
>
> SparkClusterReconciler.prepareEventSources registers the Pod informer with
> basicLabelSecondaryToPrimaryMapper(LABEL_SPARK_APPLICATION_NAME), i.e. it
> maps a Pod back to its primary SparkCluster by reading the label
> spark.operator/spark-app-name.
> SparkCluster pods never carry that label. SparkClusterResourceSpec adds only
> spark.operator/spark-role, spark.operator/spark-cluster-name and
> spark.operator/spark-version to the master and worker pod templates (lines
> 234-237 and 316-319), and Utils.sparkClusterResourceLabels adds
> spark.operator/spark-cluster-name plus commonManagedResourceLabels, which is
> only spark.operator/name. LABEL_SPARK_APPLICATION_NAME is set exclusively by
> Utils.sparkAppResourceLabels, for SparkApplication resources.
> So Utils.basicLabelSecondaryToPrimaryMapper takes its `name == null` branch
> and returns Set.of() for every SparkCluster pod. The pods still match the
> informer's commonResourceLabelsStr() selector and are cached, but no primary
> ResourceID is produced, so a Pod event never triggers a SparkCluster
> reconciliation. An owner-reference mapper would not help either: cluster pods
> are owned by their StatefulSet, not by the SparkCluster, which is presumably
> why a label mapper is used.
> Effect: a master or worker pod dying, becoming ready, or being evicted is not
> noticed until the next periodic reconcile,
> spark.kubernetes.operator.reconciler.intervalSeconds, 120 seconds by default.
> SparkApplication is unaffected; SparkAppReconciler uses the same mapper with
> the label its pods actually carry.
> The fix is to use LABEL_SPARK_CLUSTER_NAME in
> SparkClusterReconciler.prepareEventSources, which is what PR #836 does for
> the new Kueue Workload informer three lines below.
> Worth adding a test that asserts the mapper resolves a realistic cluster pod
> to its SparkCluster. The existing SparkClusterReconcilerTest only asserts the
> number of event sources and their resource classes, so it does not catch a
> wrong label: swapping the label in the Workload source leaves all 11 tests
> green.
> Found while reviewing
> https://github.com/apache/spark-kubernetes-operator/pull/836
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
