Peter Toth created SPARK-59691:
----------------------------------
Summary: Report a failed suspend check of a SparkApplication or
SparkCluster
Key: SPARK-59691
URL: https://issues.apache.org/jira/browse/SPARK-59691
Project: Spark
Issue Type: Sub-task
Components: Kubernetes
Affects Versions: kubernetes-operator-1.1.0
Reporter: Peter Toth
*This is a follow-up of SPARK-59680, raised while reviewing
https://github.com/apache/spark-kubernetes-operator/pull/857.*
SPARK-59680 makes both init steps stop reading a failed lookup of the driver
pod or the master {{StatefulSet}} as "never requested". When the lookup fails,
the suspend branch returns {{completeAndDefaultRequeue()}} without going
through {{SuspendUtils.holdForSuspend}}, so that no {{SuspendHeld}} event
claims that no driver or master was requested.
That is right, but it publishes nothing at all, so a *persistent* failure
leaves the resource with no signal:
* a suspended resource never gets its initial {{Submitted}} status persisted,
which {{docs/spark_custom_resources.md}} states outright, so {{kubectl get}}
shows an empty {{Current State}};
* the branch returns before the event, so {{kubectl describe}} shows nothing
either;
* the reconciliation loops at the default interval indefinitely, because the
operator cannot decide whether the driver or master is live.
Silence is the right answer for a transport-level failure, for the same reason
the Kueue admission branch gives: writing an event would only add load to an
API server that is often the cause. It is the persistent half that is
asymmetric, since the admission branch a few lines down classifies the
identical exception with {{ReconcilerUtils.isTransientError}} and reports it as
{{KueueAdmissionRequestFailed}}.
{{KueueAdmissionRequestFailed}} does not fit here, since no admission is
involved, so this needs its own {{Warning}} reason:
* a {{REASON_SUSPEND_CHECK_FAILED}} constant in {{EventUtils}};
* the classified publish in both suspend branches, {{AppInitStep}} (which came
in with SPARK-59677) and {{ClusterInitStep}} (SPARK-59680);
* a row for the reason in the {{Warning}} table of {{docs/configuration.md}}.
The reason is user-facing on its own, which is why it is a separate change
rather than part of SPARK-59680.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
