This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 1ca60092 [SPARK-60008] Enable K8s events by default
1ca60092 is described below
commit 1ca60092e30dba0ed2fc92ab2d6da27a4297667c
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Oct 6 15:20:43 2026 -0700
[SPARK-60008] Enable K8s events by default
### What changes were proposed in this pull request?
This PR aims to enable K8s events by default.
### Why are the changes needed?
Currently, users see no K8s events unless the operator administrator
enables them, while other operators like Kubeflow Spark Operator, Flink
Kubernetes Operator and Argo Workflows publish events by default. The known
downsides of enabling them were addressed by SPARK-59926 and SPARK-59931.
### Does this PR introduce _any_ user-facing change?
Yes. Unlike
[1.0.0](https://github.com/apache/spark-kubernetes-operator/releases/tag/1.0.0)
(2026-07-23), the operator publishes K8s `Event` objects into the namespaces of
`SparkApplication` and `SparkCluster` resources by default. To restore the
previous behavior, set `spark.kubernetes.operator.events.enabled` to `false`.
### How was this patch tested?
Pass the CIs with the updated test cases.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #941 from dongjoon-hyun/SPARK-60008.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/config_properties.md | 2 +-
docs/configuration.md | 8 ++++--
docs/migration_guide.md | 10 +++++++
docs/spark_custom_resources.md | 10 +++----
.../k8s/operator/config/SparkOperatorConf.java | 6 ++--
.../operator/utils/ConfigurableEventRecorder.java | 4 +--
.../ConfigurableEventRecorderDefaultSinkTest.java | 32 ++++++++++++++++++++++
.../utils/ConfigurableEventRecorderTest.java | 17 +++++++-----
tests/e2e/events/chainsaw-test.yaml | 5 ++--
tests/e2e/helm/events-config-values.yaml | 1 -
tests/e2e/helm/kueue-config-values.yaml | 4 ---
tests/e2e/suspend-events/chainsaw-test.yaml | 4 +--
12 files changed, 73 insertions(+), 30 deletions(-)
diff --git a/docs/config_properties.md b/docs/config_properties.md
index 83f920f9..60153a86 100644
--- a/docs/config_properties.md
+++ b/docs/config_properties.md
@@ -16,7 +16,7 @@
| spark.kubernetes.operator.dynamicConfig.reloadIntervalSeconds | Long | 60 |
false | Interval (in seconds) at which the dynamic config file is re-read. Used
by the 'file' source. |
| spark.kubernetes.operator.dynamicConfig.selector | String |
app.kubernetes.io/name=spark-kubernetes-operator,app.kubernetes.io/component=operator-dynamic-config-overrides
| false | The selector str applied to dynamic config map. Used by the
'configMap' source. |
| spark.kubernetes.operator.dynamicConfig.source | String | configMap | false
| Source of dynamic config overrides when
spark.kubernetes.operator.dynamicConfig.enabled is true. Supported values:
'configMap' (default) watches a ConfigMap via a Kubernetes informer and
requires RBAC to read ConfigMaps; 'file' periodically reloads a properties file
mounted from a ConfigMap and requires no extra RBAC. |
- | spark.kubernetes.operator.events.enabled | Boolean | false | true | When
enabled, operator would publish Kubernetes Event objects into the namespace of
the Spark resource they describe, making them visible to users of that
namespace via 'kubectl describe' and 'kubectl get events'. Events are published
when a resource transitions into a new state, as Warning for failure states and
Normal for most other states, and on other occasions, e.g. when an operation on
a resource fails, or while [...]
+ | spark.kubernetes.operator.events.enabled | Boolean | true | true | When
enabled, operator would publish Kubernetes Event objects into the namespace of
the Spark resource they describe, making them visible to users of that
namespace via 'kubectl describe' and 'kubectl get events'. Events are published
when a resource transitions into a new state, as Warning for failure states and
Normal for most other states, and on other occasions, e.g. when an operation on
a resource fails, or while [...]
| spark.kubernetes.operator.events.excludedReasons | String | | true |
Comma-separated list of Java regular expressions of Kubernetes Event reasons
that operator would not publish even when
'spark.kubernetes.operator.events.enabled' is true, e.g.
'RunningHealthy,RunningWithPartialCapacity' or 'Running.*'. Each expression
must match the whole reason and is case-sensitive. An invalid expression
matches only the reason identical to it. If empty, operator would publish
events of all reasons. |
| spark.kubernetes.operator.events.minIntervalSeconds | Long | 300 | true |
Minimum interval (in seconds) between two Kubernetes Events with the same
reason and the same message on the same Spark resource. The first event of a
reason on a resource is always published right away, only a repeat that says
exactly what the last one said is dropped. Such a repeat only bumps the 'count'
of the one Event object, so it carries no information while it still costs a
read and a write on the API se [...]
| spark.kubernetes.operator.health.probePort | Integer | 19091 | false | The
port used for health/readiness check probe status. |
diff --git a/docs/configuration.md b/docs/configuration.md
index a67ee0fc..591262c9 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -57,9 +57,11 @@ operatorConfiguration:
## Kubernetes Events
-When `spark.kubernetes.operator.events.enabled` is `true`, the operator
publishes Kubernetes
-`Event` objects about `SparkApplication` and `SparkCluster` resources into
their namespaces. They
-are visible via `kubectl describe` and `kubectl get events`.
+By default, the operator publishes Kubernetes `Event` objects about
`SparkApplication` and
+`SparkCluster` resources into their namespaces. They are visible via `kubectl
describe` and
+`kubectl get events`. To publish none, set
`spark.kubernetes.operator.events.enabled` to `false`.
+This option supports dynamic override. To skip only some reasons, use
+`spark.kubernetes.operator.events.excludedReasons` instead, as described below.
Publishing them requires the `get`, `create` and `patch` permissions on
`events` in the core API
group in each namespace of the Spark resources. Without them, the operator
logs a
diff --git a/docs/migration_guide.md b/docs/migration_guide.md
index 7175caa0..5a72d48f 100644
--- a/docs/migration_guide.md
+++ b/docs/migration_guide.md
@@ -122,6 +122,16 @@ affect building the project, CI, tests or examples.
`NullPointerException` for most properties. The operator logs each invalid
value once as a
warning, instead of as an error with a stack trace whenever it reads the
value. Use `true` or
`false`, which behave the same in both versions
([SPARK-XXXXX](https://issues.apache.org/jira/browse/SPARK-XXXXX)).
+- Since 1.1.0, the operator publishes Kubernetes `Event` objects about
`SparkApplication` and
+ `SparkCluster` resources into their namespaces, e.g. whenever a resource
transitions into a new
+ state, since `spark.kubernetes.operator.events.enabled` is `true` by
default. 1.0 published
+ none. Publishing them requires the `get`, `create` and `patch` permissions
on `events` in the
+ core API group, which the Helm chart grants. Without them, e.g. if you
manage the RBAC of the
+ operator yourself, the operator logs a `Could not record ... event` warning
for each event. An
+ event exporter may also forward the `Warning` events, such as `Failed`, to
chat or alerting
+ services outside the cluster. To restore the behavior before 1.1.0, set
+ `spark.kubernetes.operator.events.enabled` to `false`. See
+ [Kubernetes Events](configuration.md#kubernetes-events) for details.
### SparkApplication
diff --git a/docs/spark_custom_resources.md b/docs/spark_custom_resources.md
index 0ddd26c8..673973b8 100644
--- a/docs/spark_custom_resources.md
+++ b/docs/spark_custom_resources.md
@@ -540,7 +540,7 @@ running `SparkCluster` is stopped instead, as described
below.
`suspend: true`, the initial `Submitted` status is not persisted to the API
server, so
`kubectl get` shows an empty `Current State` and no state transition events
are published until
initialization resumes. Instead, the `SuspendHeld`
[event](configuration.md#kubernetes-events) is
-published when enabled. Since the status is not there to fall back on, it is
republished every 30
+published by default. Since the status is not there to fall back on, it is
republished every 30
minutes by default while the hold lasts, so that it outlives the event
retention of the API
server. An application held later, in `ScheduledToRestart`, keeps the status
its previous attempt
already wrote and gets the same event, since that status says that a restart
is due, not that the
@@ -569,7 +569,7 @@ spec:
HorizontalPodAutoscaler and PodDisruptionBudget of its workers if any. Its
Services and
NetworkPolicy are kept. Any application running on the cluster is terminated
with it, since there
is no graceful decommission. A failure to delete them, other than one which
may clear on its own,
- publishes the `SuspendReleaseFailed`
[event](configuration.md#kubernetes-events) when enabled.
+ publishes the `SuspendReleaseFailed`
[event](configuration.md#kubernetes-events) by default.
* Setting it back to `false` moves the cluster to `Submitted` once its master
and worker pods are
gone and its Kueue `Workload`, if any, is released, and the master and
workers are requested
again from scratch, like a newly created cluster. Only pods labeled with the
`master` or `worker`
@@ -624,7 +624,7 @@ spec:
label is ignored, so the resource is not queued and starts right away, and
the operator does
not access any Kueue resource, like Kueue ignores a job whose integration is
not enabled. Since
the author of the resource may not see the operator configuration, the
`KueueDisabled` warning
- [event](configuration.md#kubernetes-events) is published when enabled.
+ [event](configuration.md#kubernetes-events) is published by default.
* The `Workload` is named `<lower-cased kind>-<resource name>` and is owned by
the Spark resource,
so it is garbage collected along with it.
* Like Kueue built-in integrations, the `Workload` gets the priority of the
`WorkloadPriorityClass`
@@ -641,7 +641,7 @@ spec:
Like `spec.suspend`, the initial `Submitted` status of the first attempt is
not persisted to the
API server, so `kubectl get` shows an empty `Current State` and no state
transition events are
published until the `Workload` is admitted. Instead, the
`KueueAdmissionPending` and
- `KueueAdmitted` [events](configuration.md#kubernetes-events) are published
when enabled. Since
+ `KueueAdmitted` [events](configuration.md#kubernetes-events) are published
by default. Since
the status is not there to fall back on, the pending event is republished
while the `Workload`
waits, so that it outlives the event retention of the API server. Use
`kubectl get workload` to
see the admission status. If the spec changes while waiting, the
@@ -787,7 +787,7 @@ spec:
`spark.executor.instances`, or a deactivation. The operator checks the
admission only before it
creates the resources, so after a later eviction the driver and executors
keep running, and
until the attempt stops, the `KueueEvictionIgnored`
[event](configuration.md#kubernetes-events)
- is published instead when enabled. An evicted `Workload` keeps its Kueue
quota until the driver
+ is published instead by default. An evicted `Workload` keeps its Kueue quota
until the driver
and executors are released, so a workload which preempts the application
waits until then,
while a deactivated one no longer counts against it. Since `spec.suspend`
does not stop a
running attempt, delete the application to release its quota earlier. With
`blockAdmission`, an
diff --git
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java
index d53e8766..39e2ccf5 100644
---
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java
+++
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java
@@ -337,9 +337,11 @@ public final class SparkOperatorConf {
+ "spec.suspend or Kueue. See the 'Kubernetes Events'
section of "
+ "docs/configuration.md for all reasons and the required
RBAC permissions. "
+ "These are Event resources in the core "
- + "API group, unrelated to the internal events that trigger
reconciliation.")
+ + "API group, unrelated to the internal events that trigger
reconciliation. "
+ + "If false, operator would publish no event. To skip only
some reasons, use "
+ + "'spark.kubernetes.operator.events.excludedReasons'
instead.")
.typeParameterClass(Boolean.class)
- .defaultValue(false)
+ .defaultValue(true)
.build();
/**
diff --git
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/ConfigurableEventRecorder.java
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/ConfigurableEventRecorder.java
index 5170a53b..ec014d38 100644
---
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/ConfigurableEventRecorder.java
+++
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/ConfigurableEventRecorder.java
@@ -47,8 +47,8 @@ import lombok.extern.slf4j.Slf4j;
import org.apache.spark.k8s.operator.client.KubernetesClientFactory;
/**
- * An {@link EventRecorder} that drops every event unless {@link
- *
org.apache.spark.k8s.operator.config.SparkOperatorConf#KUBERNETES_EVENTS_ENABLED}
is set, drops
+ * An {@link EventRecorder} that drops every event while {@link
+ *
org.apache.spark.k8s.operator.config.SparkOperatorConf#KUBERNETES_EVENTS_ENABLED}
is false, drops
* events whose reason matches one of the patterns in {@link
*
org.apache.spark.k8s.operator.config.SparkOperatorConf#KUBERNETES_EVENTS_EXCLUDED_REASONS},
and
* drops an event repeated on the same resource with the same reason and the
same message within
diff --git
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/utils/ConfigurableEventRecorderDefaultSinkTest.java
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/utils/ConfigurableEventRecorderDefaultSinkTest.java
index 2fc6d1c0..c63b45b8 100644
---
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/utils/ConfigurableEventRecorderDefaultSinkTest.java
+++
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/utils/ConfigurableEventRecorderDefaultSinkTest.java
@@ -19,10 +19,12 @@
package org.apache.spark.k8s.operator.utils;
+import static java.net.HttpURLConnection.HTTP_FORBIDDEN;
import static java.net.HttpURLConnection.HTTP_UNAVAILABLE;
import static org.apache.spark.k8s.operator.Constants.HTTP_TOO_MANY_REQUESTS;
import static
org.apache.spark.k8s.operator.config.SparkOperatorConf.KUBERNETES_EVENTS_ENABLED;
import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatCode;
import static org.mockito.Mockito.doReturn;
import static org.mockito.Mockito.mock;
@@ -38,6 +40,7 @@ import
io.javaoperatorsdk.operator.api.config.ControllerConfiguration;
import io.javaoperatorsdk.operator.api.event.EventRecord;
import io.javaoperatorsdk.operator.api.reconciler.Context;
import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
@@ -79,6 +82,35 @@ class ConfigurableEventRecorderDefaultSinkTest {
.containsExactly("published");
}
+ @Test
+ void makesNoRequestWhileDisabled() {
+
SparkOperatorConfManager.INSTANCE.refresh(Map.of(KUBERNETES_EVENTS_ENABLED.getKey(),
"false"));
+ ConfigurableEventRecorder recorder =
ConfigurableEventRecorder.withDefaultSink(client);
+
+ recorder.record(EventRecord.warning("ReconcileError", "dropped"),
appContext());
+
+ assertThat(server.getRequestCount()).isZero();
+ }
+
+ @Test
+ void dropsAForbiddenWriteWithoutThrowing() {
+ // Without the RBAC permission to create events, every write is rejected,
which must not throw
+ // out of the reconciliation that records the event.
+
SparkOperatorConfManager.INSTANCE.refresh(Map.of(KUBERNETES_EVENTS_ENABLED.getKey(),
"true"));
+ ConfigurableEventRecorder recorder =
ConfigurableEventRecorder.withDefaultSink(client);
+ server
+ .expect()
+ .post()
+ .withPath("/api/v1/namespaces/ns-1/events")
+ .andReturn(HTTP_FORBIDDEN, null)
+ .always();
+
+ assertThatCode(
+ () -> recorder.record(EventRecord.warning("ReconcileError",
"dropped"), appContext()))
+ .doesNotThrowAnyException();
+
assertThat(client.v1().events().inNamespace("ns-1").list().getItems()).isEmpty();
+ }
+
/** Returns a context from which the JOSDK recorder can build an Event about
app-1 in ns-1. */
private static Context<?> appContext() {
SparkApplication app = new SparkApplication();
diff --git
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/utils/ConfigurableEventRecorderTest.java
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/utils/ConfigurableEventRecorderTest.java
index 445d556a..b9b3fcb0 100644
---
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/utils/ConfigurableEventRecorderTest.java
+++
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/utils/ConfigurableEventRecorderTest.java
@@ -98,15 +98,16 @@ class ConfigurableEventRecorderTest {
@Test
void dropsEventsWhileDisabled() {
- // The option defaults to false, so an operator that never opts in writes
nothing.
+ setEventsEnabled(false);
+
recorder.record(EventRecord.warning("ReconcileError", "boom"), context);
verifyNoInteractions(delegate);
}
@Test
- void forwardsEventsWhileEnabled() {
- setEventsEnabled(true);
+ void forwardsEventsByDefault() {
+ // The option defaults to true, so an operator that never configures it
publishes events.
EventRecord event = EventRecord.warning("ReconcileError", "boom");
recorder.record(event, context);
@@ -118,6 +119,7 @@ class ConfigurableEventRecorderTest {
void boundRecorderIsGatedOnTheFlagRatherThanOnBindingTime() {
// Bound while disabled, then enabled mid reconciliation: the dynamic
override has to take
// effect, which it only does if the flag is read per event and not when
forContext is called.
+ setEventsEnabled(false);
var bound = recorder.forContext(context);
bound.warn("ReconcileError", "dropped");
verifyNoInteractions(delegate);
@@ -144,15 +146,16 @@ class ConfigurableEventRecorderTest {
}
@Test
- void treatsAMalformedOverrideAsDisabled() {
- // A malformed override falls back to the default value, which is
disabled. Publishing an
+ void treatsAMalformedOverrideAsEnabled() {
+ // A malformed override falls back to the default value, which is enabled.
Publishing an
// event must never be the thing that throws out of a reconciliation.
SparkOperatorConfManager.INSTANCE.refresh(
Map.of(KUBERNETES_EVENTS_ENABLED.getKey(), "null"));
+ EventRecord event = EventRecord.warning("ReconcileError", "boom");
- recorder.record(EventRecord.warning("ReconcileError", "boom"), context);
+ recorder.record(event, context);
- verifyNoInteractions(delegate);
+ verify(delegate).record(event, context);
}
@Test
diff --git a/tests/e2e/events/chainsaw-test.yaml
b/tests/e2e/events/chainsaw-test.yaml
index 790fadd5..aba2985e 100644
--- a/tests/e2e/events/chainsaw-test.yaml
+++ b/tests/e2e/events/chainsaw-test.yaml
@@ -18,9 +18,8 @@
# Asserts the events published by the state transitions of a SparkApplication
which succeeds, a
# SparkApplication which fails, and a SparkCluster.
#
-# Like the `suspend-events` suite, this suite requires an operator installed
with
-# `spark.kubernetes.operator.events.enabled=true`, as the `events` CI mode
does with
-# tests/e2e/helm/events-config-values.yaml. Run it locally with:
+# Like the `suspend-events` suite, this suite runs in the `events` CI mode,
which installs the
+# operator with tests/e2e/helm/events-config-values.yaml. Run it locally with:
#
# helm install spark --create-namespace \
# -f build-tools/helm/spark-kubernetes-operator/values.yaml \
diff --git a/tests/e2e/helm/events-config-values.yaml
b/tests/e2e/helm/events-config-values.yaml
index 141b2bd3..f92f1705 100644
--- a/tests/e2e/helm/events-config-values.yaml
+++ b/tests/e2e/helm/events-config-values.yaml
@@ -15,7 +15,6 @@
operatorConfiguration:
spark-operator.properties: |+
- spark.kubernetes.operator.events.enabled=true
# The suspend-events suite asserts that a held resource republishes its
SuspendHeld event.
# The default is 1800 seconds, which no assert can wait for.
spark.kubernetes.operator.reconciler.suspendHoldRequeueIntervalSeconds=10
diff --git a/tests/e2e/helm/kueue-config-values.yaml
b/tests/e2e/helm/kueue-config-values.yaml
index 6fd65be8..604e4957 100644
--- a/tests/e2e/helm/kueue-config-values.yaml
+++ b/tests/e2e/helm/kueue-config-values.yaml
@@ -26,7 +26,3 @@ operatorDeployment:
operatorRbac:
kueue:
enabled: true
-
-operatorConfiguration:
- spark-operator.properties: |+
- spark.kubernetes.operator.events.enabled=true
diff --git a/tests/e2e/suspend-events/chainsaw-test.yaml
b/tests/e2e/suspend-events/chainsaw-test.yaml
index f9176b38..e73db445 100644
--- a/tests/e2e/suspend-events/chainsaw-test.yaml
+++ b/tests/e2e/suspend-events/chainsaw-test.yaml
@@ -18,8 +18,8 @@
# Asserts the `SuspendHeld` event of a held SparkApplication and SparkCluster.
#
# Unlike the `suspend` suite, which covers the hold itself with the shipped
defaults, this suite
-# requires an operator installed with
`spark.kubernetes.operator.events.enabled=true`, as the
-# `events` CI mode does with tests/e2e/helm/events-config-values.yaml. Run it
locally with:
+# requires an operator installed with
tests/e2e/helm/events-config-values.yaml, as the `events`
+# CI mode does. Run it locally with:
#
# helm install spark --create-namespace \
# -f build-tools/helm/spark-kubernetes-operator/values.yaml \
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
