This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 0b3e3822 [SPARK-60015] Remove JIRA IDs from `migration_guide.md`
0b3e3822 is described below
commit 0b3e3822213cf29984761bcfd84a570b137b27ec
Author: ykisana <[email protected]>
AuthorDate: Tue Oct 6 20:40:04 2026 -0700
[SPARK-60015] Remove JIRA IDs from `migration_guide.md`
### What changes were proposed in this pull request?
This PR removes all JIRA IDs from `docs/migration_guide.md`, i.e. the
trailing `([SPARK-NNNNN](https://issues.apache.org/jira/browse/SPARK-NNNNN),
...)` of each item, including the `SPARK-XXXXX` placeholder. An item whose JIRA
IDs were on their own line now ends on the previous line. The text of the items
is unchanged.
### Why are the changes needed?
As discussed in #943, the migration guide should describe the behavior
changes without JIRA IDs. This also removes the invalid `SPARK-XXXXX` link,
which #943 was going to fix.
### Does this PR introduce _any_ user-facing change?
No behavior change. The migration guide items no longer link to JIRA
issues. All of them are about the unreleased 1.1.0.
### How was this patch tested?
Documentation-only change, so no tests were added. Confirmed that no JIRA
reference remains and that the diff only removes the JIRA IDs:
```bash
$ grep -nE 'SPARK-|issues\.apache\.org' docs/migration_guide.md
$ git diff --word-diff HEAD~1 -- docs/migration_guide.md
```
### Was this patch authored or co-authored using generative AI tooling?
Yes.
Generated-by: Claude Opus 5.5
Closes #944 from ykisana/SPARK-60015.
Authored-by: ykisana <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/migration_guide.md | 51 +++++++++++++++++++++++--------------------------
1 file changed, 24 insertions(+), 27 deletions(-)
diff --git a/docs/migration_guide.md b/docs/migration_guide.md
index 5a72d48f..8a2b0c76 100644
--- a/docs/migration_guide.md
+++ b/docs/migration_guide.md
@@ -32,12 +32,12 @@ affect building the project, CI, tests or examples.
- Since 1.1.0, `spark.apache.org/v1beta1` of the CRDs is deprecated and
scheduled for removal in
2.0.0. Its schema is no longer updated, e.g. it lacks `spec.suspend`. Use
`spark.apache.org/v1`.
-- Since 1.1.0, K8s 1.35 or newer is recommended instead of 1.34
([SPARK-59692](https://issues.apache.org/jira/browse/SPARK-59692)).
+- Since 1.1.0, K8s 1.35 or newer is recommended instead of 1.34.
- Since 1.1.0, the CRDs have `spec.suspend` for `SparkApplication` and
`SparkCluster`, and the
`Suspended` state for `SparkCluster`. Since `helm upgrade` does not update
CRDs, replace them
before upgrading the operator, e.g. with the CRDs of the Helm chart `1.9.0`,
which deploys
1.1.0. Otherwise, the API server rejects or drops `spec.suspend`, and
rejects the `Suspended`
- state ([SPARK-59475](https://issues.apache.org/jira/browse/SPARK-59475),
[SPARK-59750](https://issues.apache.org/jira/browse/SPARK-59750)).
+ state.
```bash
helm repo update
@@ -51,20 +51,19 @@ affect building the project, CI, tests or examples.
chart with the default values of chart `1.8.0`, which lack
`operatorDeployment.networkPolicy.enabled`,
`operatorConfiguration.dynamicConfig.enabled` and
`operatorRbac.kueue` that the chart schema now requires. Use
`--reset-then-reuse-values` of Helm
- 3.14 or newer, or pass the values file again with `-f`
([SPARK-59504](https://issues.apache.org/jira/browse/SPARK-59504),
[SPARK-59519](https://issues.apache.org/jira/browse/SPARK-59519)).
+ 3.14 or newer, or pass the values file again with `-f`.
- Since 1.1.0, `operatorDeployment.networkPolicy.enable` and
`operatorConfiguration.dynamicConfig.enable` are deprecated in favor of
`enabled`, and
`helm install` and `helm upgrade` print a warning for them. They are still
honored until chart
`2.0.0`: a feature is enabled when either key is `true`, so a legacy
`enable: true` must be
- removed to disable it
([SPARK-59504](https://issues.apache.org/jira/browse/SPARK-59504),
[SPARK-59540](https://issues.apache.org/jira/browse/SPARK-59540)).
+ removed to disable it.
- Since 1.1.0, `operatorRbac.configManagement.create: false` takes effect. The
chart then no longer
creates the Role and RoleBinding for the `configMap` source of the dynamic
config, which 1.0
- created regardless. To restore the behavior before 1.1.0, set it to `true`,
the default
- ([SPARK-59537](https://issues.apache.org/jira/browse/SPARK-59537)).
+ created regardless. To restore the behavior before 1.1.0, set it to `true`,
the default.
- Since 1.1.0, the operator ClusterRole and Roles grant only `get`, `create`
and `patch` on
`events`, which are the verbs the operator uses. 1.0 also granted `list`,
`watch`, `update` and
`delete`. If other workloads use the operator ServiceAccount and need them,
grant them with
- another Role or ClusterRole
([SPARK-59587](https://issues.apache.org/jira/browse/SPARK-59587)).
+ another Role or ClusterRole.
- Since 1.1.0, the workload ClusterRole and Roles also grant `get`, `create`
and `delete` on
`deployments` in the `apps` API group, and `get` and `update` on the `scale`
subresource of
`statefulsets` and `deployments`. Spark drivers need them for
@@ -92,7 +91,7 @@ affect building the project, CI, tests or examples.
operator container, since the percentages apply to the node memory
otherwise. The options also
take precedence over `JAVA_TOOL_OPTIONS` and `JDK_JAVA_OPTIONS` in
`operatorContainer.env`, and
cannot contain a space, since the value is split on spaces. To restore the
behavior before
- 1.1.0, set `jvmArgs` to `-Dfile.encoding=UTF8`
([SPARK-58426](https://issues.apache.org/jira/browse/SPARK-58426)).
+ 1.1.0, set `jvmArgs` to `-Dfile.encoding=UTF8`.
- Since 1.1.0, `operatorConfiguration.append` is removed, and the chart no
longer reads
`conf/log4j2.properties` and `conf/spark-operator.properties` in the chart
directory, which it
has never included. A values file which still sets `append` keeps working,
since the key is
@@ -106,14 +105,14 @@ affect building the project, CI, tests or examples.
which is the default, leaves the interval between retries on reconciliation
errors unlimited as
documented. 1.0 capped it at about 15 seconds, so with the other defaults,
the retries now span
about 49 minutes instead of about 3 minutes. To restore the behavior before
1.1.0, set it to
- `15` ([SPARK-59892](https://issues.apache.org/jira/browse/SPARK-59892)).
+ `15`.
- Since 1.1.0, the `_sum` of the Prometheus summaries of the operator
histograms and timers, e.g.
`kubernetes_client_http_response_latency_seconds_sum`, is the sum of all
recorded values. 1.0
exported the mean of roughly the last 5 minutes multiplied by the count
instead, which can
decrease between scrapes, so `rate()` of it and the average derived from it
were wrong. Also, the
`0.5` and `0.999` quantiles of the `operator_sdk_*` histograms are the
median and the 99.9th
percentile instead of the mean and the 99th percentile. A histogram or timer
of a custom metrics
- source has no `_sum` unless it is a `SummingHistogram` or a `SummingTimer`
([SPARK-59935](https://issues.apache.org/jira/browse/SPARK-59935)).
+ source has no `_sum` unless it is a `SummingHistogram` or a `SummingTimer`.
- Since 1.1.0, a Boolean configuration property accepts `true` or `false` in
any case, ignoring
surrounding whitespace, like Apache Spark. Any other value, e.g. `0`, `1`,
`yes` or `off`, is
ignored, and the default value is used. 1.0 ignored a value like `False` or
`TRUE` and used the
@@ -121,7 +120,7 @@ affect building the project, CI, tests or examples.
falls back to the default value for a property of any type, while 1.0 failed
with a
`NullPointerException` for most properties. The operator logs each invalid
value once as a
warning, instead of as an error with a stack trace whenever it reads the
value. Use `true` or
- `false`, which behave the same in both versions
([SPARK-XXXXX](https://issues.apache.org/jira/browse/SPARK-XXXXX)).
+ `false`, which behave the same in both versions.
- Since 1.1.0, the operator publishes Kubernetes `Event` objects about
`SparkApplication` and
`SparkCluster` resources into their namespaces, e.g. whenever a resource
transitions into a new
state, since `spark.kubernetes.operator.events.enabled` is `true` by
default. 1.0 published
@@ -141,27 +140,25 @@ affect building the project, CI, tests or examples.
and retried errors other than `409` and `429`, e.g. `503`, right away. A
request which the API
server never answered, e.g. on a connection reset or a timeout, is retried
with exponential
backoff too, while 1.0 failed the attempt with `SchedulingFailure` at once.
The number of
- attempts is still bounded by
`spark.kubernetes.operator.api.secondaryResourceCreateMaxAttempts`
- ([SPARK-59092](https://issues.apache.org/jira/browse/SPARK-59092),
[SPARK-59694](https://issues.apache.org/jira/browse/SPARK-59694)).
+ attempts is still bounded by
`spark.kubernetes.operator.api.secondaryResourceCreateMaxAttempts`.
- Since 1.1.0, the duration of an attempt which is compared with
`restartCounterResetMillis` is
measured from the first state after `Submitted` or `ScheduledToRestart`,
normally
`DriverRequested`, instead of from `Submitted` or `ScheduledToRestart`. The
restart backoff no
longer counts, so an attempt which reset the restart counters in 1.0 may not
reset them, and the
application may reach `maxRestartAttempts` earlier. To keep a similar
threshold, decrease
- `restartCounterResetMillis` by the restart backoff, e.g.
`restartBackoffMillis`
- ([SPARK-59542](https://issues.apache.org/jira/browse/SPARK-59542)).
+ `restartCounterResetMillis` by the restart backoff, e.g.
`restartBackoffMillis`.
- Since 1.1.0, an application which is configured to restart ends in
`ResourceReleased` instead of
`TerminatedWithoutReleaseResources` after its last attempt, even if its
`resourceRetainPolicy`
retains the resources, since they are released at the end of every attempt.
So
`resourceRetainDurationMillis` no longer applies to it, while
`ttlAfterStopMillis` still deletes
it at the same time as in 1.0. A tool which waits for
`TerminatedWithoutReleaseResources` has to
- wait for `ResourceReleased` instead
([SPARK-59732](https://issues.apache.org/jira/browse/SPARK-59732)).
+ wait for `ResourceReleased` instead.
- Since 1.1.0, an application whose driver pod fails before any of its
containers starts, e.g.
when the pod is evicted or its node is deleted, moves to `Failed`, or
`DriverEvicted` if it is
evicted, right away. 1.0 kept it in `DriverRequested` until
`DriverStartTimedOut`, or forever if
`driverStartTimeoutMillis` was disabled. Since `DriverStartTimedOut` is an
infrastructure failure
while the others are not, `restartPolicy: OnInfrastructureFailure` no longer
restarts such an
- application. Use `OnFailure` to restart it
([SPARK-59729](https://issues.apache.org/jira/browse/SPARK-59729)).
+ application. Use `OnFailure` to restart it.
### SparkCluster
@@ -171,31 +168,31 @@ is recreated, e.g. by suspending and resuming it with
[`spec.suspend`](spark_cus
- Since 1.1.0, the workers start the external shuffle service by default, so
that dynamic
allocation works out of the box. To restore the behavior before 1.1.0, set
- `spark.shuffle.service.enabled` to `false` in `spec.sparkConf`
([SPARK-58835](https://issues.apache.org/jira/browse/SPARK-58835)).
+ `spark.shuffle.service.enabled` to `false` in `spec.sparkConf`.
- Since 1.1.0, the NetworkPolicy of the workers also admits ingress from every
driver pod
(`spark-role: driver`) in the namespace, so that the driver of an
application running on the
cluster can fetch task results larger than `spark.task.maxDirectResultSize`
from the executors.
- 1.0 admitted only the pods of the cluster
([SPARK-58649](https://issues.apache.org/jira/browse/SPARK-58649)).
+ 1.0 admitted only the pods of the cluster.
- Since 1.1.0, the `terminationGracePeriodSeconds` of the master and worker
pod templates is
respected. 1.0 always overwrote it with `0`, which is still the default. To
restore the behavior
- before 1.1.0, remove it from the pod templates
([SPARK-59795](https://issues.apache.org/jira/browse/SPARK-59795)).
+ before 1.1.0, remove it from the pod templates.
- Since 1.1.0, the `dnsConfig` of the worker pod template is kept, and the
domain of the worker
service is appended to its `searches`. 1.0 replaced the whole `dnsConfig`.
To restore the
- behavior before 1.1.0, remove `dnsConfig` from the worker pod template
([SPARK-59797](https://issues.apache.org/jira/browse/SPARK-59797)).
+ behavior before 1.1.0, remove `dnsConfig` from the worker pod template.
- Since 1.1.0, a cluster whose master and worker resources cannot be requested
because of a
failure which may clear on its own, e.g. a timeout, a `429`, a `500` or a
request which the API
server never answered, stays `Submitted` and is retried at the next
reconciliation, instead of
failing with `SchedulingFailure` and then `Failed`. A request which the API
server rejected
- still fails the cluster
([SPARK-59750](https://issues.apache.org/jira/browse/SPARK-59750)).
+ still fails the cluster.
- Since 1.1.0, the message of the `Failed` state after `SchedulingFailure` is
`Cluster failed, since requesting its resources failed. See the preceding
SchedulingFailure state for the cause.`
- instead of `Cannot process cluster status.`
([SPARK-59870](https://issues.apache.org/jira/browse/SPARK-59870)).
+ instead of `Cannot process cluster status.`.
### Docker Image
- Since 1.1.0, the image is based on `alpine:3.24` instead of `alpine:3.23`,
and it does not
install `libstdc++`, which the operator does not use. An image built on top
of it which needs
- the package has to install it
([SPARK-59834](https://issues.apache.org/jira/browse/SPARK-59834),
[SPARK-59836](https://issues.apache.org/jira/browse/SPARK-59836)).
+ the package has to install it.
```dockerfile
FROM apache/spark-kubernetes-operator:1.1.0
@@ -207,10 +204,10 @@ is recreated, e.g. by suspending and resuming it with
[`spec.suspend`](spark_cus
### Java API
- Since 1.1.0, `org.apache.spark.k8s.operator.Constants` is `final` and has a
private constructor,
- so it cannot be extended or instantiated
([SPARK-59679](https://issues.apache.org/jira/browse/SPARK-59679)).
+ so it cannot be extended or instantiated.
- Since 1.1.0, `ApplicationStatus.terminateOrRestart(RestartConfig,
ResourceRetainPolicy, String, boolean)`
is deprecated for removal. It ignores `resourceRetainPolicy`, so it no
longer returns
`TerminatedWithoutReleaseResources`. Use `terminateOrRestart(RestartConfig,
String, boolean)`
- instead ([SPARK-59734](https://issues.apache.org/jira/browse/SPARK-59734)).
+ instead.
- Since 1.1.0, `ClusterStateSummary` has the `Suspended` constant, so an
exhaustive `switch` over
- it needs a case for it
([SPARK-59750](https://issues.apache.org/jira/browse/SPARK-59750)).
+ it needs a case for it.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
