This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e5e751b98f9e [SPARK-48887][K8S] Enable
`spark.kubernetes.executor.checkAllContainers` by default
e5e751b98f9e is described below
commit e5e751b98f9ef5b8640079c07a9a342ef471d75d
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Jul 12 14:54:13 2024 -0700
[SPARK-48887][K8S] Enable `spark.kubernetes.executor.checkAllContainers` by
default
### What changes were proposed in this pull request?
This PR aims to enable `spark.kubernetes.executor.checkAllContainers` by
default from Apache Spark 4.0.0.
### Why are the changes needed?
Since Apache Spark 3.1.0, `spark.kubernetes.executor.checkAllContainers` is
supported and useful because [sidecar
pattern](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/)
is used in many cases. Also, it prevents user mistakes which forget and ignore
the sidecars' failures by always reporting sidecar failures via executor status.
- https://github.com/apache/spark/pull/29924
### Does this PR introduce _any_ user-facing change?
- This configuration is no-op when there is no other container.
- This will report user containers' error correctly when there exist other
containers which are provided by the users.
### How was this patch tested?
Both `true` and `false` are covered by our CI test coverage since Apache
Spark 3.1.0.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #47337 from dongjoon-hyun/SPARK-48887.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/core-migration-guide.md | 2 ++
docs/running-on-kubernetes.md | 2 +-
.../core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala | 2 +-
3 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/docs/core-migration-guide.md b/docs/core-migration-guide.md
index 26b0ff32cf5d..aeb5118175ff 100644
--- a/docs/core-migration-guide.md
+++ b/docs/core-migration-guide.md
@@ -38,6 +38,8 @@ license: |
- Since Spark 4.0, Spark uses `ReadWriteOncePod` instead of `ReadWriteOnce`
access mode in persistence volume claims. To restore the legacy behavior, you
can set `spark.kubernetes.legacy.useReadWriteOnceAccessMode` to `true`.
+- Since Spark 4.0, Spark reports its executor pod status by checking all
containers of that pod. To restore the legacy behavior, you can set
`spark.kubernetes.executor.checkAllContainers` to `false`.
+
- Since Spark 4.0, Spark uses `~/.ivy2.5.2` as Ivy user directory by default
to isolate the existing systems from Apache Ivy's incompatibility. To restore
the legacy behavior, you can set `spark.jars.ivy` to `~/.ivy2`.
- Since Spark 4.0, Spark uses the external shuffle service for deleting
shuffle blocks for deallocated executors when the shuffle is no longer needed.
To restore the legacy behavior, you can set
`spark.shuffle.service.removeShuffle` to `false`.
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index f9a4b645786b..655b30756a29 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -1327,7 +1327,7 @@ See the [configuration page](configuration.html) for
information on Spark config
</tr>
<tr>
<td><code>spark.kubernetes.executor.checkAllContainers</code></td>
- <td><code>false</code></td>
+ <td><code>true</code></td>
<td>
Specify whether executor pods should be check all containers (including
sidecars) or only the executor container when determining the pod status.
</td>
diff --git
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
index a25876de5aee..2f9ee6943fe6 100644
---
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
+++
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
@@ -711,7 +711,7 @@ private[spark] object Config extends Logging {
"executor status.")
.version("3.1.0")
.booleanConf
- .createWithDefault(false)
+ .createWithDefault(true)
val KUBERNETES_EXECUTOR_MISSING_POD_DETECT_DELTA =
ConfigBuilder("spark.kubernetes.executor.missingPodDetectDelta")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
