This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 65516a0bfd44 [SPARK-58057][K8S][TEST] Make VolumeSuite OnDemand PVC
test robust to slow container startup
65516a0bfd44 is described below
commit 65516a0bfd44d04bd8caacdaa86c11a1e785f504
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Wed Jul 8 21:44:05 2026 -0700
[SPARK-58057][K8S][TEST] Make VolumeSuite OnDemand PVC test robust to slow
container startup
### What changes were proposed in this pull request?
Make the Kubernetes integration test `VolumeSuite` more robust to slow
driver-container startup, and stop the failure reporter from masking the real
error:
1. `checkDisk()` (`VolumeSuite.scala`) runs `df <path>` **inside** the
driver container via `Utils.executeCommand`, wrapped in `eventually(...)`. It
used a short fixed timeout of `Span(10, Seconds)`. For the `OnDemand` PVC
cases, dynamic PVC provisioning and mounting can keep the driver container in
`ContainerCreating` for longer than that, so the in-pod command is not yet
runnable and the assertion never passes within 10s. This changes `checkDisk` to
use the shared `TIMEOUT` (3 minut [...]
2. `logForFailedTest()` (`KubernetesSuite.scala`) fetches the driver pod
log with `.getLog` **without** a guard. When a test fails while the driver pod
is still in `ContainerCreating`, that `.getLog` throws
`KubernetesClientException` (HTTP 400, "container ... is waiting to start:
ContainerCreating"), which then surfaces as the reported test failure and hides
the actual cause. The executor-pod log fetch right below already wraps
`.getLog` in a `try/catch`; this applies the same guard [...]
### Why are the changes needed?
The scheduled `build_java21.yml` workflow on `branch-4.0` intermittently
fails the K8s integration test "A driver-only Spark job with an OnDemand PVC
volume". In the failing run the test failed after ~15s while the same test
passed on the in-run retry (~13s), which is characteristic of a too-tight
readiness timeout rather than a product bug. The reported exception was the
masking `KubernetesClientException` from `logForFailedTest`, not the underlying
timeout, making triage harder.
### Does this PR introduce _any_ user-facing change?
No. Test-only change in the Kubernetes integration tests.
### How was this patch tested?
Kubernetes integration test `VolumeSuite`.
CI evidence (before/after):
- Failing scheduled run on `branch-4.0` (original):
https://github.com/apache/spark/actions/runs/28921150970 — "A driver-only Spark
job with an OnDemand PVC volume" `*** FAILED ***`.
- Passing run with this fix (fork, `k8s-integration-tests`):
https://github.com/HyukjinKwon/spark/actions/runs/28984700141 — the K8s
integration test suite reports `Tests: succeeded 86, failed 0` and "A
driver-only Spark job with an OnDemand PVC volume" passes.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #57146 from HyukjinKwon/DO-NOT-MERGE/deflake-k8s-volumesuite.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 067a6b39303fcd7e6f9610c88cd08477c1e7f95f)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark/deploy/k8s/integrationtest/KubernetesSuite.scala | 9 +++++++--
.../apache/spark/deploy/k8s/integrationtest/VolumeSuite.scala | 5 ++++-
2 files changed, 11 insertions(+), 3 deletions(-)
diff --git
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
index 71fa9e653e52..624d5b2dae2c 100644
---
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
+++
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
@@ -87,12 +87,17 @@ class KubernetesSuite extends SparkFunSuite
.asScala
.headOption
driverPodOption.foreach { driverPod =>
- logInfo("BEGIN driver POD log\n" +
+ val driverPodLog = try {
kubernetesTestComponents.kubernetesClient
.pods()
.inNamespace(kubernetesTestComponents.namespace)
.withName(driverPod.getMetadata.getName)
- .getLog)
+ .getLog
+ } catch {
+ case e: KubernetesClientException =>
+ s"Error fetching log (pod is likely not ready) $e"
+ }
+ logInfo("BEGIN driver POD log\n" + driverPodLog)
logInfo("END driver POD log")
}
kubernetesTestComponents.kubernetesClient
diff --git
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/VolumeSuite.scala
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/VolumeSuite.scala
index 9b543fd000d7..a261e9630379 100644
---
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/VolumeSuite.scala
+++
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/VolumeSuite.scala
@@ -30,7 +30,10 @@ private[spark] trait VolumeSuite { k8sSuite: KubernetesSuite
=>
val IGNORE = Some((Some(PatienceConfiguration.Interval(Span(0, Seconds))),
None))
private def checkDisk(pod: Pod, path: String, expected: String) = {
- eventually(PatienceConfiguration.Timeout(Span(10, Seconds)), INTERVAL) {
+ // Use the shared TIMEOUT rather than a short fixed timeout: when the
volume is an OnDemand
+ // PVC, dynamic provisioning and mounting can keep the driver container in
ContainerCreating
+ // for more than a few seconds, so the in-pod `df` command may not be
runnable immediately.
+ eventually(TIMEOUT, INTERVAL) {
implicit val podName: String = pod.getMetadata.getName
implicit val components: KubernetesTestComponents =
kubernetesTestComponents
assert(Utils.executeCommand("df", path).contains(expected))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
