Hyukjin Kwon created SPARK-58057:
------------------------------------
Summary: Make Kubernetes VolumeSuite OnDemand PVC test robust to
slow container startup
Key: SPARK-58057
URL: https://issues.apache.org/jira/browse/SPARK-58057
Project: Spark
Issue Type: Bug
Components: Kubernetes, Tests
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The Kubernetes integration test VolumeSuite "A driver-only Spark job with an
OnDemand PVC volume" is flaky (observed on the scheduled build_java21 workflow
on branch-4.0).
checkDisk() runs 'df <path>' inside the driver container via
Utils.executeCommand, wrapped in eventually(...), but used a short fixed
timeout of 10 seconds. For the OnDemand PVC cases, dynamic PVC provisioning and
mounting can keep the driver container in ContainerCreating longer than that,
so the in-pod command is not yet runnable and the assertion never passes within
10s. The same test passes on the in-run retry, which is characteristic of a
too-tight readiness timeout rather than a product bug.
In addition, logForFailedTest() fetches the driver pod log with .getLog without
a guard. When a test fails while the driver pod is still in ContainerCreating,
that .getLog throws KubernetesClientException (HTTP 400, "container ... is
waiting to start: ContainerCreating"), which then surfaces as the reported
failure and masks the actual cause. The executor-pod log fetch right below
already wraps .getLog in a try/catch.
Fix: use the shared TIMEOUT (3 minutes) for checkDisk, matching the other
in-pod readiness waits, and apply the same try/catch guard to the driver-pod
log fetch in logForFailedTest.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
