HyukjinKwon opened a new pull request, #57146: URL: https://github.com/apache/spark/pull/57146
[DO-NOT-MERGE] This is a flaky-test fix under CI verification; the title will be updated with the JIRA id once the fork CI run confirms the fix. Please do not merge yet. ### What changes were proposed in this pull request? Make the Kubernetes integration test `VolumeSuite` more robust to slow driver-container startup, and stop the failure reporter from masking the real error: 1. `checkDisk()` (`VolumeSuite.scala`) runs `df <path>` **inside** the driver container via `Utils.executeCommand`, wrapped in `eventually(...)`. It used a short fixed timeout of `Span(10, Seconds)`. For the `OnDemand` PVC cases, dynamic PVC provisioning and mounting can keep the driver container in `ContainerCreating` for longer than that, so the in-pod command is not yet runnable and the assertion never passes within 10s. This changes `checkDisk` to use the shared `TIMEOUT` (3 minutes), matching the other in-pod readiness waits in the suite. 2. `logForFailedTest()` (`KubernetesSuite.scala`) fetches the driver pod log with `.getLog` **without** a guard. When a test fails while the driver pod is still in `ContainerCreating`, that `.getLog` throws `KubernetesClientException` (HTTP 400, "container ... is waiting to start: ContainerCreating"), which then surfaces as the reported test failure and hides the actual cause. The executor-pod log fetch right below already wraps `.getLog` in a `try/catch`; this applies the same guard to the driver-pod fetch. ### Why are the changes needed? `build_java21.yml` on `branch-4.0` intermittently fails the K8s integration test "A driver-only Spark job with an OnDemand PVC volume". In the failing run the test failed after ~15s while the same test passed on the in-run retry (~13s), which is characteristic of a too-tight readiness timeout rather than a product bug. The reported exception was the masking `KubernetesClientException` from `logForFailedTest`, not the underlying timeout, making triage harder. ### Does this PR introduce _any_ user-facing change? No. Test-only change in the Kubernetes integration tests. ### How was this patch tested? Existing K8s integration test `VolumeSuite`. The change widens an existing timeout and adds an already-used try/catch guard; verified via a Kubernetes integration test run on a fork. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
