HyukjinKwon opened a new pull request, #57037:
URL: https://github.com/apache/spark/pull/57037
### What changes were proposed in this pull request?
Make `MetricsFailureInjectionSuite`'s `Non-deterministic stage block failure
injection -
injectFailure=true` case re-run its query (resetting the metrics each
attempt) until the
injected shuffle fetch-failure actually forces a stage-1 recompute, up to 10
attempts, before
running the existing assertions. Test-only.
### Why are the changes needed?
The test is flaky. In the full suite it fails ~1 run in 6 with:
```
- Non-deterministic stage block failure injection - injectFailure=true ***
FAILED ***
300 was not greater than 300 (MetricsFailureInjectionSuite.scala:...)
```
Diagnosis (repeated runs on a macOS arm64 runner, capturing the metric
values and scheduler
markers):
- The test **passes 10/10 when run in isolation**, but flakes in the full
suite - so it is a
cross-test/scheduling interaction, not a problem with the query itself.
- On a failing run the metrics are `stage1=300 stage2=5` (exactly the
no-recompute values) and
the case finishes in ~370ms versus ~860ms on a passing run, and the test's
own shuffle gets
**no injected `FetchFailed`**.
`INJECT_SHUFFLE_FETCH_FAILURES` corrupts mapper-0 of the shuffle map stage,
but whether the
downstream reducer observes the resulting `FetchFailed` - and thus forces
the stage-1 recompute
that makes the raw `stage1Metric` exceed 300 - depends on task scheduling
within the shared
`SparkContext`. It occasionally does not fire, leaving `stage1Metric` at
exactly 300.
Retrying until the injection fires makes the test robust without weakening
what it verifies (it
still requires a real recompute), and without touching the shared
`DAGScheduler` injection
machinery that other suites (`SQLLastAttemptMetric*`, DSv2 DML metric tests)
rely on.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
Ran the **full** `MetricsFailureInjectionSuite` **8×** on a `macos-15`
runner (where it was ~1/6
flaky before): all 8 iterations green (`Tests: succeeded 13, failed 0`).
- Before (flaky on `macos-26`):
https://github.com/apache/spark/actions/runs/28753698265
- After (this fix, 8/8 green on fork):
https://github.com/HyukjinKwon/spark/actions/runs/28780271388
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
