HyukjinKwon opened a new pull request, #57029:
URL: https://github.com/apache/spark/pull/57029
### What changes were proposed in this pull request?
Make the test helper `retryOnOOM` (in `SparkFunSuite`) retry a few times
(default 3) with
a full `runGC()` between attempts, instead of running GC and retrying only
once. The sole
caller (`SorterSuite`) is updated to the new `retryOnOOM()(body)` form.
### Why are the changes needed?
The `core` module test job intermittently aborts with an OOM (observed ~1 in
5 runs, e.g.
in the branch-4.x Maven build):
```
*** RUN ABORTED ***
An exception or error caused a run to abort: Java heap space
java.lang.OutOfMemoryError: Java heap space
at
org.apache.spark.util.collection.SorterSuite.$anonfun$new$12(SorterSuite.scala:145)
at org.apache.spark.SparkFunSuite.retryOnOOM(SparkFunSuite.scala:122)
```
`SorterSuite` allocates a single ~1 GB `byte[]` (heap is `-Xmx4g`).
SPARK-57037 already
wrapped that allocation in `retryOnOOM`, which does one `System.gc()` + one
retry. But a
single GC after the first failure does not always reclaim the previous
test's large array
in time, so the lone retry allocates against a still-pressured heap and OOMs
again.
Because the failure is intermittent (the job passes ~4 of 5 runs), 4 GB is
not fundamentally
too small — this is transient heap pressure, exactly the case where
additional GC+retry
attempts help. Retrying up to 3 times (with `runGC()`, which waits for a
weak reference to
clear, between attempts) gives the JVM more chances to reclaim memory.
This is a test flakiness reduction, not a correctness change: a genuinely
too-large or
leaked allocation still fails once the attempts are exhausted, so real OOMs
are not masked.
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
`SorterSuite` (which exercises `retryOnOOM`) on a fork via GitHub Actions;
confirmed the
signature change compiles and the suite passes.
**CI evidence**
- Failing run (apache/spark, branch-4.x `build_maven`, `core` module —
SorterSuite OOM):
https://github.com/apache/spark/actions/runs/28747189185
- Passing validation (fork, compiles `core` + runs `SorterSuite`):
https://github.com/HyukjinKwon/spark-agent6/actions/runs/28766544512
Note: the underlying OOM is intermittent (~1/5), so this reduces the flake
probability;
it is not claimed to statistically eliminate it.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
