This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 5ae5aa59a82b [SPARK-57952][CORE][TEST] Retry retryOnOOM a few times to
reduce SorterSuite flaky OOM
5ae5aa59a82b is described below
commit 5ae5aa59a82b4d0a75b0412a32b0204a6ea2508f
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Jul 6 17:27:37 2026 +0900
[SPARK-57952][CORE][TEST] Retry retryOnOOM a few times to reduce
SorterSuite flaky OOM
### What changes were proposed in this pull request?
Make the test helper `retryOnOOM` (in `SparkFunSuite`) retry a few times
(default 3) with
a full `runGC()` between attempts, instead of running GC and retrying only
once. The sole
caller (`SorterSuite`) is updated to the new `retryOnOOM()(body)` form.
### Why are the changes needed?
The `core` module test job intermittently aborts with an OOM (observed ~1
in 5 runs, e.g.
in the branch-4.x Maven build):
```
*** RUN ABORTED ***
An exception or error caused a run to abort: Java heap space
java.lang.OutOfMemoryError: Java heap space
at
org.apache.spark.util.collection.SorterSuite.$anonfun$new$12(SorterSuite.scala:145)
at org.apache.spark.SparkFunSuite.retryOnOOM(SparkFunSuite.scala:122)
```
`SorterSuite` allocates a single ~1 GB `byte[]` (heap is `-Xmx4g`).
SPARK-57037 already
wrapped that allocation in `retryOnOOM`, which does one `System.gc()` + one
retry. But a
single GC after the first failure does not always reclaim the previous
test's large array
in time, so the lone retry allocates against a still-pressured heap and
OOMs again.
Because the failure is intermittent (the job passes ~4 of 5 runs), 4 GB is
not fundamentally
too small — this is transient heap pressure, exactly the case where
additional GC+retry
attempts help. Retrying up to 3 times (with `runGC()`, which waits for a
weak reference to
clear, between attempts) gives the JVM more chances to reclaim memory.
This is a test flakiness reduction, not a correctness change: a genuinely
too-large or
leaked allocation still fails once the attempts are exhausted, so real OOMs
are not masked.
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
`SorterSuite` (which exercises `retryOnOOM`) on a fork via GitHub Actions;
confirmed the
signature change compiles and the suite passes.
**CI evidence**
- Failing run (apache/spark, branch-4.x `build_maven`, `core` module —
SorterSuite OOM):
https://github.com/apache/spark/actions/runs/28747189185
- Passing validation (fork, compiles `core` + runs `SorterSuite`):
https://github.com/HyukjinKwon/spark-agent6/actions/runs/28766544512
Note: the underlying OOM is intermittent (~1/5), so this reduces the flake
probability;
it is not claimed to statistically eliminate it.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57029 from HyukjinKwon/ci-fix/agent6-oom-pr.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../scala/org/apache/spark/SparkFunSuite.scala | 29 ++++++++++++++++------
.../apache/spark/util/collection/SorterSuite.scala | 4 +--
2 files changed, 24 insertions(+), 9 deletions(-)
diff --git a/core/src/test/scala/org/apache/spark/SparkFunSuite.scala
b/core/src/test/scala/org/apache/spark/SparkFunSuite.scala
index a0f17f8af3f3..a18b1ec5dbf9 100644
--- a/core/src/test/scala/org/apache/spark/SparkFunSuite.scala
+++ b/core/src/test/scala/org/apache/spark/SparkFunSuite.scala
@@ -113,13 +113,28 @@ abstract class SparkFunSuite
}
}
- /** Run `body`; if it throws OutOfMemoryError, force a GC and retry once. */
- protected def retryOnOOM[T](body: => T): T = {
- try body
- catch {
- case _: OutOfMemoryError =>
- runGC()
- body
+ /**
+ * Run `body`; if it throws OutOfMemoryError, force a GC and retry, up to
`maxAttempts`
+ * total attempts (default 3). A single GC+retry is not always enough in
memory-constrained
+ * CI: a large array retained by a previous test may not be reclaimed by the
first
+ * System.gc(), so the retried allocation OOMs again. Retrying a few times
(with a full
+ * runGC() that waits for a weak reference to clear between attempts) gives
the JVM
+ * additional chances to reclaim memory. This does not mask genuine leaks:
an unbounded
+ * allocation still fails once the attempts are exhausted.
+ */
+ protected def retryOnOOM[T](maxAttempts: Int = 3)(body: => T): T = {
+ var attempt = 1
+ while (true) {
+ try {
+ return body
+ } catch {
+ case oom: OutOfMemoryError =>
+ if (attempt >= maxAttempts) throw oom
+ runGC()
+ attempt += 1
+ }
}
+ // Unreachable: the loop either returns a value or rethrows the OOM.
+ throw new IllegalStateException("retryOnOOM exited its retry loop
unexpectedly")
}
}
diff --git
a/core/src/test/scala/org/apache/spark/util/collection/SorterSuite.scala
b/core/src/test/scala/org/apache/spark/util/collection/SorterSuite.scala
index 2767769924bc..f654501ab436 100644
--- a/core/src/test/scala/org/apache/spark/util/collection/SorterSuite.scala
+++ b/core/src/test/scala/org/apache/spark/util/collection/SorterSuite.scala
@@ -141,8 +141,8 @@ class SorterSuite extends SparkFunSuite {
val arrayToSortSize = 1091482190
// Memory held by the previous test (e.g. the ~256 MB int array in
"SPARK-5984
// TimSort bug") may not be reclaimed before this >1 GB allocation,
causing flaky
- // OOM in CI. Force a GC and retry once on OOM.
- val arrayToSort = retryOnOOM(new Array[Byte](arrayToSortSize))
+ // OOM in CI. Force a GC and retry a few times on OOM.
+ val arrayToSort = retryOnOOM()(new Array[Byte](arrayToSortSize))
var sum: Int = -1
for (i <- runLengths) {
sum += i
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
