HyukjinKwon opened a new pull request, #57057:
URL: https://github.com/apache/spark/pull/57057
> **⚠️ [DO-NOT-MERGE]** — Draft opened on a fork to validate a CI deflake
against the
> `Build / Maven (Scala 2.13, JDK 21, MacOS-26)` workflow. Not for upstream
merge as-is.
### What changes were proposed in this pull request?
Widen the post-sync wall-clock skew tolerance in `BarrierTaskContextSuite`
from a hard-coded
`<= 1000` (ms) to a documented `maxSyncSkewMs = 3000` constant, applied to
the three tests that
assert how closely tasks finish a `barrier()` / `allGather()` global sync:
- `global sync by barrier() call`
- `successively sync with allGather and barrier`
- `support multiple barrier() call within a single task`
### Why are the changes needed?
The `Build / Maven (Scala 2.13, JDK 21, MacOS-26)` scheduled workflow has
been failing on every
recent run. One of the reproducing failures is:
```
BarrierTaskContextSuite:
- successively sync with allGather and barrier *** FAILED ***
1078 was not less than or equal to 1000 (BarrierTaskContextSuite.scala:122)
```
These tests capture `System.currentTimeMillis()` in each task immediately
after a
barrier/allGather returns and assert `times.max - times.min <= 1000`. **That
skew is bounded below
by the barrier client's own polling granularity, not by sync correctness.**
`BarrierTaskContext.runBarrier` waits for the coordinator RPC in a
`Thread.sleep(1000)` loop
([`BarrierTaskContext.scala:102`](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/BarrierTaskContext.scala#L102)),
so two tasks can observe the release up to ~1s apart from the poll interval
alone — before any
thread-scheduling or GC jitter on a busy CI host is added.
Evidence gathered:
- SPARK-49983 (2024, PR #48487) previously saw `1038` and only *halved the
pre-barrier sleep*.
That reduces arrival spread but cannot reduce poll-granularity skew, so
the flake persists on
slower runners like `macos-26` arm64.
- Reproduced locally on **macOS 26 arm64 / JDK 21** (the same OS/arch as the
failing runner):
measured post-sync skew up to **994 ms while the machine was idle** — i.e.
essentially zero
margin under the old `1000` bound, so any CI load tips it over.
The new bound (twice the 1 s poll interval plus a jitter allowance)
preserves the test's intent —
tasks stay loosely in lockstep and none can race an entire extra sync ahead
— without coupling the
assertion to a razor-thin wall-clock margin.
### Scope note
This PR fixes only the `BarrierTaskContextSuite` flake. The same workflow
also exhibits other,
independent flakes (Kafka `stress test for failOnDataLoss=false`
topic-deletion timeout, mllib GMM
FP-tolerance on arm64, `StateStoreSuite.maintenance` timeout). Those are out
of scope here and
would be separate changes. (The earlier `AF_UNIX path too long` UDF-worker
flake was already fixed
upstream by SPARK-57949.)
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
`build/mvn -pl core test
-DwildcardSuites=org.apache.spark.scheduler.BarrierTaskContextSuite`
on macOS 26 arm64 with JDK 21: all 13 tests pass; scalastyle clean.
### Was this patch authored or co-authored using generative AI tooling?
Yes, drafted with assistance from Claude.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
