yew1eb opened a new pull request, #3811:
URL: https://github.com/apache/celeborn/pull/3811
### What changes were proposed in this pull request?
Increase `celeborn.client.application.heartbeatInterval` from `30s` to `1h`
in `RemoteShuffleMasterSuiteJ.setUp()`, which effectively pauses the
LifecycleManager app heartbeater for the duration of this test suite. All
version-specific subclasses (`RemoteShuffleMasterSuiteJV118` ~ `V23`) inherit
from this base suite, so a single change covers all of them.
### Why are the changes needed?
`RemoteShuffleMasterSuiteJ.testRegisterPartitionWithProducer` (and the other
`shuffleCount` assertions in the same suite) is flaky:
```
java.lang.AssertionError: expected:<1> but was:<0>
at
RemoteShuffleMasterSuiteJ.testRegisterPartitionWithProducer(RemoteShuffleMasterSuiteJ.java:124)
```
Root cause is a race on `LifecycleManager#shuffleCount`:
- `RemoteShuffleMaster.registerPartitionWithProducer` increments the counter
(`RemoteShuffleMaster.java:161`)
- the LifecycleManager app-heartbeat thread periodically ships the counter
to the master via `shuffleCount.sumThenReset()` (`LifecycleManager.scala:234`)
The suite configures `celeborn.client.application.heartbeatInterval=30s`. On
slow CI runners a test can run past the 30s mark, so a heartbeat fires between
the increment and the assertion and resets the counter to 0.
`shuffleFallbackCounts` assertions are affected by the same race
(`resetFallbackCounts`). No test in this suite depends on an app heartbeat
actually being sent, so enlarging the interval has no side effects.
Example failure (flink1 (11, 1.18)):
https://github.com/apache/celeborn/actions/runs/32273603672/job/96135975529
### Does this PR resolve a correctness bug?
<!-- Check if yes. The `correctness` label will be added/removed
automatically. -->
- [ ] Yes
### Does this PR introduce _any_ user-facing change?
<!-- Check if yes. -->
- [ ] Yes
### How was this patch tested?
`./build/mvn -Pflink-1.18 -pl client-flink/flink-1.18 test
-Dtest=RemoteShuffleMasterSuiteJV118` — all 7 tests passed (previously
`shuffleCount` assertions could intermittently fail on slow runners).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
