HyukjinKwon opened a new pull request, #56790:
URL: https://github.com/apache/spark/pull/56790
### What changes were proposed in this pull request?
Wrap the final `noLiveData()` assertion in
`SQLAppStatusListenerMemoryLeakSuite`
("no memory leak") in an `eventually(...)` block so it polls for the
asynchronous
cleanup instead of asserting once immediately.
### Why are the changes needed?
The test is flaky on the SBT `sql - other tests` job, failing with
`noLiveData() was false`:
```
assert(statusStore.listener.get.noLiveData())
```
`SQLAppStatusListener` removes entries from `liveExecutions` / `stageMetrics`
only once the per-execution end-event count reaches `jobs.size + 1`. The
final
increment + removal happens in the metrics-aggregation callback triggered by
`SparkListenerSQLExecutionEnd` (`onExecutionEnd` -> `kvstore.doAsync { ...
}`).
Asserting `noLiveData()` right after `sc.listenerBus.waitUntilEmpty()` is
racy:
the listener bus queue can be drained before that cleanup callback has fully
removed the live entries, so the test intermittently observes leftover live
data.
Polling with `eventually` lets the cleanup settle without weakening the
assertion.
### Does this PR introduce _any_ user-facing change?
No, test only.
### How was this patch tested?
Existing test, run repeatedly. This reproduces under SBT, so the standard
fork
"Build" (SBT) exercises it.
### CI evidence
- Before (red): https://github.com/apache/spark/actions/runs/28000645341
(master SBT, `sql - other tests` — `noLiveData() was false`)
- Also red: https://github.com/apache/spark/actions/runs/28004073554
- After (green):
https://github.com/HyukjinKwon/spark/actions/runs/28204756051
### Was this patch authored or co-authored using generative AI tooling?
Yes, drafted with assistance from Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
