HyukjinKwon opened a new pull request, #57145:
URL: https://github.com/apache/spark/pull/57145
> **[DO-NOT-MERGE]** — draft to validate the deflake on `build_java21` CI
before filing the JIRA and
> flipping to a proper `[SPARK-XXXXX]` title. The prefix will be replaced
once CI is green.
### What changes were proposed in this pull request?
Deflake `ClientStreamingQuerySuite."listener events"`, which intermittently
fails the
`Build / Java21 (Scala 2.13, JDK 21)` scheduled workflow (in the
`streaming, sql-kafka-0-10, ..., connect, avro` module job) with:
```
- listener events *** FAILED ***
The code passed to eventually never returned normally. Attempted 70 times
over
1.0033581898666666 minutes. Last failure message: List() was empty.
(ClientStreamingQuerySuite.scala:580)
```
Two independent changes:
1. **Server (`SparkConnectListenerBusListener.send`)** — previously, any
`onNext` exception removed
the listener and cleaned up all resources, so the long-running thread
stopped forwarding **all**
subsequent events. A single transient failure on a frequent
`QueryProgressEvent` therefore
silently dropped the later terminal `QueryTerminatedEvent`, and the
client's `listener.terminate`
stayed empty forever. `onNext` is now retried a bounded number of times
(`maxSendAttempts = 3`,
200ms backoff) before falling through to the existing cleanup path, so a
momentary gRPC hiccup no
longer tears the listener down.
2. **Test (`ClientStreamingQuerySuite` / `MyListener`)** — the listener's
`start`/`progress`/
`terminate` fields are mutated on the listener dispatch thread and read
on the test thread, so
they are now `@volatile` to guarantee visibility. Assertions also carry a
`diag(stage)` message
reporting `q.isActive`, `q.exception`, event counts, and the client-side
listener count, so a
future flake in a scheduled job is diagnosable from the log (the connect
server runs in a
separate process whose logs are not captured in CI).
### Why are the changes needed?
`ClientStreamingQuerySuite."listener events"` is a genuine non-deterministic
flake (a lost terminal
event plus a cross-thread visibility gap), not a logic breakage, and it
periodically reddens the
`build_java21` badge. The bounded retry hardens the server against transient
send failures while
still cleaning up when the client is genuinely unresponsive.
### Does this PR introduce _any_ user-facing change?
No. The server change only adds a bounded retry on a transient transmit
failure before the existing
teardown; the rest is test-only.
### How was this patch tested?
Existing `ClientStreamingQuerySuite` (`connect-client-jvm`) exercises the
path. Local full
compilation is not possible in this environment (Maven Central is
network-restricted), so this
draft validates the deflake by re-running the affected suite on
`build_java21` CI on the fork
before being finalized.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
