He-Pin opened a new issue, #3247:
URL: https://github.com/apache/pekko/issues/3247
### Description
When shutting down an `ActorSystem` using Artery with Aeron-UDP transport,
termination can take excessively long due to stacked Aeron liveness timeouts
and the embedded media driver drain sequence. This is most pronounced in
virtualized environments.
### Evidence from Pekko CI
The issue has been acknowledged through test timeout widening commits:
- Commit `4d17e08c5b`: widened `shutdownAll` await for aeron-udp drain on
JDK 25 nightly. The commit message states: *"aeron-udp transport is the slowest
to drain (embedded media driver + stacked Aeron liveness timeouts)"*
- Commit `3fcbea76ae`: widened TCK `actor-system-terminate` phase timeout
because *"the inner CoordinatedShutdown actor-system-terminate phase keeps its
default 10s, which is occasionally too tight"*
Both commits show `CoordinatedShutdown` phase `actor-system-terminate`
timing out with `ActorGraphInterpreter` and stream actors still mid-shutdown.
### Root cause
The shutdown flow involves multiple sequential waits that accumulate:
1. **FlushOnShutdown** — waits for all associations to acknowledge
termination (up to `shutdown-flush-timeout` = 1s)
2. **ArteryTransport.shutdown()** — `internalShutdown()` waits for
`streamsCompleted` (all outbound/inbound stream completions)
3. **shutdownHook** — blocking `Await.result` with up to `DriverTimeout +
3.seconds` (23s total)
4. **Aeron liveness timeouts** stack up:
- `client-liveness-timeout` = 20s
- `publication-unblock-timeout` = 40s
- `image-liveness-timeout` = 10s
- `driver-timeout` = 20s
5. **give-up-message-after** = 60s — messages can be retried for up to 60
seconds
### Impact
- TCP transport is not affected
- Healthy shutdowns complete in under a second
- The issue manifests with Aeron-UDP in virtualized environments or with
embedded media drivers
- Test CI intermittently times out, requiring repeated timeout widening
### Current mitigation
All timeouts are configurable via `reference.conf`, but the default stacking
of Aeron liveness timeouts makes the worst-case shutdown duration unnecessarily
long.
### Suggested approach
- Decouple the `actor-system-terminate` phase from Aeron liveness timeouts
(force-drain with a shorter deadline during shutdown)
- Add a dedicated `artery-shutdown-timeout` that bounds the total Artery
drain time independently of individual Aeron liveness settings
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
