He-Pin opened a new issue, #3135:
URL: https://github.com/apache/pekko/issues/3135
## Summary
`TcpOutgoingConnection` retries a failed `finishConnect` by scheduling a
callback
on the system scheduler that calls `channelRegistry.register(channel,
OP_CONNECT)`
directly, outside the actor's message loop. If that callback throws, the
exception
is caught and reported by the scheduler / dispatcher and **never re-enters
the
actor's supervision**, so `Tcp.CommandFailed` is never delivered and the
commander can stay stuck in the `connecting` state indefinitely.
## Code location
`actor/src/main/scala/org/apache/pekko/io/TcpOutgoingConnection.scala`
```scala
// connecting(registration, remainingFinishConnectRetries)
case ChannelConnectable =>
reportConnectFailure {
if (channel.finishConnect()) { ... }
else {
if (remainingFinishConnectRetries > 0) {
context.system.scheduler.scheduleOnce(1.millisecond) {
channelRegistry.register(channel, SelectionKey.OP_CONNECT) //
<-- off-loop
}(context.dispatcher)
context.become(connecting(registration,
remainingFinishConnectRetries - 1))
} else {
stop(FinishConnectNeverReturnedTrueException)
}
}
}
```
The surrounding `reportConnectFailure { ... }` only covers the body of the
`ChannelConnectable` handler — it does **not** cover the scheduled lambda.
## Why this is a problem
`channelRegistry.register` (the `SelectorBasedChannelRegistry` in
`SelectionHandler.scala`) dispatches a `Task` to the selector's execution
context whose `tryRun` only catches `ClosedChannelException`. Any other
throwable (e.g. `IllegalArgumentException` from an invalid ops mask,
`CancelledKeyException` if the key races with `postStop` shutdown,
`IllegalBlockingModeException`, or a JVM/NIO bug-induced `RuntimeException`)
propagates out of the task, is handed to `executionContext.reportFailure`,
and
is logged — but:
- it does **not** enter `reportConnectFailure`,
- it does **not** trigger `stop(e)`,
- it does **not** emit `Tcp.CommandFailed(connect).withCause(e)` to the
commander,
- the `TcpOutgoingConnection` actor keeps running in the `connecting` state
with the retry counter already decremented, and the commander can wait
forever for `Connected` / `CommandFailed`.
In practice this has been observed in the wild on akka.net under an analogous
code shape — see
[akkadotnet/akka.net#8195](https://github.com/akkadotnet/akka.net/issues/8195)
for the original report and the failure sequence. The Pekko code path has the
same structural issue: a raw scheduled callback performs I/O work that is not
wrapped in the actor's failure-reporting envelope.
## Expected behaviour
Any exception thrown by the scheduled retry should surface to the commander
as
`Tcp.CommandFailed` and the connection actor should stop cleanly, exactly as
happens for exceptions thrown inside the normal `receive` path.
## Suggested fix
Schedule the retry as a self-message so it runs inside the actor's message
loop and is covered by the existing `reportConnectFailure` envelope — e.g.
introduce an internal `RetryRegister` message and, in `connecting`, handle
`case RetryRegister => reportConnectFailure {
channelRegistry.register(channel, SelectionKey.OP_CONNECT) }`.
Using `Timers.StartSingleTimer` (as `TcpListener` does in the same module) is
also preferable to `context.system.scheduler.scheduleOnce`, since the pending
timer is cancelled automatically when the actor stops, removing the latent
"register against a channel that is already being torn down" window.
## Environment
- Pekko version: current `main`
- Module: `actor` (`org.apache.pekko.io.Tcp*`)
- JDK/OS: any — the bug is structural; triggering it depends on when
`channel.register` throws a non-`ClosedChannelException` throwable.
## Reproduction notes
The akka.net report reproduces the swallowed-exception path by pre-seeding
DNS
so the host resolves to both IPv4 and IPv6, forcing an IPv4-only socket, and
letting the IPv6 fallback retry throw a platform-wide
`NotSupportedException`.
A symmetric Pekko test would force the scheduled `channelRegistry.register`
call to throw (e.g. by injecting a `ChannelRegistry` whose `register` throws
`IllegalArgumentException`) and assert that the commander still receives
`Tcp.CommandFailed` rather than hanging.
## References
- Original field report and analysis:
https://github.com/akkadotnet/akka.net/issues/8195
- Pekko source:
`actor/src/main/scala/org/apache/pekko/io/TcpOutgoingConnection.scala`,
`actor/src/main/scala/org/apache/pekko/io/SelectionHandler.scala`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
