He-Pin opened a new issue, #3693:
URL: https://github.com/apache/pekko/issues/3693
## Summary
Two related weaknesses in the multi-node testkit's conductor setup:
1. `Controller` answers `GetSockAddr` by calling `Netty`'s blocking
`ChannelFuture.sync()` inside its message handler, so a bind that is slow or
has failed blocks (or throws on) the thread of the actor that also hosts the
`BarrierCoordinator`.
2. `MultiNodeSpec` defaults the conductor port to a fixed `4711`, so two
multi-node runs on the same machine — or one leftover process — collide, and
the failure surfaces far from its cause.
## Evidence
### Blocking `.sync()` in the message handler
`multi-node-testkit/src/main/scala/org/apache/pekko/remote/testconductor/Conductor.scala:530-532`:
```scala
case GetNodes => sender() ! nodes.keys
case GetSockAddr =>
sender() ! connection.channelFuture.sync().channel().localAddress()
}
```
`connection` is created in the actor's constructor body, `:436-446`:
```scala
private[pekko] class Controller(private var initialParticipants: Int,
controllerPort: InetSocketAddress) extends Actor {
...
val connection: RemoteConnection = RemoteConnection(
Server,
controllerPort,
settings.ServerSocketWorkerPoolSize,
new ConductorHandler(settings.QueryTimeout, self,
Logging(context.system, classOf[ConductorHandler])))
```
with the bind itself at `RemoteConnection.scala:162`. `sync()` rethrows a
bind failure, so an address already in use becomes an exception thrown out of
`Controller`'s receive. The same actor also owns the barrier coordinator,
`Conductor.scala:468`:
```scala
val barrier = context.actorOf(Props[BarrierCoordinator](), "barriers")
```
`Controller`'s `supervisorStrategy` (`:454-461`) only decides on
barrier-specific exceptions (`BarrierTimeout`, `FailedBarrier`, `BarrierEmpty`,
`WrongBarrier`, `ClientLost`, `DuplicateNode`), so a bind failure is not
covered by an explicit directive.
### Fixed default port
`multi-node-testkit/src/main/scala/org/apache/pekko/remote/testkit/MultiNodeSpec.scala:227-232`:
```scala
* -Dmultinode.server-port=4711
...
val serverPort: Int = Integer.getInteger("multinode.server-port", 4711)
require(serverPort > 0 && serverPort < 65535, "multinode.server-port is
out of bounds: " + serverPort)
```
`:479-485`:
```scala
private val controllerAddr = new InetSocketAddress(serverName, serverPort)
...
if (selfIndex == 0) tc.startController(initialParticipants, myself,
controllerAddr)
else tc.startClient(myself, controllerAddr)
```
Note that `require` rejects `0`, so the ephemeral-port option is not
currently reachable through this setting.
Port-0 support already exists on the controller side —
`Conductor.scala:103-110` documents it and resolves the real address by asking
the controller:
```scala
/**
* Obtain the port to which the controller’s socket is actually bound. This
* will deviate from the configuration in `pekko.testconductor.port` in
case
* that was given as zero.
*/
def sockAddr: Future[InetSocketAddress] = {
import Settings.QueryTimeout
(controller ? GetSockAddr).mapTo(classTag[InetSocketAddress])
}
```
and `startController` (`:89-99`) already chains the resolved address into
`startClient`. What is missing is a way for node 0, which runs in its own JVM,
to publish the chosen port to the other nodes before they call `startClient`
with a pre-computed `controllerAddr`.
## Impact
A port collision does not fail fast with "address already in use". Instead
the non-controller nodes attempt to reach a conductor that is not there and
eventually time out on their queries or barriers, which reads as an unexplained
hang or a barrier failure in whichever spec lost the race. Because the port is
fixed, this is reproducible by running two multi-node suites concurrently on
one machine, or by a previous run leaving a socket in `TIME_WAIT`.
Blocking on `sync()` additionally means a slow bind stalls every barrier in
the suite, since `Controller` and `BarrierCoordinator` share the actor.
## Suggested direction
- Replace the blocking `sync()` with an asynchronous completion: register a
listener on the bind future and answer `GetSockAddr` from the callback, or move
the bind out of the constructor into `preStart` and pipe the result to self so
a failure becomes an explicit message that can stop the actor with a clear
reason instead of throwing out of a message handler.
- Allow `multinode.server-port=0` (relax the `require` at `:232`) and have
node 0 print the resolved port in a machine-readable form that the multi-node
runner can capture and pass to the remaining nodes as a system property.
`project/MultiNode.scala` is where the runner is wired.
- Consider failing the whole run immediately and legibly when the conductor
cannot bind, rather than letting the other nodes time out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
