He-Pin opened a new issue, #3692:
URL: https://github.com/apache/pekko/issues/3692
## Summary
When `ServerSocketChannel.accept()` throws, `TcpListener` logs at ERROR and
immediately re-arms `OP_ACCEPT` with no backoff. For a persistent accept error
— most commonly descriptor exhaustion (`EMFILE` / `ENFILE`) — the listening
channel stays ready, so the selector keeps firing and the listener spins in a
tight loop emitting one ERROR per iteration.
## Evidence
`actor/src/main/scala/org/apache/pekko/io/TcpListener.scala:104-107`:
```scala
def bound(registration: ChannelRegistration): Receive = {
case ChannelAcceptable =>
acceptLimit = acceptAllPending(registration, acceptLimit)
if (acceptLimit > 0)
registration.enableInterest(SelectionKey.OP_ACCEPT)
```
`:135-152`:
```scala
@tailrec final def acceptAllPending(registration: ChannelRegistration,
limit: Int): Int = {
val socketChannel =
if (limit > 0) {
try channel.accept()
catch {
case NonFatal(e) => { log.error(e, "Accept error: could not accept
new connection"); null }
}
} else null
if (socketChannel ne null) {
log.debug("New connection accepted")
...
acceptAllPending(registration, limit - 1)
} else if (bind.pullMode) limit
else BatchAcceptLimit
}
```
On error the result is `null`, so the non-pull-mode branch returns
`BatchAcceptLimit`, which is constrained positive —
`actor/src/main/scala/org/apache/pekko/io/Tcp.scala:618`:
```scala
val BatchAcceptLimit: Int = getInt("batch-accept-limit").requiring(_ >
0, "batch-accept-limit must be > 0")
```
`acceptLimit > 0` therefore holds and `:107` re-registers `OP_ACCEPT` at
once. There is no `scheduleOnce`, no retry message and no backoff state
anywhere in the file. In pull mode the incoming `limit` is returned unchanged
(`:150`), which is likewise still positive whenever the listener was accepting.
Java NIO selectors are level-triggered, and a connection that could not be
accepted remains in the backlog, so the channel is immediately ready again.
## Impact
Two compounding effects while the condition persists:
- **Log flooding** — one `log.error` per accept attempt, at selector-loop
rate. This can fill disk or drown the log that an operator needs in order to
diagnose the underlying exhaustion.
- **CPU spin** — the listener actor and its selector thread stay hot,
competing with the rest of the system at exactly the moment resources are
scarce.
Note the listener does *not* stop accepting, which is the right behaviour —
this is purely about the absence of pacing. The unrelated
`FailedRegisterIncoming` path (`:113-118`) already handles selector capacity
limits deliberately, by closing the connection and warning once.
## Suggested direction
Distinguish transient from fatal accept failures:
- On a retriable `IOException` (descriptor exhaustion, `ECONNABORTED`), do
not re-arm immediately; schedule a re-arm after a short delay instead, and
rate-limit or de-duplicate the log line so a sustained condition produces a
bounded number of messages rather than one per attempt.
- Treat `ClosedChannelException` / an already-closed selector quietly, since
that is ordinary shutdown.
A configurable delay (or reuse of an existing backoff setting) would let
operators tune it; the default should be small enough not to cost throughput
under a brief spike.
## Test
Descriptor exhaustion is awkward to provoke portably. A test could
substitute a `ServerSocketChannel` whose `accept()` throws a controlled
`IOException` for the first *n* calls, and assert that the listener (a) does
not re-arm synchronously in a loop, (b) emits a bounded number of error lines,
and (c) resumes accepting normally once `accept()` succeeds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
