[
https://issues.apache.org/jira/browse/CAMEL-26574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-26574 started by shashank.
----------------------------------------
> camel-netty - Client mode consumer: stop leaves the connection and the
> reconnect running with a shared worker group, and the reconnect blocks the
> event loop
> ------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26574
> URL: https://issues.apache.org/jira/browse/CAMEL-26574
> Project: Camel
> Issue Type: Bug
> Components: camel-netty
> Reporter: shashank
> Assignee: shashank
> Priority: Major
>
> Found with a TLA+ model of {{ClientModeTCPNettyServerBootstrapFactory}} (the
> lifecycle thread starting and stopping the consumer, the event loops running
> the close listener and the reconnect task, connect outcomes and the server
> closing the connection chosen nondeterministically; a second model with two
> consumers on one shared group of two event loops), then reproduced on the
> real classes against an embedded Netty server on localhost.
> A consumer with {{clientMode=true}} connects to a server and, with
> {{reconnect=true}} (the default), connects again when the connection is lost.
> Several things go wrong:
> *1. A stop does not close the connection when the worker group is shared.*
> {{stopServerBootstrap()}} never closes the channel; it only shuts down a
> worker group the factory created itself. With {{workerGroup=#myGroup}} (the
> documented way to share event loops between routes) the stopped consumer
> keeps its connection: whatever the server sends after the stop is read and
> dropped (the server sees it as delivered). The close listener and the
> scheduled reconnect stay active too:
> * a reconnect scheduled before the stop (the server was down at start, or had
> just closed the connection) connects after the stop;
> * when the server closes the leftover connection after the stop, the stopped
> consumer connects again;
> * a route restart opens a second connection while the first one stays open,
> and both feed the route.
> TLA+ trace (main): start while the server is down (reconnect scheduled),
> stop: after the stop the reconnect is still scheduled ({{StopNoReconnect}}
> violated in 5 steps); start, stop, start: two open connections
> ({{OneConnection}} violated).
> *2. The reconnect waits on an event loop.* The reconnect task runs on an
> event loop and calls {{openChannel}}, which waits for the connect with
> {{awaitUninterruptibly()}}.
> * When the new channel gets the same event loop ({{workerCount=1}}, or a
> shared group whose round robin returns the loop of the task), Netty throws
> {{BlockingOperationException}}. The task logs "Will attempt again in ...
> millis. This exception is ignored." but schedules nothing; the connect goes
> on without the close listener, so after the next connection loss the consumer
> never reconnects.
> * When two reconnect tasks on a shared group of two loops (two consumers
> whose server restarted, or a stopped consumer's leftover reconnect and the
> restarted one) each connect a channel on the loop of the other, both loops
> wait for a connect that only the other loop can complete. The connect timeout
> is scheduled on the same blocked loops, so this never ends: every channel of
> the shared group (other routes, producers) hangs, and a route start waiting
> for its connect on one of these loops never returns (nor does the shutdown of
> the context, which waits for that start).
> TLA+ trace (two consumers, shared group of two loops): both connections drop;
> the reconnect of consumer 2 (loop 2) connects a channel on loop 1 and waits;
> the reconnect of consumer 1 (loop 1) connects a channel on loop 2 and waits:
> {{NoLoopDeadlock}} violated in 5 steps. Another trace: a reconnect whose
> channel lands on its own loop ends the reconnects of that consumer
> ({{ChainsKept}} violated in 3 steps).
> Reproduced on main (1b12eea752e6) with an embedded Netty server on localhost:
> * {{workerGroup=#shared}}, route stopped: the server side of the connection
> is still open 10 s later; with {{reconnect=true}}, the server closing that
> connection after the stop is followed by a new connection from the stopped
> consumer; after a stop and a start two connections are open and messages on
> both reach the route.
> * {{workerCount=1}}: the consumer connects once and never again after the
> server closes the connection ({{BlockingOperationException}} in the log).
> * Two client consumers on one shared group of two loops, the server closing
> both connections at once five times: fails 20 of 20 runs.
> * A hook-free loop of 200 stop/start cycles (shared group, the server closing
> every connection after 0-3 ms): a connection after the stop in 105 of the
> first 110 cycles, then the start of the next cycle hung on the event loop
> deadlock (the thread dump shows both loops in {{openChannel}} ->
> {{awaitUninterruptibly}}). With the change: 0 of 200, no hang.
> *Proposal*
> * {{stopServerBootstrap}} cancels the scheduled reconnect and closes the
> channel and a connect in progress, and waits for the close (up to
> {{shutdownTimeout}}, as the server mode does, not when called on the event
> loop of the channel).
> * The reconnect does not wait on the event loop: the connect outcome is
> handled by a listener (track the channel and its close listener, or schedule
> the next reconnect).
> * The reconnect state (channel, connect in progress, scheduled task, stopping
> flag, a generation per start) is guarded by one lock; the close listener, the
> reconnect task and the connect listener act only if the consumer is not
> stopping and belongs to the current start, so a stop racing a reconnect
> leaves nothing behind and a restart cannot run two reconnect chains.
> * No option or default changes. The code is the same on camel-4.23.x, 4.18.x
> and 4.14.x.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
