[
https://issues.apache.org/jira/browse/CAMEL-25713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18128389#comment-18128389
]
Serdar Gökay commented on CAMEL-25713:
--------------------------------------
PR: https://github.com/apache/camel/pull/28049
Instead of the client's executor suggested above, each client connection now
has a scheduled thread pool of its own from the ExecutorServiceManager of the
CamelContext, shut down with the connection. It handles the outcome of the
producer requests and runs the reconnect job, so a failed request no longer
waits in OpcUaClient.disconnect() on a thread that Milo needs for the reconnect
attempt. Against opc-plc 2.15.4 with the client limited to half a CPU, main did
not rebuild the connection in 4 of 12 runs; with the PR it was rebuilt in all
12. The pool never has more threads than Milo's shared scheduled executor, so
no guard against a stale failure is needed: with one processor it handles the
failures in order on a single thread. It could also serve as the executor that
CAMEL-25692 proposes. The change does not touch the producer guards that
CAMEL-25694 (#27956) changes.
_Claude Code on behalf of etzel-mes_
> camel-milo - a failed producer request can block the connection rebuild on
> Milo's shared scheduled executor
> -----------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25713
> URL: https://issues.apache.org/jira/browse/CAMEL-25713
> Project: Camel
> Issue Type: Bug
> Components: camel-milo
> Reporter: Serdar Gökay
> Assignee: Serdar Gökay
> Priority: Minor
>
> Found while testing CAMEL-25691 (PR
> https://github.com/apache/camel/pull/27910) against a real server; that PR
> does not change this path.
> When a write, method call, read or browse of {{milo-client}} or
> {{milo-browse}} fails, the {{handleAsync}} block of
> {{SubscriptionManager.write/call/readValues/browse}} calls
> {{handleConnectionFailure}} on {{executor}}, which {{MiloClientConnection}}
> sets to {{Stack.sharedScheduledExecutor()}}. {{handleConnectionFailure}}
> holds the lock of the manager and calls {{OpcUaClient.disconnect()}}. A
> CloseSession that is sent after the client noticed the lost connection waits
> until a reconnect attempt restores the channel or fails, and Milo schedules
> those attempts on the same shared scheduled executor. That executor is sized
> to the processor count, so on a host with one processor its only thread waits
> for a reconnect attempt that it would have to run itself: the connection is
> never rebuilt, the consumers stay silent after the server is reachable again,
> and every call that needs the lock (for example
> {{MiloClientConnection.getOpcUaClient()}}) waits.
> Reproduction: opc-plc 2.15.4, Camel and Milo defaults, the client in a
> container limited to one CPU, a producer writing every 2 s on the connection
> of a consumer, all traffic to the server dropped silently (iptables DROP) for
> 30 s. Upstream main (a16cccbde62) hung in 3 of 11 runs; the thread dump shows
> {{milo-shared-scheduled-executor-0}} in {{SubscriptionManager.write}} ->
> {{handleConnectionFailure}} -> {{OpcUaClient.disconnect()}}. From the logs it
> is a race: after the failed keep-alive Milo closes the channel, and a
> CloseSession that reaches the transport before the channel state machine has
> handled the close fails at once, while a later one waits for the reconnect.
> With two CPUs (writes every 500 ms) 4 of 4 runs recovered.
> Expected: the connection is rebuilt once the server can be reached again.
> Handling the failures of the producer requests on the client's executor
> ({{getTransport().getConfig().getExecutor()}}, Milo's shared cached pool), as
> CAMEL-25691 does for the recovery of the subscription, recovered in 9 of 9
> runs of the same test. It also moves the successful completions to that
> executor and needs a guard so that a stale failure does not start a second
> reconnect. The reconnect job ({{connect()}}) also runs on {{executor}};
> CAMEL-25692 proposes an executor owned by the component, which would cover
> both. The change touches the same {{handleAsync}} blocks as CAMEL-25690 and
> CAMEL-25694.
> I can send a PR.
> _Claude Code on behalf of etzel-mes_
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
