Serdar Gökay created CAMEL-25713:
------------------------------------
Summary: camel-milo - a failed producer request can block the
connection rebuild on Milo's shared scheduled executor
Key: CAMEL-25713
URL: https://issues.apache.org/jira/browse/CAMEL-25713
Project: Camel
Issue Type: Bug
Components: camel-milo
Reporter: Serdar Gökay
Found while testing CAMEL-25691 (PR https://github.com/apache/camel/pull/27910)
against a real server; that PR does not change this path.
When a write, method call, read or browse of {{milo-client}} or {{milo-browse}}
fails, the {{handleAsync}} block of
{{SubscriptionManager.write/call/readValues/browse}} calls
{{handleConnectionFailure}} on {{executor}}, which {{MiloClientConnection}}
sets to {{Stack.sharedScheduledExecutor()}}. {{handleConnectionFailure}} holds
the lock of the manager and calls {{OpcUaClient.disconnect()}}. A CloseSession
that is sent after the client noticed the lost connection waits until a
reconnect attempt restores the channel or fails, and Milo schedules those
attempts on the same shared scheduled executor. That executor is sized to the
processor count, so on a host with one processor its only thread waits for a
reconnect attempt that it would have to run itself: the connection is never
rebuilt, the consumers stay silent after the server is reachable again, and
every call that needs the lock (for example
{{MiloClientConnection.getOpcUaClient()}}) waits.
Reproduction: opc-plc 2.15.4, Camel and Milo defaults, the client in a
container limited to one CPU, a producer writing every 2 s on the connection of
a consumer, all traffic to the server dropped silently (iptables DROP) for 30
s. Upstream main (a16cccbde62) hung in 3 of 11 runs; the thread dump shows
{{milo-shared-scheduled-executor-0}} in {{SubscriptionManager.write}} ->
{{handleConnectionFailure}} -> {{OpcUaClient.disconnect()}}. From the logs it
is a race: after the failed keep-alive Milo closes the channel, and a
CloseSession that reaches the transport before the channel state machine has
handled the close fails at once, while a later one waits for the reconnect.
With two CPUs (writes every 500 ms) 4 of 4 runs recovered.
Expected: the connection is rebuilt once the server can be reached again.
Handling the failures of the producer requests on the client's executor
({{getTransport().getConfig().getExecutor()}}, Milo's shared cached pool), as
CAMEL-25691 does for the recovery of the subscription, recovered in 9 of 9 runs
of the same test. It also moves the successful completions to that executor and
needs a guard so that a stale failure does not start a second reconnect. The
reconnect job ({{connect()}}) also runs on {{executor}}; CAMEL-25692 proposes
an executor owned by the component, which would cover both. The change touches
the same {{handleAsync}} blocks as CAMEL-25690 and CAMEL-25694.
I can send a PR.
_Claude Code on behalf of etzel-mes_
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
