etzel-mes opened a new pull request, #28111: URL: https://github.com/apache/camel/pull/28111
# Description [CAMEL-25812](https://issues.apache.org/jira/browse/CAMEL-25812), the follow-up to #28049 (CAMEL-25713) that @davsclaus suggested in https://github.com/apache/camel/pull/28049#discussion_r4307398068. Since #28049, a `milo-client` or `milo-browse` connection handles the outcome of its write, method call, read and browse requests, and runs its reconnects, on a scheduled thread pool of its own with one thread per processor. The threads of a scheduled pool never end while the connection is open, and with virtual threads Camel builds that pool with a single worker. The change: - `MiloClientConnection` handles the request outcomes on a thread pool from `ExecutorServiceManager.newThreadPool` named `MiloClient`. It has up to one thread per processor, and its idle threads end after the keep-alive time of the default thread pool profile (60 s). With virtual threads and more than one processor, Camel builds it as a virtual thread per task. Its queue is not bounded (`Integer.MAX_VALUE`, because `-1` gives a `SynchronousQueue` that rejects when every thread is busy), and the rejected policy stays `Abort`, so a request that completes after the connection is closed still fails with a `RejectedExecutionException`. - The reconnects run on `newSingleThreadScheduledExecutor` named `MiloClientReconnect`. Its thread starts with the first reconnect, and the reconnects run one after the other; before, two connects could run at the same time. They no longer wait in the queue behind the outcomes of the failed requests. - `SubscriptionManager` takes the two executors; only `triggerReconnect` uses the scheduled one. `dispose()` cancels a reconnect that is still waiting, so the `MiloClientReconnect` thread ends when the connection is closed instead of up to 10 s later. - `close()` shuts down the `MiloClient` pool even if shutting down the reconnect executor fails. - `MiloClientExecutorTest` gets a test that a connection that cannot connect has a `MiloClientReconnect` thread, and that the thread ends within 5 s after the route stops (it fails without the cancel in `dispose()`). The idle timeout and the virtual thread per task come from Camel's thread pool factory; they are covered by the measurements below, not by a unit test. - The 4.24 upgrade-guide section from #28049 describes the two pools. Neither executor is Milo's shared scheduled executor, so a failed request still disconnects without waiting for a thread that only the disconnect could free; `MiloClientExecutorTest` from #28049 still covers that. The values that a reconnect delivers to the consumers run their routes on the reconnect thread, and an awaited request from such a route completes on the `MiloClient` pool. On #28049, with one processor or with virtual threads, the reconnect and that completion shared the single worker of the pool. Because a reconnect no longer queues behind the outcomes of the failed requests, a failure that is handled after the reconnect can now dispose the new connection, which then reconnects once more. This is not new: a failure handler does not check that it belongs to the current connection. In the reconnect probe below it shows as one more connect with one processor and awaited writes (3 instead of 2), and the connection still recovers. I plan to fix that check separately, after #27956, which changes the same producer code. I first tried a cached pool for the request outcomes, as suggested, but it does not work with platform threads. When the connection is lost, every request in flight fails, and each failure waits for the lock of the connection while the first failure disconnects. With four producers sending writes that are not awaited, and the server frozen for 30 s, the cached pool started about 4,080 threads until the JVM could not create more (`pthread_create failed (EAGAIN)`), and the connection was never rebuilt. So with platform threads the pool keeps the bound of #28049 and only gains the idle timeout; with virtual threads, it starts a thread per task. Measured with host JVMs (Java 21, client limited with `-XX:ActiveProcessorCount=8`, against a local Milo server, two runs each): | | main | #28049 | this PR | |---|---|---|---| | 30 idle connections, `MiloClient` threads 5 s / 65 s after the load | 0 / 0 | 240 / 240 | 240 / 0 | | platform threads of the JVM 65 s after the load | 44–47 | 282–291 | 42–43 | | virtual threads, 1 connection × 32 callers, 10 ms of work after each awaited write | 668–671/s | 85/s | 2,121–2,301/s | | platform threads, same | 641–681/s | 671–682/s | 670–672/s | | 30 connections × 4 callers, 10 ms of work, platform threads (*) | 677–692/s | 9,166–9,585/s | 8,588–9,543/s | | same, virtual threads (*) | 683–685/s | 2,347–2,555/s | 7,686–8,612/s | | 30 connections × 1 caller, no work after the write, platform threads (*) | – | 16,644–21,040/s | 16,065–23,342/s | | same, virtual threads (*) | – | 16,416–25,225/s | 24,382–28,150/s | (*) #28049 and this PR alternated run by run, three runs each, each started only when the 1-minute load of the host was below 6. Main is from the earlier matrix; its runs without work varied twofold on the busier host, so they are left out. A reconnect probe froze the server JVM (`SIGSTOP`, so the sockets stay open and nothing answers) for 30 s, while one connection had a consumer and four producers, and checked when the consumer received values again after the server resumed. This PR recovered in every run, 1–6 s after the resume: with 1 and 8 processors, with awaited writes and with writes that are not awaited, and with virtual threads (8 processors). Its peak was 135 platform threads. On main, with one processor and awaited writes, the probe also reproduced CAMEL-25713: `milo-shared-scheduled-executor-0` waited in `OpcUaClient.disconnect()` from `handleConnectionFailure` of a write, and the connection was not rebuilt. Against a real server, opc-plc 2.15.4 in Docker, a client with a consumer and a producer that does an awaited write every 2 s lost its traffic silently (an iptables DROP) for 30 s, while #28049 ran beside it on a second connection, positions alternating. This PR recovered in all 16 runs (6 with 1 processor, 6 with virtual threads and 2 processors, and 4 more with 1 processor on the final commit), 1–12 s after the traffic was restored, in the same range as #28049 in the same runs. With 1 processor it more often needed a third connect, the race described above. `mvn -Psourcecheck clean verify` in `components/camel-milo` passes (82 tests, 1 skipped; #28049 has 81), and the module build regenerates no files. @davsclaus @oscerd could you take a look? # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. _Claude Code on behalf of etzel-mes_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
