saravanakumar1987 opened a new pull request, #30066:
URL: https://github.com/apache/camel/pull/30066

   # Description
   
   Fixes [CAMEL-26971](https://issues.apache.org/jira/browse/CAMEL-26971).
   
   ## What was wrong
   
   Two defects in how `AlibabaOtsEndpoint` creates its `SyncClient`. They were 
listed in the description of apache/camel#29991 (CAMEL-23832) and found by 
reading the code.
   
   1. `AlibabaOtsEndpoint.initClient`, which `AlibabaOtsProducer.process` calls 
for each exchange, read and wrote the `otsClient` field without a lock, and the 
field was not volatile. Messages processed at the same time before a client 
existed could each create a `SyncClient`. The endpoint kept only the last one, 
and `doStop` shut down only that one, so the others were never shut down.
   
   2. `AlibabaOtsUtils.createClient` called `new SyncClient(endpoint, 
accessKey, secretKey, instanceName)`. In Tablestore SDK 5.17.14, this 
constructor ends in `this(endpoint, 
CredentialsProviderFactory.newDefaultCredentialProvider(...), instanceName, 
config, new ResourceManager(config, callbackExecutor))` 
(`InternalClient.java:124`), so the `ResourceManager` exists before the 
constructor that it calls checks anything. Creating it creates 
`ClientResources`, whose `AsyncServiceClient` starts the reactor thread of the 
HTTP client, which starts the I/O dispatcher threads (one per CPU by default), 
and an `IdleConnectionEvictor` thread. Nothing shuts it down when that 
constructor then rejects the endpoint or the instance name:
      - in its first checks (`InternalClient.java:130-140`): an endpoint that 
does not start with `http://` or `https://`, or an instance name with a 
multibyte character or a character other than letters, digits, `_` and `-`;
      - later, when `new LauncherFactory(...)` creates the `OTSUri` objects 
(`InternalClient.java:167`): an endpoint that does not match 
`Constants.ENDPOINT_REGEX`, for example one with a path.
   
      The field stays null, so the creation was tried again for each message, 
and each message left these threads behind: 18 threads per message on the 
16-CPU machine of the tests below. The reactor and I/O dispatcher threads are 
not daemon threads, so they can also keep a JVM from exiting.
   
   ## The change
   
   ### One client per endpoint
   
   - `otsClient` is volatile. `initClient` reads it once; when it is null, it 
takes the `lock` that the endpoint inherits from `BaseService`, reads the field 
again, and creates the client only if it is still null. This is the lazy 
creation of `MilvusEndpoint.getClient`, `QdrantEndpoint.getClient` and 
`WeaviateVectorDbEndpoint.getClient`, which create their client when their 
producer starts and close it in `doStop`. Some other endpoints use a private 
lock for the same pattern; the inherited one also makes the creation and 
`doStop` exclude each other. Unlike those three, `initClient` returns the value 
it read or created, not the field read again after the lock is released. Once 
the client exists, a message does not take the lock. The client is still 
created when the producer processes its first message, as the description of 
the `otsClient` option says.
   - `setOtsClient` and `doStop` take the same lock, so the client and the 
`autowiredOtsClient` flag change together, and `doStop` cannot run in the 
middle of a creation. `BaseService.stop()` already holds this lock when it 
calls `doStop`, and the lock is reentrant.
   - The creation is in a new package-private method `createClient()`, which 
calls `AlibabaOtsUtils.createClient(this)`. The concurrency test overrides it 
to hold the creation.
   
   ### No SDK threads left behind by a failed creation
   
   - `AlibabaOtsUtils.createClient` creates the `CredentialsProvider` first 
(`CredentialsProviderFactory.newDefaultCredentialProvider`, which rejects an 
empty AccessKey ID or secret and an AccessKey ID of an invalid format; as 
before, these are rejected before any thread is started), then one default 
`ClientConfiguration`, then `new ResourceManager(configuration)`, and passes 
the three to the public constructor `SyncClient(endpoint, CredentialsProvider, 
instanceName, ClientConfiguration, ResourceManager)`. When this constructor 
throws, it shuts the `ResourceManager` down and rethrows the exception (a 
failure of the shutdown is added to it as suppressed).
   - A `ResourceManager` created with its `ClientConfiguration` constructor 
owns its resources (`owned = true`), so `SyncClient.shutdown()` still releases 
them through `InternalClient.shutdown()` and `ResourceManager.shutdown()` when 
the Camel endpoint stops.
   - For valid options, the client is the same as before: a default 
`ClientConfiguration`, no STS token, and the callback executor that the SDK 
creates. Before, the `ResourceManager` and the client each had their own 
default `ClientConfiguration`; now they share one.
   - Passing `null` as the `ResourceManager` is not enough: `InternalClient` 
then runs its first checks before it creates a `ResourceManager`, but it 
creates one before `OTSUri` rejects an endpoint, so an endpoint with a path 
still leaks (verified, see below). The ticket also suggests checking the 
endpoint and the instance name before the `SyncClient` constructor is called; 
that would copy the rules of the SDK (prefix, multibyte characters, the pattern 
of the instance name, `ENDPOINT_REGEX`) into Camel, where they could drift from 
the SDK, while shutting the `ResourceManager` down covers every failure of the 
constructor.
   
   No option, header or default changes, so there is no upgrade-guide entry, 
and no generated file changes.
   
   ## Tests
   
   Two new tests in `AlibabaOtsEndpointTest`, and `org.awaitility:awaitility` 
as a test dependency of the module:
   
   - `concurrentInitClientCreatesOneClient`: eight threads call `initClient` on 
an endpoint whose `createClient` records the calling thread, waits on a latch, 
and returns a new mock `SyncClient`. The latch is released only when each 
caller has either entered `createClient` or waits for a lock that a thread in 
`createClient` holds (`ThreadMXBean.getThreadInfo(...).getLockOwnerId()`), so 
the outcome does not depend on how the threads are scheduled: without the lock, 
all eight enter `createClient`; with it, one does and seven wait. The test then 
checks that one client was created, that each caller got it, that the endpoint 
holds it, and that `doStop` shuts it down.
   - `failedClientCreationLeavesNoSdkThread`, for 
`my-instance.cn-hangzhou.ots.aliyuncs.com` (rejected by the first checks of 
`InternalClient`) and `https://my-instance.cn-hangzhou.ots.aliyuncs.com/path` 
(rejected by `OTSUri`): it calls `initClient` three times, checks that each 
call throws an `IllegalArgumentException` and that the endpoint keeps no 
client, then waits, with Awaitility for at most 10 seconds, until no SDK thread 
is alive besides those that were alive before the test. An SDK thread is a live 
thread whose class, or a frame of whose stack, is in 
`com.alicloud.openservices.tablestore.` or in `com.aliyun.ots.`, the package of 
the HTTP client that the SDK shades (artifact 
`tablestore-shaded-httpasyncclient`). The thread names cannot be used: the 
evictor and the reactor have the JDK default names `Thread-N` and 
`pool-N-thread-1`, and the `tablestore-` pools of the SDK start no thread 
before a request is sent.
   
   ## How it was verified
   
   - Without the fixes, with only `createClient()` added to the endpoint so 
that the first test can run:
     - `concurrentInitClientCreatesOneClient` fails with `Expected size: 1 but 
was: 8`. Run 200 times (with `@RepeatedTest(200)`, not committed), it failed 
200 times, each time with 8 clients.
     - `failedClientCreationLeavesNoSdkThread` fails for both values with a 
`ConditionTimeoutException` after 10 seconds, which lists 54 extra threads per 
value: for each of the three calls, the `IdleConnectionEvictor` (`Thread-N`), 
the reactor thread (`pool-N-thread-1`) and 16 `I/O dispatcher N` threads.
   - With only the fix of `initClient`, the first test passes and the second 
still fails for both values.
   - With a `createClient` that passes `null` as the `ResourceManager` (not 
committed), the second test passes for the endpoint without `https://` and 
fails for the endpoint with a path, with the same 54 extra threads.
   - With both fixes, both tests pass. Run 200 times, the first test passed 200 
times.
   - With both fixes, all 13 tests of camel-alibaba-ots pass, with no failure, 
error or rerun.
   
   ## Limits, in the SDK
   
   - `AsyncServiceClient.shutdown()` of the SDK ignores an 
`InterruptedException` from joining its evictor thread and then does not close 
its HTTP client. So when the thread that shuts the client down is interrupted, 
the reactor and I/O dispatcher threads stay alive, and the interrupt flag is 
lost. This applies to the shutdown after a failed creation as it already did to 
`SyncClient.shutdown()` in `doStop`.
   - In the shaded httpcore-nio 4.3.2, a reactor that is shut down before its 
thread has started to run does not close its `Selector`. A failed creation 
shuts the `ResourceManager` down right after it is created, so this can happen: 
in 1,600 failed creations of a scratch program that repeats the new failure 
path (not committed), no thread was left, but 69 `Selector`s stayed open (42 of 
800 for the endpoint without a scheme, 27 of 800 for the endpoint with a path). 
Before the change, every failed creation left the threads and this `Selector` 
open.
   - A message to an endpoint whose client cannot be created still creates and 
shuts down a `ResourceManager`, which starts and stops these threads. In 800 of 
these failed creations, the shutdown took about 20 ms on average, but twice 
more than one second (1.5 s and 2.5 s), and the messages that wait for the lock 
in `initClient` wait for it.
   
   ## Other Alibaba endpoints
   
   The other Alibaba endpoints create their client with the same unsynchronized 
check-then-create in `initClient`. They are not changed here:
   
   | Module | Method | Their `doStop` |
   |---|---|---|
   | camel-alibaba-eventbridge | `AlibabaEventBridgeEndpoint.initClient` | 
drops the reference |
   | camel-alibaba-fc | `FCEndpoint.initClient` | drops the reference |
   | camel-alibaba-kms | `KMSEndpoint.initClient` | drops the reference |
   | camel-alibaba-mns | `MNSEndpoint.initClient` | closes the client |
   | camel-alibaba-oss | `OSSEndpoint.initClient` | closes the client |
   | camel-alibaba-sls | `AlibabaSlsEndpoint.initClient` | drops the reference |
   | camel-alibaba-sms | `SMSEndpoint.initClient` | drops the reference |
   
   With mns and oss, a client created by a second message at the same time is 
never closed, as it was with ots. Whether their SDKs also leave threads behind 
when a creation fails was not checked.
   
   # Target
   
   - [x] I checked that the commit is targeting the correct branch (Camel 4 
uses the `main` branch)
   
   # Tracking
   
   - [x] If this is a large change, bug fix, or code improvement, I checked 
there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for 
the change (usually before you start working on it).
   
   # Apache Camel coding standards and style
   
   - [x] I checked that each commit in the pull request has a meaningful 
subject line and body.
   - [x] I have run `mvn clean install -Dquickly -DskipTests` from the root 
folder on upstream `main` (31e144597200, the parent of this commit), and then, 
with the change, `mvn clean install -Psourcecheck` in camel-alibaba-ots, which 
also runs its code generators, the formatter and the import sorter. They 
changed no file, so there are no auto-generated changes to commit.
   
   The commands that were run:
   
   ```
   # in components/camel-alibaba/camel-alibaba-ots, without the fixes and with 
only the fix of initClient
   mvn -o -B test 
-Dtest='AlibabaOtsEndpointTest#concurrentInitClientCreatesOneClient+failedClientCreationLeavesNoSdkThread'
 -Dsurefire.failIfNoSpecifiedTests=false
   # with the null ResourceManager variant
   mvn -o -B test 
-Dtest='AlibabaOtsEndpointTest#failedClientCreationLeavesNoSdkThread' 
-Dsurefire.failIfNoSpecifiedTests=false -Dsurefire.rerunFailingTestsCount=0
   # with both fixes
   mvn -o -B test 
-Dtest='AlibabaOtsEndpointTest#concurrentInitClientCreatesOneClient+failedClientCreationLeavesNoSdkThread'
 -Dsurefire.failIfNoSpecifiedTests=false -Dsurefire.rerunFailingTestsCount=0
   # with @RepeatedTest(200) on the first test, with both fixes and without the 
lock
   mvn -o -B test 
-Dtest='AlibabaOtsEndpointTest#concurrentInitClientCreatesOneClient' 
-Dsurefire.failIfNoSpecifiedTests=false -Dsurefire.rerunFailingTestsCount=0
   # with both fixes
   mvn -o -B formatter:format impsort:sort
   mvn -o -B clean install -Psourcecheck
   ```
   
   # AI-assisted contributions
   
   - [x] If this PR includes AI-generated code, commits have proper 
co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR 
description identifies the AI tool used.
   
   The change, the tests and this text were drafted with Claude Code 
(Anthropic), from the ticket and the sources of Tablestore SDK 5.17.14. The 
tests were run as listed above. The claims about the SDK and the change were 
checked by independent AI reviews.
   
   _Claude Code on behalf of saravanakumar1987_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to