shashank created CAMEL-26538:
--------------------------------
Summary: camel-support - API components: stopping one CamelContext
breaks the API calls of the other CamelContexts (shared static thread pool)
Key: CAMEL-26538
URL: https://issues.apache.org/jira/browse/CAMEL-26538
Project: Camel
Issue Type: Bug
Components: camel-support, camel-box, camel-google-drive,
camel-zendesk
Reporter: shashank
Assignee: shashank
Found with a TLA+ model of the thread pool that {{AbstractApiEndpoint}} shares
between the CamelContexts of a JVM, then reproduced with a test on the real
classes.
The producers of the API components submit each call to a thread pool
({{AbstractApiProducer.process}}:
{{endpoint.getExecutorService().submit(invocation)}}). This affects 11
components: as2, box, braintree, dhis2, fhir, google-calendar, google-drive,
google-mail, google-sheets, twilio and zendesk. olingo2 and olingo4 have their
own {{process}} and do not use the pool.
The pool is kept in a static map, {{EXECUTOR_SERVICE_MAP}}, with one pool per
endpoint class for the whole JVM. The ExecutorServiceManager of the first
CamelContext that calls the API creates it, so that CamelContext shuts it down
when it stops. Each endpoint also keeps the pool in a field and never checks it
again ({{getExecutorService()}}: {{if (executorService == null)}}).
When a JVM has two CamelContexts that use the same API component, stopping the
first one breaks the second one:
* Calls of the second CamelContext that are still running when the shutdown
await time ends ({{shutdownAwaitTermination}}, 10 seconds by default) are
interrupted ({{shutdownNow}}).
* Every later call of the second CamelContext is dropped. The default thread
pool profile uses {{CallerRuns}}, and the JDK {{CallerRunsPolicy}} discards a
task submitted to a pool that is shut down. So the call never runs, the
callback is never called, and the exchange never completes. A synchronous
caller (the thread of a synchronous consumer, {{template.request}}) waits for
good, and nothing is logged. With an {{Abort}} profile, the call fails with a
{{RejectedExecutionException}} instead.
{noformat}
1 context A : sends to google-drive -> creates the pool (owned by A), puts it
in the static map
2 context B : sends to google-drive -> takes A's pool from the static map,
keeps it in its endpoint
3 context A : stop() -> A's ExecutorServiceManager shuts the pool down
4 context B : sends again -> submit() to the shut down pool -> CallerRuns drops
the task -> the exchange hangs
{noformat}
This happens with any two CamelContexts in one JVM, for example several
CamelContexts in one application, or one of several deployments on a shared
runtime being undeployed. An endpoint created after step 3 is not affected: the
static lookup sees that the pool is shut down and creates a new one. Only the
endpoints that kept the pool before the stop are broken, and the routes of a
running CamelContext keep their endpoints.
An endpoint object that is kept across a stop and start of its own CamelContext
breaks the same way. For example, a route built with the endpoint instance
({{to(endpoint)}}) keeps it, and that route's next call is dropped. Routes that
use a URI get a new endpoint when the CamelContext starts again, so they are
not affected.
Reproduced with:
* a stub API component in camel-core: step 4 never completes, a call running
during step 3 is interrupted, the endpoint kept across a restart never
completes its call, and endpoints of two CamelContexts that look up the pool at
the same time share pools across the CamelContexts (and one CamelContext ends
up with several pools);
* camel-google-drive with a mock HTTP transport, where the send of the second
CamelContext after the first one stopped never completes.
*Model.* A TLA+ model with two CamelContexts, one endpoint each, checks that an
exchange of a running CamelContext is never dropped and that stopping one
CamelContext never interrupts a call of another. Both fail on main (11-step
traces), and both also fail if the endpoint only re-checks the pool it kept
(one pool per JVM). They hold with one pool per CamelContext.
*Proposal*
* Keep the pools per CamelContext, for example in a context plugin that holds
one pool per endpoint class. Create each pool through that CamelContext's
ExecutorServiceManager, as now.
* Let the endpoint look up the pool again when the one it kept was shut down.
This covers an endpoint that is used again after its CamelContext was started
again.
* Each CamelContext then has its own pool. The size of each comes from the
thread pool profile of the component (such as {{CamelGoogleDrive}}) or the
default profile of its own CamelContext. Add an upgrade guide note.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
