NITESH KUMAR created CAMEL-23975:
------------------------------------
Summary: java.util.concurrent.RejectedExecutionException:
CamelContext is stopped : at runTime
Key: CAMEL-23975
URL: https://issues.apache.org/jira/browse/CAMEL-23975
Project: Camel
Issue Type: Bug
Environment: Prod
Reporter: NITESH KUMAR
Hi Team,
I am facing an issue with an Apache Camel application running on Spring Boot in
a clustered environment and would appreciate your guidance.
*🔧* *Setup Details*
* *Framework:* Spring Boot(3.5x) with Apache Camel (4.10.9)
* *Deployment:* 5 EC2 instances running the same service
* *Caching:* Embedded Hazelcast cluster (5.5.0)
* *Usage:* Hazelcast distributed IMap with a custom MapLoader implementation
*⚠️* *Issue Description*
During application startup, specifically while Hazelcast performs {*}initial
cache population{*}, we intermittently encounter the following exception:
java.util.concurrent.RejectedExecutionException: CamelContext is stopped
But My IMaps are LAZY and executes at the time of message processing indicates
routes & camel Context are up
Note : This Exception We Also gets intermittently when cache reloads after
eviction.
*🔍* *Stack Trace (Relevant Section)*
at
com.hazelcast.map.impl.MapStoreWrapper.lambda$load$7(MapStoreWrapper.java:176)
at
com.hazelcast.internal.namespace.NamespaceUtil.callWithClassLoader(NamespaceUtil.java:200)
at
com.hazelcast.internal.namespace.NamespaceUtil.callWithOwnClassLoader(NamespaceUtil.java:152)
at com.hazelcast.map.impl.MapStoreWrapper.load(MapStoreWrapper.java:176)
at
com.hazelcast.map.impl.mapstore.writethrough.WriteThroughStore.load(WriteThroughStore.java:88)
at
com.hazelcast.map.impl.mapstore.writethrough.WriteThroughStore.load(WriteThroughStore.java:31)
at
com.hazelcast.map.impl.recordstore.DefaultRecordStore.loadValueWithExpiry(DefaultRecordStore.java:494)
at
com.hazelcast.map.impl.operation.steps.GetOpSteps$2.runStep(GetOpSteps.java:58)
at
com.hazelcast.map.impl.operation.steps.GetOpSteps$2.runStep(GetOpSteps.java:48)
at
com.hazelcast.map.impl.operation.steps.engine.StepSupplier.runStepWithState(StepSupplier.java:203)
at
com.hazelcast.map.impl.operation.steps.engine.StepSupplier$1.run(StepSupplier.java:146)
at
com.hazelcast.internal.namespace.impl.NoOpUserCodeNamespaceService.runWithNamespace(NoOpUserCodeNamespaceService.java:82)
at
com.hazelcast.internal.namespace.NamespaceUtil.runWithNamespace(NamespaceUtil.java:106)
at
com.hazelcast.map.impl.operation.steps.engine.StepRunner.runDirect(StepRunner.java:234)
at
com.hazelcast.map.impl.operation.steps.engine.StepRunner.run(StepRunner.java:176)
at
com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:220)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
at
com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
at
com.hazelcast.internal.util.executor.PoolExecutorThreadFactory$ManagedThread.executeRun(PoolExecutorThreadFactory.java:74)
at
com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111)
Caused by: java.util.concurrent.RejectedExecutionException: CamelContext is
stopped
at
org.apache.camel.support.cache.DefaultProducerCache.send(DefaultProducerCache.java:155)
at
org.apache.camel.impl.engine.DefaultProducerTemplate.send(DefaultProducerTemplate.java:175)
at
org.apache.camel.impl.engine.DefaultProducerTemplate.send(DefaultProducerTemplate.java:171)
at
org.apache.camel.impl.engine.DefaultProducerTemplate.requestBody(DefaultProducerTemplate.java:413)
... 25 common frames omitted
*🔍* *Key Implementation Detail*
We are using a custom Hazelcast MapLoader, where:
* loadAllKeys() is triggered during first message processing as maps are lazy.
(indirectly camel is fully UP)
* Inside loadAllKeys(), we invoke a Camel route using ProducerTemplate
*Example:*
Java
@Override
public Iterable<K> loadAllKeys() {
producerTemplate.requestBody("direct:loadKeysRoute", payload);
return keys;
}
*🔍* *Observations*
* Issue occurs primarily during *initial cache loading*
* Sometimes observed during steady-state runtime
* Occurs across multiple EC2 nodes concurrently
* No explicit shutdown or stop is triggered from application code
* CamelContext stops automatically after the exception
* Suspected causes:
* Interaction between Hazelcast-managed threads and Camel execution
*📉* *Impact*
* Message processing is interrupted
*✅* *Request*
Kindly help with:
* Root cause analysis
* Recommended configuration or architectural changes
* Preventive measures to avoid CamelContext shutdown in such scenarios
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
