[
https://issues.apache.org/jira/browse/CAMEL-27088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shashank updated CAMEL-27088:
-----------------------------
Description:
h3. What happens
The Delay EIP, the Throttle EIP with {{asyncDelayed}} and the Threads EIP hand
the exchange to a task on their own thread pool and let the caller go on
asynchronously; the task calls the callback later. When the route is removed,
the processor's {{doShutdown}} shuts that pool down with {{shutdownNow}}, and
the tasks still queued in it are dropped: nobody calls their callback.
This happens when a route is stopped with a timeout that runs out (the default
graceful timeout, or {{stopRoute(id, timeout, unit)}}) and then removed, which
is also what a route reload does. A delay longer than the timeout, a throttle
with queued exchanges, or a busy Threads pool is enough. Then:
# a synchronous caller (a {{ProducerTemplate}} request, a {{direct}} caller)
waits for ever; only a CamelContext stop releases it;
# the exchange stays in the in-flight registry and its unit of work is never
done (no onCompletion, no commit or rollback).
A route stop alone is not affected: the pool keeps running, and the task fails
the exchange with a RouteStoppingException later (CAMEL-25502).
Found with a TLA+ model of the protocol (task scheduled on the pool, pool
thread runs it, route stop with graceful wait or timeout, route removal with
shutdownNow). On main, "once the route is removed and nothing is left to run,
every exchange that entered the route was completed" is violated with the
trace: exchange scheduled, route stop, timeout, removal, the exchange is never
completed. Failing every scheduled exchange at shutdown without a claim breaks
"the callback is called at most once" (a task already taken by a pool thread
still runs).
h3. How to reproduce
{code:java}
from("direct:delay").routeId("delay").delay(60000).asyncDelayed().to("mock:result");
CompletableFuture<Exchange> f = template.asyncSend("direct:delay", e ->
e.getMessage().setBody("x"));
// wait until the exchange is in the route
context.getRouteController().stopRoute("delay", 100, TimeUnit.MILLISECONDS);
context.removeRoute("delay");
f.get(10, TimeUnit.SECONDS); // TimeoutException: the exchange is never
completed
{code}
The same with
{{throttle(1).totalRequestsMode().timePeriodMillis(60000).asyncDelayed()}}
after a first exchange, {{throttle(1).concurrentRequestsMode().asyncDelayed()}}
with a saturated pool, and {{threads(1, 1)}} with a busy thread.
h3. Fix
Delay and Throttle keep the tasks they scheduled and that have not started in a
concurrent set; the task and the shutdown each claim a task by removing it, and
only the claimer acts (the shutdown fails the exchange with a
RouteStoppingException and calls the callback). Threads rejects the tasks that
shutdownNow returns (its task is already Rejectable). As for the parallel EIPs,
this happens only when the EIP shut down its own thread pool; a user-supplied
pool is not shut down, so its tasks still run and fail the exchange with a
RouteStoppingException, as before.
The parallel Multicast, Split and Recipient List drop their queued parts the
same way; that is a separate issue (linked).
_Filed with Claude Code on behalf of allthingssecurity._
was:
h3. What happens
The Delay EIP, the Throttle EIP with {{asyncDelayed}} and the Threads EIP hand
the exchange to a task on their own thread pool and let the caller go on
asynchronously; the task calls the callback later. When the route is removed,
the processor's {{doShutdown}} shuts that pool down with {{shutdownNow}}, and
the tasks still queued in it are dropped: nobody calls their callback.
This happens when a route is stopped with a timeout that runs out (the default
graceful timeout, or {{stopRoute(id, timeout, unit)}}) and then removed, which
is also what a route reload does. A delay longer than the timeout, a throttle
with queued exchanges, or a busy Threads pool is enough. Then:
# a synchronous caller (a {{ProducerTemplate}} request, a {{direct}} caller)
waits for ever; only a CamelContext stop releases it;
# the exchange stays in the in-flight registry and its unit of work is never
done (no onCompletion, no commit or rollback).
A route stop alone is not affected: the pool keeps running, and the task fails
the exchange with a RouteStoppingException later (CAMEL-25502).
Found with a TLA+ model of the protocol (task scheduled on the pool, pool
thread runs it, route stop with graceful wait or timeout, route removal with
shutdownNow). On main, "once the route is removed and nothing is left to run,
every exchange that entered the route was completed" is violated with the
trace: exchange scheduled, route stop, timeout, removal, the exchange is never
completed. Failing every scheduled exchange at shutdown without a claim breaks
"the callback is called at most once" (a task already taken by a pool thread
still runs).
h3. How to reproduce
{code:java}
from("direct:delay").routeId("delay").delay(60000).asyncDelayed().to("mock:result");
CompletableFuture<Exchange> f = template.asyncSend("direct:delay", e ->
e.getMessage().setBody("x"));
// wait until the exchange is in the route
context.getRouteController().stopRoute("delay", 100, TimeUnit.MILLISECONDS);
context.removeRoute("delay");
f.get(10, TimeUnit.SECONDS); // TimeoutException: the exchange is never
completed
{code}
The same with
{{throttle(1).totalRequestsMode().timePeriodMillis(60000).asyncDelayed()}}
after a first exchange, {{throttle(1).concurrentRequestsMode().asyncDelayed()}}
with a saturated pool, and {{threads(1, 1)}} with a busy thread.
h3. Fix
Delay and Throttle keep the tasks they scheduled and that have not started in a
concurrent set; the task and the shutdown each claim a task by removing it, and
only the claimer acts (the shutdown fails the exchange with a
RouteStoppingException and calls the callback). Threads rejects the tasks that
shutdownNow returns (its task is already Rejectable).
The parallel Multicast, Split and Recipient List drop their queued parts the
same way; that is a separate issue (linked).
_Filed with Claude Code on behalf of allthingssecurity._
> camel-core - Delay, Throttle (asyncDelayed) and Threads EIP: an exchange
> queued in the thread pool when its route is removed after a forced stop is
> never completed, and its caller waits for ever
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-27088
> URL: https://issues.apache.org/jira/browse/CAMEL-27088
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Reporter: shashank
> Assignee: shashank
> Priority: Major
>
> h3. What happens
> The Delay EIP, the Throttle EIP with {{asyncDelayed}} and the Threads EIP
> hand the exchange to a task on their own thread pool and let the caller go on
> asynchronously; the task calls the callback later. When the route is removed,
> the processor's {{doShutdown}} shuts that pool down with {{shutdownNow}}, and
> the tasks still queued in it are dropped: nobody calls their callback.
> This happens when a route is stopped with a timeout that runs out (the
> default graceful timeout, or {{stopRoute(id, timeout, unit)}}) and then
> removed, which is also what a route reload does. A delay longer than the
> timeout, a throttle with queued exchanges, or a busy Threads pool is enough.
> Then:
> # a synchronous caller (a {{ProducerTemplate}} request, a {{direct}} caller)
> waits for ever; only a CamelContext stop releases it;
> # the exchange stays in the in-flight registry and its unit of work is never
> done (no onCompletion, no commit or rollback).
> A route stop alone is not affected: the pool keeps running, and the task
> fails the exchange with a RouteStoppingException later (CAMEL-25502).
> Found with a TLA+ model of the protocol (task scheduled on the pool, pool
> thread runs it, route stop with graceful wait or timeout, route removal with
> shutdownNow). On main, "once the route is removed and nothing is left to run,
> every exchange that entered the route was completed" is violated with the
> trace: exchange scheduled, route stop, timeout, removal, the exchange is
> never completed. Failing every scheduled exchange at shutdown without a claim
> breaks "the callback is called at most once" (a task already taken by a pool
> thread still runs).
> h3. How to reproduce
> {code:java}
> from("direct:delay").routeId("delay").delay(60000).asyncDelayed().to("mock:result");
> CompletableFuture<Exchange> f = template.asyncSend("direct:delay", e ->
> e.getMessage().setBody("x"));
> // wait until the exchange is in the route
> context.getRouteController().stopRoute("delay", 100, TimeUnit.MILLISECONDS);
> context.removeRoute("delay");
> f.get(10, TimeUnit.SECONDS); // TimeoutException: the exchange is never
> completed
> {code}
> The same with
> {{throttle(1).totalRequestsMode().timePeriodMillis(60000).asyncDelayed()}}
> after a first exchange,
> {{throttle(1).concurrentRequestsMode().asyncDelayed()}} with a saturated
> pool, and {{threads(1, 1)}} with a busy thread.
> h3. Fix
> Delay and Throttle keep the tasks they scheduled and that have not started in
> a concurrent set; the task and the shutdown each claim a task by removing it,
> and only the claimer acts (the shutdown fails the exchange with a
> RouteStoppingException and calls the callback). Threads rejects the tasks
> that shutdownNow returns (its task is already Rejectable). As for the
> parallel EIPs, this happens only when the EIP shut down its own thread pool;
> a user-supplied pool is not shut down, so its tasks still run and fail the
> exchange with a RouteStoppingException, as before.
> The parallel Multicast, Split and Recipient List drop their queued parts the
> same way; that is a separate issue (linked).
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)