[ 
https://issues.apache.org/jira/browse/CAMEL-27088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

shashank updated CAMEL-27088:
-----------------------------
    Description: 
h3. What happens

The Delay EIP, the Throttle EIP with {{asyncDelayed}} and the Threads EIP hand 
the exchange to a task on their own thread pool and let the caller go on 
asynchronously; the task calls the callback later. When the route is removed, 
the processor's {{doShutdown}} shuts that pool down with {{shutdownNow}}, and 
the tasks still queued in it are dropped: nobody calls their callback.

This happens when a route is stopped with a timeout that runs out (the default 
graceful timeout, or {{stopRoute(id, timeout, unit)}}) and then removed, which 
is also what a route reload does. A delay longer than the timeout, a throttle 
with queued exchanges, or a busy Threads pool is enough. Then:
# a synchronous caller (a {{ProducerTemplate}} request, a {{direct}} caller) 
waits for ever; only a CamelContext stop releases it;
# the exchange stays in the in-flight registry and its unit of work is never 
done (no onCompletion, no commit or rollback).

A route stop alone is not affected: the pool keeps running, and the task fails 
the exchange with a RouteStoppingException later (CAMEL-25502).

Found with a TLA+ model of the protocol (task scheduled on the pool, pool 
thread runs it, route stop with graceful wait or timeout, route removal with 
shutdownNow). On main, "once the route is removed and nothing is left to run, 
every exchange that entered the route was completed" is violated with the 
trace: exchange scheduled, route stop, timeout, removal, the exchange is never 
completed. Failing every scheduled exchange at shutdown without a claim breaks 
"the callback is called at most once" (a task already taken by a pool thread 
still runs).

h3. How to reproduce

{code:java}
from("direct:delay").routeId("delay").delay(60000).asyncDelayed().to("mock:result");

CompletableFuture<Exchange> f = template.asyncSend("direct:delay", e -> 
e.getMessage().setBody("x"));
// wait until the exchange is in the route
context.getRouteController().stopRoute("delay", 100, TimeUnit.MILLISECONDS);
context.removeRoute("delay");
f.get(10, TimeUnit.SECONDS); // TimeoutException: the exchange is never 
completed
{code}

The same with 
{{throttle(1).totalRequestsMode().timePeriodMillis(60000).asyncDelayed()}} 
after a first exchange, {{throttle(1).concurrentRequestsMode().asyncDelayed()}} 
with a saturated pool, and {{threads(1, 1)}} with a busy thread.

h3. Fix

Delay and Throttle keep the tasks they scheduled and that have not started in a 
concurrent set; the task and the shutdown each claim a task by removing it, and 
only the claimer acts (the shutdown fails the exchange with a 
RouteStoppingException and calls the callback). Threads rejects the tasks that 
shutdownNow returns (its task is already Rejectable). As for the parallel EIPs, 
this happens only when the EIP shut down its own thread pool; a user-supplied 
pool is not shut down, so its tasks still run and fail the exchange with a 
RouteStoppingException, as before.

The parallel Multicast, Split and Recipient List drop their queued parts the 
same way; that is a separate issue (linked).

_Filed with Claude Code on behalf of allthingssecurity._

  was:
h3. What happens

The Delay EIP, the Throttle EIP with {{asyncDelayed}} and the Threads EIP hand 
the exchange to a task on their own thread pool and let the caller go on 
asynchronously; the task calls the callback later. When the route is removed, 
the processor's {{doShutdown}} shuts that pool down with {{shutdownNow}}, and 
the tasks still queued in it are dropped: nobody calls their callback.

This happens when a route is stopped with a timeout that runs out (the default 
graceful timeout, or {{stopRoute(id, timeout, unit)}}) and then removed, which 
is also what a route reload does. A delay longer than the timeout, a throttle 
with queued exchanges, or a busy Threads pool is enough. Then:
# a synchronous caller (a {{ProducerTemplate}} request, a {{direct}} caller) 
waits for ever; only a CamelContext stop releases it;
# the exchange stays in the in-flight registry and its unit of work is never 
done (no onCompletion, no commit or rollback).

A route stop alone is not affected: the pool keeps running, and the task fails 
the exchange with a RouteStoppingException later (CAMEL-25502).

Found with a TLA+ model of the protocol (task scheduled on the pool, pool 
thread runs it, route stop with graceful wait or timeout, route removal with 
shutdownNow). On main, "once the route is removed and nothing is left to run, 
every exchange that entered the route was completed" is violated with the 
trace: exchange scheduled, route stop, timeout, removal, the exchange is never 
completed. Failing every scheduled exchange at shutdown without a claim breaks 
"the callback is called at most once" (a task already taken by a pool thread 
still runs).

h3. How to reproduce

{code:java}
from("direct:delay").routeId("delay").delay(60000).asyncDelayed().to("mock:result");

CompletableFuture<Exchange> f = template.asyncSend("direct:delay", e -> 
e.getMessage().setBody("x"));
// wait until the exchange is in the route
context.getRouteController().stopRoute("delay", 100, TimeUnit.MILLISECONDS);
context.removeRoute("delay");
f.get(10, TimeUnit.SECONDS); // TimeoutException: the exchange is never 
completed
{code}

The same with 
{{throttle(1).totalRequestsMode().timePeriodMillis(60000).asyncDelayed()}} 
after a first exchange, {{throttle(1).concurrentRequestsMode().asyncDelayed()}} 
with a saturated pool, and {{threads(1, 1)}} with a busy thread.

h3. Fix

Delay and Throttle keep the tasks they scheduled and that have not started in a 
concurrent set; the task and the shutdown each claim a task by removing it, and 
only the claimer acts (the shutdown fails the exchange with a 
RouteStoppingException and calls the callback). Threads rejects the tasks that 
shutdownNow returns (its task is already Rejectable).

The parallel Multicast, Split and Recipient List drop their queued parts the 
same way; that is a separate issue (linked).

_Filed with Claude Code on behalf of allthingssecurity._


> camel-core - Delay, Throttle (asyncDelayed) and Threads EIP: an exchange 
> queued in the thread pool when its route is removed after a forced stop is 
> never completed, and its caller waits for ever
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-27088
>                 URL: https://issues.apache.org/jira/browse/CAMEL-27088
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-core
>            Reporter: shashank
>            Assignee: shashank
>            Priority: Major
>
> h3. What happens
> The Delay EIP, the Throttle EIP with {{asyncDelayed}} and the Threads EIP 
> hand the exchange to a task on their own thread pool and let the caller go on 
> asynchronously; the task calls the callback later. When the route is removed, 
> the processor's {{doShutdown}} shuts that pool down with {{shutdownNow}}, and 
> the tasks still queued in it are dropped: nobody calls their callback.
> This happens when a route is stopped with a timeout that runs out (the 
> default graceful timeout, or {{stopRoute(id, timeout, unit)}}) and then 
> removed, which is also what a route reload does. A delay longer than the 
> timeout, a throttle with queued exchanges, or a busy Threads pool is enough. 
> Then:
> # a synchronous caller (a {{ProducerTemplate}} request, a {{direct}} caller) 
> waits for ever; only a CamelContext stop releases it;
> # the exchange stays in the in-flight registry and its unit of work is never 
> done (no onCompletion, no commit or rollback).
> A route stop alone is not affected: the pool keeps running, and the task 
> fails the exchange with a RouteStoppingException later (CAMEL-25502).
> Found with a TLA+ model of the protocol (task scheduled on the pool, pool 
> thread runs it, route stop with graceful wait or timeout, route removal with 
> shutdownNow). On main, "once the route is removed and nothing is left to run, 
> every exchange that entered the route was completed" is violated with the 
> trace: exchange scheduled, route stop, timeout, removal, the exchange is 
> never completed. Failing every scheduled exchange at shutdown without a claim 
> breaks "the callback is called at most once" (a task already taken by a pool 
> thread still runs).
> h3. How to reproduce
> {code:java}
> from("direct:delay").routeId("delay").delay(60000).asyncDelayed().to("mock:result");
> CompletableFuture<Exchange> f = template.asyncSend("direct:delay", e -> 
> e.getMessage().setBody("x"));
> // wait until the exchange is in the route
> context.getRouteController().stopRoute("delay", 100, TimeUnit.MILLISECONDS);
> context.removeRoute("delay");
> f.get(10, TimeUnit.SECONDS); // TimeoutException: the exchange is never 
> completed
> {code}
> The same with 
> {{throttle(1).totalRequestsMode().timePeriodMillis(60000).asyncDelayed()}} 
> after a first exchange, 
> {{throttle(1).concurrentRequestsMode().asyncDelayed()}} with a saturated 
> pool, and {{threads(1, 1)}} with a busy thread.
> h3. Fix
> Delay and Throttle keep the tasks they scheduled and that have not started in 
> a concurrent set; the task and the shutdown each claim a task by removing it, 
> and only the claimer acts (the shutdown fails the exchange with a 
> RouteStoppingException and calls the callback). Threads rejects the tasks 
> that shutdownNow returns (its task is already Rejectable). As for the 
> parallel EIPs, this happens only when the EIP shut down its own thread pool; 
> a user-supplied pool is not shut down, so its tasks still run and fail the 
> exchange with a RouteStoppingException, as before.
> The parallel Multicast, Split and Recipient List drop their queued parts the 
> same way; that is a separate issue (linked).
> _Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to