Serdar Gökay created CAMEL-25823:
------------------------------------
Summary: camel-core - Split and Multicast start their child
producers before CamelContext starts, so with the supervising route controller
a producer that cannot start yet fails the whole context
Key: CAMEL-25823
URL: https://issues.apache.org/jira/browse/CAMEL-25823
Project: Camel
Issue Type: Bug
Components: camel-core
Affects Versions: 4.22.1, 4.23.0
Reporter: Serdar Gökay
With the supervising route controller enabled, a route whose producer cannot
start yet (for example its broker is down) is retried with back-off, and Camel
starts. This works for a {{to}} directly in the route. In a quick check on
4.23.0 Camel also started, and the route ended up Started, with the {{to}}
inside filter, choice, doTry, loop, loadBalance and wireTap, and with enrich
and recipientList (which create their producer at runtime). When the same
{{to}} is inside {{split}} or {{multicast}}, the producer is started during
route initialization, before the CamelContext starts and outside the
supervising controller. The start failure then fails the CamelContext, and the
route is never supervised.
*Reproduction* (plain camel-core unit test, no external system;
SupervisingRouteControllerMulticastProducerStartTest, ready for a PR): a
component whose producer fails its first two {{doStart}} calls with "Cannot
connect", supervising controller enabled (backOffDelay 25, backOffMaxAttempts
10, initialDelay 10).
||Route||Result||
|{{from(direct).to(flaky)}}|Camel starts, route retried and Started after 2
failed starts|
|{{from(direct).split(body()).to(flaky)}}|{{context.start()}} throws
{{IllegalStateException: Cannot connect}}|
|{{from(direct).multicast().to(flaky)}}|{{context.start()}} throws
{{IllegalStateException: Cannot connect}}|
Stack of the split case:
{noformat}
IllegalStateException: Cannot connect
at ...FlakyComponent$1$1.doStart
at
org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:129)
at
org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.start(AsyncProcessorConverterHelper.java:116)
at
org.apache.camel.impl.engine.DeferServiceStartupListener.doStart(DeferServiceStartupListener.java:63)
at
org.apache.camel.impl.engine.DeferServiceStartupListener.onCamelContextStarting(DeferServiceStartupListener.java:49)
at
org.apache.camel.impl.engine.AbstractCamelContext.doStartCamel(AbstractCamelContext.java:3190)
...
at
org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java:214)
{noformat}
We first hit this with a camel-paho-mqtt5 producer inside a split while the
broker host did not resolve; the same {{to}} outside the split was retried by
the controller. Any producer that connects in {{doStart}} behaves the same way.
*Cause* (main 04333431a65):
# {{MulticastProcessor.doInit}} (MulticastProcessor.java:313-321) calls
{{wrapInErrorHandler}} for every child processor. Split and multicast both use
it.
# {{wrapInErrorHandler}} starts the wrapped child right away: "must start the
error handler", {{ServiceHelper.startService(answer)}}
(MulticastProcessor.java:1213-1214). This starts the child's {{SendProcessor}}.
# {{SendProcessor.doStart}} creates the producer and adds it with
{{camelContext.addService(producer, true, true)}} (SendProcessor.java:335-350).
The context is not started yet, so {{InternalServiceManager}} defers it as an
early service (InternalServiceManager.java:164).
# {{DeferServiceStartupListener.onCamelContextStarting}} starts the early
services before any route starts, outside
{{DefaultSupervisingRouteController}}, and the exception fails the context.
For a {{to}} directly in the route, the producer is started by the route start,
which the supervising controller runs and retries.
*History*: The start inside {{wrapInErrorHandler}} dates from CAMEL-2360 /
CAMEL-3497, when the error handlers were created while processing the first
exchange, so the context and route were already started. Commit 7d70ad80c7d
("Support for the splitter eip in lightweight mode", 2020) moved the creation
into {{doInit}} for the lightweight CamelContext. CAMEL-19292 removed the
lightweight context in 4.0, and the eager creation in {{doInit}} stayed.
*Related*:
* CAMEL-18319: the supervising controller should not warm up routes during the
normal startup, so that start errors stay under its control. This is a
remaining eager start.
* CAMEL-21319: the test added there
(SmppTRXProducerSupervisingRouteControllerIT) shows a producer directly in the
route being restarted by the controller. {{lazyStartProducer}} was suggested
there as an alternative; it also avoids this failure, but the route then starts
and every message fails until the producer works, instead of the route being
supervised the same way as in the other EIPs.
* CAMEL-21775 (split/multicast in onException with the supervising controller)
is a different problem.
*Possible direction* (not tried): in {{doInit}} only create and initialize the
error handlers, and start them in {{doStart}}, which already starts
{{processors}} (MulticastProcessor.java:1280) and runs under the route start. I
can open a PR with the test, and with the fix if this direction is acceptable.
_Claude Code on behalf of etzel-mes_
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
