[
https://issues.apache.org/jira/browse/CAMEL-26554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Gonçalves updated CAMEL-26554:
------------------------------------
Description:
h3. Problem
{{JettyHttpComponent.disconnect(HttpConsumer, ConnectorRef)}} stops the Jetty
connector and server inside the {{CONNECTORS.computeIfPresent}} lambda:
{code:java}
if (connectorRef.decrement() == 0) {
connectorRef.server.removeConnector(connectorRef.connector);
connectorRef.connector.stop();
connectorRef.server.stop();
...
return null;
}
{code}
If any of these calls throws, the lambda never returns {{null}}, so the entry
for {{protocol:host:port}} stays in the static {{CONNECTORS}} map although its
listening socket is already closed.
{{connect(...)}} then reuses that entry for every later consumer on the same
port ("Using existing connector key"). It increments the ref count and
registers the servlet, without checking that the server is still running. The
route reports {{Started}}, but the port refuses connections. Only a JVM restart
clears it.
h3. How it happens in practice
Stopping a route group with {{DefaultShutdownStrategy.shutdown(context, routes,
timeout, unit)}} while an exchange is still in flight. On timeout the strategy
calls {{cancel(true)}} on its {{ShutdownTask}}. The task restores the interrupt
flag and goes on to force-stop the deferred consumers. Stopping the jetty
consumer on that interrupted thread makes Jetty's {{ManagedSelector.doStop}}
throw:
{noformat}
WARN DefaultShutdownStrategy - Error occurred while shutting down route: ...
This exception will be ignored.
java.lang.RuntimeException: java.lang.InterruptedException
at
org.eclipse.jetty.util.component.ContainerLifeCycle.remove(ContainerLifeCycle.java:730)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.removeBean(ContainerLifeCycle.java:697)
at org.eclipse.jetty.server.Server.removeConnector(Server.java:406)
at
org.apache.camel.component.jetty.JettyHttpComponent.disconnect(JettyHttpComponent.java:500)
at
org.apache.camel.component.jetty.JettyHttpComponent.lambda$disconnect$1(JettyHttpComponent.java:487)
at
java.base/java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1828)
at
org.apache.camel.component.jetty.JettyHttpComponent.disconnect(JettyHttpComponent.java:485)
at
org.apache.camel.http.common.HttpCommonEndpoint.disconnect(HttpCommonEndpoint.java:237)
at org.apache.camel.http.common.HttpConsumer.doStop(HttpConsumer.java:73)
...
at
org.apache.camel.impl.engine.DefaultShutdownStrategy.shutdownNow(DefaultShutdownStrategy.java:434)
at
org.apache.camel.impl.engine.DefaultShutdownStrategy$ShutdownTask.run(DefaultShutdownStrategy.java:743)
Caused by: java.lang.InterruptedException
at
java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1100)
at
java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
at org.eclipse.jetty.io.ManagedSelector.doStop(ManagedSelector.java:138)
...
at
org.eclipse.jetty.server.AbstractConnector.doStop(AbstractConnector.java:375)
{noformat}
After this, no "Started ServerConnector" is logged for that port again, while
every new jetty route on it logs as started.
The calling thread force-stops the same consumers at the same time, so whether
the interrupted {{ShutdownTask}} reaches the jetty consumer first is a race. It
happened to us on two separate occasions in production, both times during
startup when a route group failed and was stopped with a timeout.
h3. Reproducer
Attached: {{jetty-interrupted-disconnect-reproducer.tar.gz}} (plain Java +
Camel 4.10.9, JDK 21, about 100 lines). To be deterministic it stops the
consumer directly on a thread with the interrupt flag set, the state the
{{ShutdownTask}} is in at that point.
# Start route {{first}} on {{jetty:http://0.0.0.0:9001/first}} (only consumer
on the port).
# Stop its consumer on an interrupted thread, then remove the route.
# Start route {{second}} on {{jetty:http://0.0.0.0:9001/second}}.
# {{GET http://localhost:9001/second}}.
{noformat}
$ mvn compile exec:java
>>> GET /first -> 200 first
>>> stopping route first failed (ignored): java.lang.RuntimeException:
>>> java.lang.InterruptedException
>>> route second is Started, GET /second -> connection refused
>>> BUG REPRODUCED: route second is started but port 9001 refuses connections
$ mvn exec:java -Dexec.args=control (same steps, consumer stopped on a
normal thread)
>>> GET /first -> 200 first
>>> route second is Started, GET /second -> 200 second
>>> OK: route second answered
{noformat}
Also reproduced with the attached reproducer on 4.18.5, 4.22.2, 4.23.0 and
4.24.0-SNAPSHOT (main, built 2026-11-03, includes the CAMEL-25850 change), on
Jetty 12.1.12 / 12.1.14. Run against another version with:
{noformat}
mvn compile exec:java -Dcamel.version=4.23.0
{noformat}
h3. Possible fixes
* In {{disconnect}}, always drop the entry once the ref count reaches 0: stop
connector, server and thread pool in try/finally and return {{null}} even when
stopping fails, then rethrow or log.
* In {{connect}}, do not reuse an existing {{ConnectorRef}} whose server is not
running (e.g. {{!server.isRunning()}}). Create a new one instead.
h3. Related
* CAMEL-25850 has the same symptom (a port accepts no connection while its
routes stay started) with a different cause: a shared thread pool with
minThreads/maxThreads. Its fix does not change exception handling in
{{disconnect}}.
was:
h3. Problem
{{JettyHttpComponent.disconnect(HttpConsumer, ConnectorRef)}} stops the Jetty
connector and server inside the {{CONNECTORS.computeIfPresent}} lambda:
{code:java}
if (connectorRef.decrement() == 0) {
connectorRef.server.removeConnector(connectorRef.connector);
connectorRef.connector.stop();
connectorRef.server.stop();
...
return null;
}
{code}
If any of these calls throws, the lambda never returns {{null}}, so the entry
for {{protocol:host:port}} stays in the static {{CONNECTORS}} map although its
listening socket is already closed.
{{connect(...)}} then reuses that entry for every later consumer on the same
port ("Using existing connector key"). It increments the ref count and
registers the servlet, without checking that the server is still running. The
route reports {{Started}}, but the port refuses connections. Only a JVM restart
clears it.
h3. How it happens in practice
Stopping a route group with {{DefaultShutdownStrategy.shutdown(context, routes,
timeout, unit)}} while an exchange is still in flight. On timeout the strategy
calls {{cancel(true)}} on its {{ShutdownTask}}. The task restores the interrupt
flag and goes on to force-stop the deferred consumers. Stopping the jetty
consumer on that interrupted thread makes Jetty's {{ManagedSelector.doStop}}
throw:
{noformat}
WARN DefaultShutdownStrategy - Error occurred while shutting down route: ...
This exception will be ignored.
java.lang.RuntimeException: java.lang.InterruptedException
at
org.eclipse.jetty.util.component.ContainerLifeCycle.remove(ContainerLifeCycle.java:730)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.removeBean(ContainerLifeCycle.java:697)
at org.eclipse.jetty.server.Server.removeConnector(Server.java:406)
at
org.apache.camel.component.jetty.JettyHttpComponent.disconnect(JettyHttpComponent.java:500)
at
org.apache.camel.component.jetty.JettyHttpComponent.lambda$disconnect$1(JettyHttpComponent.java:487)
at
java.base/java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1828)
at
org.apache.camel.component.jetty.JettyHttpComponent.disconnect(JettyHttpComponent.java:485)
at
org.apache.camel.http.common.HttpCommonEndpoint.disconnect(HttpCommonEndpoint.java:237)
at org.apache.camel.http.common.HttpConsumer.doStop(HttpConsumer.java:73)
...
at
org.apache.camel.impl.engine.DefaultShutdownStrategy.shutdownNow(DefaultShutdownStrategy.java:434)
at
org.apache.camel.impl.engine.DefaultShutdownStrategy$ShutdownTask.run(DefaultShutdownStrategy.java:743)
Caused by: java.lang.InterruptedException
at
java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1100)
at
java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
at org.eclipse.jetty.io.ManagedSelector.doStop(ManagedSelector.java:138)
...
at
org.eclipse.jetty.server.AbstractConnector.doStop(AbstractConnector.java:375)
{noformat}
After this, no "Started ServerConnector" is logged for that port again, while
every new jetty route on it logs as started.
The calling thread force-stops the same consumers at the same time, so whether
the interrupted {{ShutdownTask}} reaches the jetty consumer first is a race. It
happened to us on two separate occasions in production, both times during
startup when a route group failed and was stopped with a timeout.
h3. Reproducer
Attached: {{jetty-interrupted-disconnect-reproducer.tar.gz}} (plain Java +
Camel 4.10.9, JDK 21, about 100 lines). To be deterministic it stops the
consumer directly on a thread with the interrupt flag set, the state the
{{ShutdownTask}} is in at that point.
# Start route {{first}} on {{jetty:http://0.0.0.0:9001/first}} (only consumer
on the port).
# Stop its consumer on an interrupted thread, then remove the route.
# Start route {{second}} on {{jetty:http://0.0.0.0:9001/second}}.
# {{GET http://localhost:9001/second}}.
{noformat}
$ mvn compile exec:java
>>> GET /first -> 200 first
>>> stopping route first failed (ignored): java.lang.RuntimeException:
>>> java.lang.InterruptedException
>>> route second is Started, GET /second -> connection refused
>>> BUG REPRODUCED: route second is started but port 9001 refuses connections
$ mvn exec:java -Dexec.args=control (same steps, consumer stopped on a
normal thread)
>>> GET /first -> 200 first
>>> route second is Started, GET /second -> 200 second
>>> OK: route second answered
{noformat}
h3. Possible fixes
* In {{disconnect}}, always drop the entry once the ref count reaches 0: stop
connector, server and thread pool in try/finally and return {{null}} even when
stopping fails, then rethrow or log.
* In {{connect}}, do not reuse an existing {{ConnectorRef}} whose server is not
running (e.g. {{!server.isRunning()}}). Create a new one instead.
h3. Related
* CAMEL-25850 has the same symptom (a port accepts no connection while its
routes stay started) with a different cause: a shared thread pool with
minThreads/maxThreads. Its fix does not change exception handling in
{{disconnect}}.
> camel-jetty - failed consumer stop leaves stale connector, port no longer
> accepts connections
> ---------------------------------------------------------------------------------------------
>
> Key: CAMEL-26554
> URL: https://issues.apache.org/jira/browse/CAMEL-26554
> Project: Camel
> Issue Type: Bug
> Components: camel-jetty
> Affects Versions: 4.10.9, 4.18.5, 4.22.2, 4.23.0, 4.24.0
> Environment: Java: JDK 21
> Reporter: Bruno Gonçalves
> Priority: Major
> Attachments: jetty-interrupted-disconnect-reproducer.tar.gz
>
>
> h3. Problem
> {{JettyHttpComponent.disconnect(HttpConsumer, ConnectorRef)}} stops the Jetty
> connector and server inside the {{CONNECTORS.computeIfPresent}} lambda:
> {code:java}
> if (connectorRef.decrement() == 0) {
> connectorRef.server.removeConnector(connectorRef.connector);
> connectorRef.connector.stop();
> connectorRef.server.stop();
> ...
> return null;
> }
> {code}
> If any of these calls throws, the lambda never returns {{null}}, so the entry
> for {{protocol:host:port}} stays in the static {{CONNECTORS}} map although
> its listening socket is already closed.
> {{connect(...)}} then reuses that entry for every later consumer on the same
> port ("Using existing connector key"). It increments the ref count and
> registers the servlet, without checking that the server is still running. The
> route reports {{Started}}, but the port refuses connections. Only a JVM
> restart clears it.
> h3. How it happens in practice
> Stopping a route group with {{DefaultShutdownStrategy.shutdown(context,
> routes, timeout, unit)}} while an exchange is still in flight. On timeout the
> strategy calls {{cancel(true)}} on its {{ShutdownTask}}. The task restores
> the interrupt flag and goes on to force-stop the deferred consumers. Stopping
> the jetty consumer on that interrupted thread makes Jetty's
> {{ManagedSelector.doStop}} throw:
> {noformat}
> WARN DefaultShutdownStrategy - Error occurred while shutting down route: ...
> This exception will be ignored.
> java.lang.RuntimeException: java.lang.InterruptedException
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.remove(ContainerLifeCycle.java:730)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.removeBean(ContainerLifeCycle.java:697)
> at org.eclipse.jetty.server.Server.removeConnector(Server.java:406)
> at
> org.apache.camel.component.jetty.JettyHttpComponent.disconnect(JettyHttpComponent.java:500)
> at
> org.apache.camel.component.jetty.JettyHttpComponent.lambda$disconnect$1(JettyHttpComponent.java:487)
> at
> java.base/java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1828)
> at
> org.apache.camel.component.jetty.JettyHttpComponent.disconnect(JettyHttpComponent.java:485)
> at
> org.apache.camel.http.common.HttpCommonEndpoint.disconnect(HttpCommonEndpoint.java:237)
> at org.apache.camel.http.common.HttpConsumer.doStop(HttpConsumer.java:73)
> ...
> at
> org.apache.camel.impl.engine.DefaultShutdownStrategy.shutdownNow(DefaultShutdownStrategy.java:434)
> at
> org.apache.camel.impl.engine.DefaultShutdownStrategy$ShutdownTask.run(DefaultShutdownStrategy.java:743)
> Caused by: java.lang.InterruptedException
> at
> java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1100)
> at
> java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
> at org.eclipse.jetty.io.ManagedSelector.doStop(ManagedSelector.java:138)
> ...
> at
> org.eclipse.jetty.server.AbstractConnector.doStop(AbstractConnector.java:375)
> {noformat}
> After this, no "Started ServerConnector" is logged for that port again, while
> every new jetty route on it logs as started.
> The calling thread force-stops the same consumers at the same time, so
> whether the interrupted {{ShutdownTask}} reaches the jetty consumer first is
> a race. It happened to us on two separate occasions in production, both times
> during startup when a route group failed and was stopped with a timeout.
> h3. Reproducer
> Attached: {{jetty-interrupted-disconnect-reproducer.tar.gz}} (plain Java +
> Camel 4.10.9, JDK 21, about 100 lines). To be deterministic it stops the
> consumer directly on a thread with the interrupt flag set, the state the
> {{ShutdownTask}} is in at that point.
> # Start route {{first}} on {{jetty:http://0.0.0.0:9001/first}} (only consumer
> on the port).
> # Stop its consumer on an interrupted thread, then remove the route.
> # Start route {{second}} on {{jetty:http://0.0.0.0:9001/second}}.
> # {{GET http://localhost:9001/second}}.
> {noformat}
> $ mvn compile exec:java
> >>> GET /first -> 200 first
> >>> stopping route first failed (ignored): java.lang.RuntimeException:
> >>> java.lang.InterruptedException
> >>> route second is Started, GET /second -> connection refused
> >>> BUG REPRODUCED: route second is started but port 9001 refuses connections
> $ mvn exec:java -Dexec.args=control (same steps, consumer stopped on a
> normal thread)
> >>> GET /first -> 200 first
> >>> route second is Started, GET /second -> 200 second
> >>> OK: route second answered
> {noformat}
> Also reproduced with the attached reproducer on 4.18.5, 4.22.2, 4.23.0 and
> 4.24.0-SNAPSHOT (main, built 2026-11-03, includes the CAMEL-25850 change), on
> Jetty 12.1.12 / 12.1.14. Run against another version with:
> {noformat}
> mvn compile exec:java -Dcamel.version=4.23.0
> {noformat}
> h3. Possible fixes
> * In {{disconnect}}, always drop the entry once the ref count reaches 0: stop
> connector, server and thread pool in try/finally and return {{null}} even
> when stopping fails, then rethrow or log.
> * In {{connect}}, do not reuse an existing {{ConnectorRef}} whose server is
> not running (e.g. {{!server.isRunning()}}). Create a new one instead.
> h3. Related
> * CAMEL-25850 has the same symptom (a port accepts no connection while its
> routes stay started) with a different cause: a shared thread pool with
> minThreads/maxThreads. Its fix does not change exception handling in
> {{disconnect}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
