Bruno Gonçalves created CAMEL-26554:
---------------------------------------
Summary: camel-jetty - failed consumer stop leaves stale
connector, port no longer accepts connections
Key: CAMEL-26554
URL: https://issues.apache.org/jira/browse/CAMEL-26554
Project: Camel
Issue Type: Bug
Components: camel-jetty
Affects Versions: 4.23.0, 4.22.2, 4.18.5, 4.10.9, 4.24.0
Environment: Java: JDK 21
Reporter: Bruno Gonçalves
Attachments: jetty-interrupted-disconnect-reproducer.tar.gz
h3. Problem
{{JettyHttpComponent.disconnect(HttpConsumer, ConnectorRef)}} stops the Jetty
connector and server inside the {{CONNECTORS.computeIfPresent}} lambda:
{code:java}
if (connectorRef.decrement() == 0) {
connectorRef.server.removeConnector(connectorRef.connector);
connectorRef.connector.stop();
connectorRef.server.stop();
...
return null;
}
{code}
If any of these calls throws, the lambda never returns {{null}}, so the entry
for {{protocol:host:port}} stays in the static {{CONNECTORS}} map although its
listening socket is already closed.
{{connect(...)}} then reuses that entry for every later consumer on the same
port ("Using existing connector key"). It increments the ref count and
registers the servlet, without checking that the server is still running. The
route reports {{Started}}, but the port refuses connections. Only a JVM restart
clears it.
h3. How it happens in practice
Stopping a route group with {{DefaultShutdownStrategy.shutdown(context, routes,
timeout, unit)}} while an exchange is still in flight. On timeout the strategy
calls {{cancel(true)}} on its {{ShutdownTask}}. The task restores the interrupt
flag and goes on to force-stop the deferred consumers. Stopping the jetty
consumer on that interrupted thread makes Jetty's {{ManagedSelector.doStop}}
throw:
{noformat}
WARN DefaultShutdownStrategy - Error occurred while shutting down route: ...
This exception will be ignored.
java.lang.RuntimeException: java.lang.InterruptedException
at
org.eclipse.jetty.util.component.ContainerLifeCycle.remove(ContainerLifeCycle.java:730)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.removeBean(ContainerLifeCycle.java:697)
at org.eclipse.jetty.server.Server.removeConnector(Server.java:406)
at
org.apache.camel.component.jetty.JettyHttpComponent.disconnect(JettyHttpComponent.java:500)
at
org.apache.camel.component.jetty.JettyHttpComponent.lambda$disconnect$1(JettyHttpComponent.java:487)
at
java.base/java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1828)
at
org.apache.camel.component.jetty.JettyHttpComponent.disconnect(JettyHttpComponent.java:485)
at
org.apache.camel.http.common.HttpCommonEndpoint.disconnect(HttpCommonEndpoint.java:237)
at org.apache.camel.http.common.HttpConsumer.doStop(HttpConsumer.java:73)
...
at
org.apache.camel.impl.engine.DefaultShutdownStrategy.shutdownNow(DefaultShutdownStrategy.java:434)
at
org.apache.camel.impl.engine.DefaultShutdownStrategy$ShutdownTask.run(DefaultShutdownStrategy.java:743)
Caused by: java.lang.InterruptedException
at
java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1100)
at
java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
at org.eclipse.jetty.io.ManagedSelector.doStop(ManagedSelector.java:138)
...
at
org.eclipse.jetty.server.AbstractConnector.doStop(AbstractConnector.java:375)
{noformat}
After this, no "Started ServerConnector" is logged for that port again, while
every new jetty route on it logs as started.
The calling thread force-stops the same consumers at the same time, so whether
the interrupted {{ShutdownTask}} reaches the jetty consumer first is a race. It
happened to us on two separate occasions in production, both times during
startup when a route group failed and was stopped with a timeout.
h3. Reproducer
Attached: {{jetty-interrupted-disconnect-reproducer.tar.gz}} (plain Java +
Camel 4.10.9, JDK 21, about 100 lines). To be deterministic it stops the
consumer directly on a thread with the interrupt flag set, the state the
{{ShutdownTask}} is in at that point.
# Start route {{first}} on {{jetty:http://0.0.0.0:9001/first}} (only consumer
on the port).
# Stop its consumer on an interrupted thread, then remove the route.
# Start route {{second}} on {{jetty:http://0.0.0.0:9001/second}}.
# {{GET http://localhost:9001/second}}.
{noformat}
$ mvn compile exec:java
>>> GET /first -> 200 first
>>> stopping route first failed (ignored): java.lang.RuntimeException:
>>> java.lang.InterruptedException
>>> route second is Started, GET /second -> connection refused
>>> BUG REPRODUCED: route second is started but port 9001 refuses connections
$ mvn exec:java -Dexec.args=control (same steps, consumer stopped on a
normal thread)
>>> GET /first -> 200 first
>>> route second is Started, GET /second -> 200 second
>>> OK: route second answered
{noformat}
h3. Possible fixes
* In {{disconnect}}, always drop the entry once the ref count reaches 0: stop
connector, server and thread pool in try/finally and return {{null}} even when
stopping fails, then rethrow or log.
* In {{connect}}, do not reuse an existing {{ConnectorRef}} whose server is not
running (e.g. {{!server.isRunning()}}). Create a new one instead.
h3. Related
* CAMEL-25850 has the same symptom (a port accepts no connection while its
routes stay started) with a different cause: a shared thread pool with
minThreads/maxThreads. Its fix does not change exception handling in
{{disconnect}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
