[
https://issues.apache.org/jira/browse/CAMEL-26554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-26554:
--------------------------------
Fix Version/s: 4.24.0
> camel-jetty - failed consumer stop leaves stale connector, port no longer
> accepts connections
> ---------------------------------------------------------------------------------------------
>
> Key: CAMEL-26554
> URL: https://issues.apache.org/jira/browse/CAMEL-26554
> Project: Camel
> Issue Type: Bug
> Components: camel-jetty
> Affects Versions: 4.10.9, 4.18.5, 4.22.2, 4.23.0, 4.24.0
> Environment: Java: JDK 21
> Reporter: Bruno Gonçalves
> Assignee: Claus Ibsen
> Priority: Major
> Fix For: 4.24.0
>
> Attachments: jetty-interrupted-disconnect-reproducer.tar.gz
>
>
> h3. Problem
> {{JettyHttpComponent.disconnect(HttpConsumer, ConnectorRef)}} stops the Jetty
> connector and server inside the {{CONNECTORS.computeIfPresent}} lambda:
> {code:java}
> if (connectorRef.decrement() == 0) {
> connectorRef.server.removeConnector(connectorRef.connector);
> connectorRef.connector.stop();
> connectorRef.server.stop();
> ...
> return null;
> }
> {code}
> If any of these calls throws, the lambda never returns {{null}}, so the entry
> for {{protocol:host:port}} stays in the static {{CONNECTORS}} map although
> its listening socket is already closed.
> {{connect(...)}} then reuses that entry for every later consumer on the same
> port ("Using existing connector key"). It increments the ref count and
> registers the servlet, without checking that the server is still running. The
> route reports {{Started}}, but the port refuses connections. Only a JVM
> restart clears it.
> h3. How it happens in practice
> Stopping a route group with {{DefaultShutdownStrategy.shutdown(context,
> routes, timeout, unit)}} while an exchange is still in flight. On timeout the
> strategy calls {{cancel(true)}} on its {{ShutdownTask}}. The task restores
> the interrupt flag and goes on to force-stop the deferred consumers. Stopping
> the jetty consumer on that interrupted thread makes Jetty's
> {{ManagedSelector.doStop}} throw:
> {noformat}
> WARN DefaultShutdownStrategy - Error occurred while shutting down route: ...
> This exception will be ignored.
> java.lang.RuntimeException: java.lang.InterruptedException
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.remove(ContainerLifeCycle.java:730)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.removeBean(ContainerLifeCycle.java:697)
> at org.eclipse.jetty.server.Server.removeConnector(Server.java:406)
> at
> org.apache.camel.component.jetty.JettyHttpComponent.disconnect(JettyHttpComponent.java:500)
> at
> org.apache.camel.component.jetty.JettyHttpComponent.lambda$disconnect$1(JettyHttpComponent.java:487)
> at
> java.base/java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1828)
> at
> org.apache.camel.component.jetty.JettyHttpComponent.disconnect(JettyHttpComponent.java:485)
> at
> org.apache.camel.http.common.HttpCommonEndpoint.disconnect(HttpCommonEndpoint.java:237)
> at org.apache.camel.http.common.HttpConsumer.doStop(HttpConsumer.java:73)
> ...
> at
> org.apache.camel.impl.engine.DefaultShutdownStrategy.shutdownNow(DefaultShutdownStrategy.java:434)
> at
> org.apache.camel.impl.engine.DefaultShutdownStrategy$ShutdownTask.run(DefaultShutdownStrategy.java:743)
> Caused by: java.lang.InterruptedException
> at
> java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1100)
> at
> java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
> at org.eclipse.jetty.io.ManagedSelector.doStop(ManagedSelector.java:138)
> ...
> at
> org.eclipse.jetty.server.AbstractConnector.doStop(AbstractConnector.java:375)
> {noformat}
> After this, no "Started ServerConnector" is logged for that port again, while
> every new jetty route on it logs as started.
> The calling thread force-stops the same consumers at the same time, so
> whether the interrupted {{ShutdownTask}} reaches the jetty consumer first is
> a race. It happened to us on two separate occasions in production, both times
> during startup when a route group failed and was stopped with a timeout.
> h3. Reproducer
> Attached: {{jetty-interrupted-disconnect-reproducer.tar.gz}} (plain Java +
> Camel 4.10.9, JDK 21, about 100 lines). To be deterministic it stops the
> consumer directly on a thread with the interrupt flag set, the state the
> {{ShutdownTask}} is in at that point.
> # Start route {{first}} on {{jetty:http://0.0.0.0:9001/first}} (only consumer
> on the port).
> # Stop its consumer on an interrupted thread, then remove the route.
> # Start route {{second}} on {{jetty:http://0.0.0.0:9001/second}}.
> # {{GET http://localhost:9001/second}}.
> {noformat}
> $ mvn compile exec:java
> >>> GET /first -> 200 first
> >>> stopping route first failed (ignored): java.lang.RuntimeException:
> >>> java.lang.InterruptedException
> >>> route second is Started, GET /second -> connection refused
> >>> BUG REPRODUCED: route second is started but port 9001 refuses connections
> $ mvn exec:java -Dexec.args=control (same steps, consumer stopped on a
> normal thread)
> >>> GET /first -> 200 first
> >>> route second is Started, GET /second -> 200 second
> >>> OK: route second answered
> {noformat}
> Also reproduced with the attached reproducer on 4.18.5, 4.22.2, 4.23.0 and
> 4.24.0-SNAPSHOT (main, built 2026-11-03, includes the CAMEL-25850 change), on
> Jetty 12.1.12 / 12.1.14. Run against another version with:
> {noformat}
> mvn compile exec:java -Dcamel.version=4.23.0
> {noformat}
> h3. Possible fixes
> * In {{disconnect}}, always drop the entry once the ref count reaches 0: stop
> connector, server and thread pool in try/finally and return {{null}} even
> when stopping fails, then rethrow or log.
> * In {{connect}}, do not reuse an existing {{ConnectorRef}} whose server is
> not running (e.g. {{!server.isRunning()}}). Create a new one instead.
> h3. Related
> * CAMEL-25850 has the same symptom (a port accepts no connection while its
> routes stay started) with a different cause: a shared thread pool with
> minThreads/maxThreads. Its fix does not change exception handling in
> {{disconnect}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
