Federico Mariani created CAMEL-27010:
----------------------------------------
Summary: camel-mongodb - stopping the tailable or change streams
consumer waits without bound for its thread when that thread is blocked reading
from a server that stopped answering
Key: CAMEL-27010
URL: https://issues.apache.org/jira/browse/CAMEL-27010
Project: Camel
Issue Type: Bug
Components: camel-mongodb
Affects Versions: 4.18.5, 4.23.0
Reporter: Federico Mariani
Fix For: 4.24.0
{{MongoAbstractConsumerThread.stop()}}
({{components/camel-mongodb/src/main/java/org/apache/camel/component/mongodb/MongoAbstractConsumerThread.java}},
line 167 on main), which {{doStop()}} of both
{{MongoDbTailableCursorConsumer}} and {{MongoDbChangeStreamsConsumer}} calls,
sets {{keepRunning = false}}, closes the cursor and then waits for the consumer
thread to end with {{stoppedLatch.await()}} (line 188), without a timeout.
Closing the cursor does not end a read in progress on the consumer thread: in
the 5.x driver, {{CursorResourceManager.close()}} only marks the cursor
{{CLOSE_PENDING}} while an operation (the {{getMore}} of {{hasNext()}}) is in
progress, and the cursor is closed when that operation ends. A read from a
connection on which the server, or the network in between, stopped answering
(the traffic is dropped without closing the connection) does not return. So the
thread does not end, and:
* {{consumer.stop()}} does not return.
* Stopping the route or the CamelContext, the graceful shutdown waits for its
whole timeout (45 s by default), then cancels its shutdown task, which
interrupts the wait: the stop of the consumer fails with an
{{InterruptedException}} ("Error occurred while shutting down route: ... This
exception will be ignored."), and the thread pool of the consumer is not shut
down. When the CamelContext stops, the fail-safe of the
{{ExecutorServiceManager}} then waits for that pool two more times
{{shutdownAwaitTermination}} (2 x 10 s: "Forcing shutdown of ExecutorService
... due first await termination elapsed.", "Cannot completely force shutdown of
ExecutorService ..."). {{CamelContext.stop()}} takes the shutdown timeout + 20
s, and the thread is left behind anyway.
Found with fault-injection tests (a proxy that silently drops the traffic of
the change streams consumer's connection).
h3. Steps to reproduce
A consumer whose cursor's {{hasNext()}} blocks, and is ended neither by
{{close()}} nor by an interrupt, as a socket read. Start the route, wait until
the consumer thread is in {{hasNext()}}, then stop the route or the
CamelContext.
h3. Actual (4.24.0-SNAPSHOT)
{{stopRoute()}} returns only when the graceful shutdown times out. With a
shutdown timeout of 3 s, {{CamelContext.stop()}} returns after 23 s (3 s + 2 x
10 s), with the warnings above; the tailable and the change streams consumer
behave the same.
h3. Expected
The stop does not wait without bound for a consumer thread that reads from
MongoDB: after a bounded wait it logs a WARN, interrupts the thread and leaves
it behind, and the consumer stops. An exchange that the route processes is
waited for as before, as the graceful shutdown bounds that wait with its own
timeout.
h3. Related
* CAMEL-9826: the stop waits for the thread with the {{CountDownLatch}}.
* CAMEL-25679: the consumer thread no longer ends when a new cursor cannot be
opened.
_Claude Code on behalf of Croway_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)