[
https://issues.apache.org/jira/browse/CAMEL-24315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Henrik updated CAMEL-24315:
---------------------------
Description:
Since CAMEL-22898 (4.18.0), GooglePubsubConsumer adds a subscriber to the
subscribers list only after startAsync().awaitRunning() returns. doStop() and
deferShutdown() only stop subscribers already in that list, so stopping the
consumer while a subscriber is still starting misses it. Nothing ever calls
stopAsync() on it, and the wrapper thread parks forever in awaitTerminated(),
an uninterruptible Guava wait.
The consumer executor then never terminates: context shutdown waits 2 x
shutdownAwaitTermination (default 10s) per stuck pool and the thread leaks even
after forced shutdown. Test suites that stop many CamelContexts (e.g. Spring
Boot integration tests) pay up to 20s per leaked consumer per context; we saw a
CI build go from 5 to 25 minutes from this alone.
The synchronous pull path has the same window: the pull future is added to
pendingSynchronousPullResponses only after futureCall() returns, so a stop
landing in between misses it and get() blocks until the pull deadline instead
of being cancelled.
Fix: after registering the subscriber or pull future, re-check the consumer
state and stop/cancel it if the consumer is no longer allowed to run. stop()
sets the volatile service status to STOPPING before doStop() runs, so either
doStop() sees the subscriber in the list or the re-check sees the stopped
status. Unit tests reproduce both races deterministically and fail without the
fix.
Workaround on affected versions: lower shutdownAwaitTermination to bound the
shutdown delay; the thread still leaks.
PR available: https://github.com/apache/camel/pull/25248
was:
Since CAMEL-22898 (4.18.0), GooglePubsubConsumer adds a subscriber to the
subscribers list only after startAsync().awaitRunning() returns. doStop() and
deferShutdown() only stop subscribers already in that list, so stopping the
consumer while a subscriber is still starting misses it. Nothing ever calls
stopAsync() on it, and the wrapper thread parks forever in awaitTerminated(),
an uninterruptible Guava wait.
The consumer executor then never terminates: context shutdown waits 2 x
shutdownAwaitTermination (default 10s) per stuck pool and the thread leaks even
after forced shutdown. Test suites that stop many CamelContexts (e.g. Spring
Boot integration tests) pay up to 20s per leaked consumer per context; we saw a
CI build go from 5 to 25 minutes from this alone.
The synchronous pull path has the same window: the pull future is added to
pendingSynchronousPullResponses only after futureCall() returns, so a stop
landing in between misses it and get() blocks until the pull deadline instead
of being cancelled.
Fix: after registering the subscriber or pull future, re-check the consumer
state and stop/cancel it if the consumer is no longer allowed to run. stop()
sets the volatile service status to STOPPING before doStop() runs, so either
doStop() sees the subscriber in the list or the re-check sees the stopped
status. Unit tests reproduce both races deterministically and fail without the
fix.
Workaround on affected versions: lower shutdownAwaitTermination to bound the
shutdown delay; the thread still leaks.
> camel-google-pubsub: consumer stop misses subscribers still starting, thread
> parks forever in awaitTerminated
> -------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-24315
> URL: https://issues.apache.org/jira/browse/CAMEL-24315
> Project: Camel
> Issue Type: Bug
> Components: camel-google-pubsub
> Affects Versions: 4.18.0, 4.18.3, 4.19.0, 4.20.0, 4.21.0
> Reporter: Henrik
> Priority: Major
> Attachments:
> 0001-CAMEL-24315-camel-google-pubsub-stop-subscribers-tha.patch
>
>
> Since CAMEL-22898 (4.18.0), GooglePubsubConsumer adds a subscriber to the
> subscribers list only after startAsync().awaitRunning() returns. doStop() and
> deferShutdown() only stop subscribers already in that list, so stopping the
> consumer while a subscriber is still starting misses it. Nothing ever calls
> stopAsync() on it, and the wrapper thread parks forever in awaitTerminated(),
> an uninterruptible Guava wait.
> The consumer executor then never terminates: context shutdown waits 2 x
> shutdownAwaitTermination (default 10s) per stuck pool and the thread leaks
> even after forced shutdown. Test suites that stop many CamelContexts (e.g.
> Spring Boot integration tests) pay up to 20s per leaked consumer per context;
> we saw a CI build go from 5 to 25 minutes from this alone.
> The synchronous pull path has the same window: the pull future is added to
> pendingSynchronousPullResponses only after futureCall() returns, so a stop
> landing in between misses it and get() blocks until the pull deadline instead
> of being cancelled.
> Fix: after registering the subscriber or pull future, re-check the consumer
> state and stop/cancel it if the consumer is no longer allowed to run. stop()
> sets the volatile service status to STOPPING before doStop() runs, so either
> doStop() sees the subscriber in the list or the re-check sees the stopped
> status. Unit tests reproduce both races deterministically and fail without
> the fix.
> Workaround on affected versions: lower shutdownAwaitTermination to bound the
> shutdown delay; the thread still leaks.
> PR available: https://github.com/apache/camel/pull/25248
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
