shashank created CAMEL-27336:
--------------------------------
Summary: camel-core - an evicted singleton producer (and the
endpoint of a toD) is stopped while it is in use, and a stale eviction entry of
ServicePool stops the wrong producer
Key: CAMEL-27336
URL: https://issues.apache.org/jira/browse/CAMEL-27336
Project: Camel
Issue Type: Bug
Components: camel-core
Reporter: shashank
Assignee: shashank
h3. What happens
{{ServicePool.SinglePool}} (the pool of a singleton producer or polling
consumer, used by every {{DefaultProducerCache}}: toD, recipientList,
routingSlip, enrich, ProducerTemplate) keeps no record of who uses its
producer. When the LRU cache of the producer cache evicts it:
# {{onEvict}} stops the pool at once when the pool map is larger than the
capacity (the usual case once the cache is full), or marks it in
{{singlePoolEvicted}} so that the next acquire or release of any single pool
stops it. Neither checks whether the producer is in use, so an in-flight send
runs on a stopped producer. For a dynamic endpoint (toD) the cleanup then also
finds the endpoint unused, and stops it and removes it from the registry during
the send. With the default LRU cache a hit does not refresh the LRU order, so
the evicted producer is the one created first, often the busiest one.
{{MultiplePool}} protects its producers in use since CAMEL-25003.
# {{onEvict}} both marks the pool and stops it, so the mark stays behind. When
the next {{acquire}} of the same endpoint has already created a new pool, the
stale mark's cleanup removes the new pool from the map
({{pool.remove(endpoint)}}, not {{remove(endpoint, this)}}), stops the
endpoint, and the new pool hands out a producer of the stopped endpoint. The
next acquire creates a second producer of the singleton endpoint, and a later
eviction of the first one stops the current producer instead
({{SinglePool.evict(s)}} ignores {{s}}) and leaks the evicted one until the
CamelContext stops.
# {{SinglePool.acquire}} reads the volatile producer field twice (check, then
return): an eviction in between makes it return null, and
{{DefaultProducerCache}} fails with "No producer, this processor has not been
started".
{{DefaultProducerCache}}'s {{lastUsedProducer}} fast path also hands out a
producer without the pool, so it can return a producer that an eviction stopped.
CAMEL-25066 fixed the endpoint stop when another producer cache uses the
endpoint; these are different gaps in the same eviction code.
h3. How to reproduce
{{DefaultProducerCache}} with cache size 1: a worker sends to {{my:a}} and
waits inside the producer's {{process()}}, another worker sends to {{my:b}}.
The producer of {{my:a}} and the endpoint are stopped while the first send is
still in {{process()}}. With {{ProducerServicePool}} (capacity 1):
{{acquire(a)}}, {{release(a)}}, {{acquire(b)}} (held), {{acquire(a)}} returns a
producer of a stopped endpoint, and a second {{acquire(a)}} a different
producer.
Found with a TLA+ model of {{ServicePool}}.
h3. Fix
{{SinglePool}} counts the users of its producer (acquire, the fast path and
release). An eviction takes the pool out of the pool map at once (so the
endpoint gets a new pool) and the producer is stopped, with the endpoint if
nothing else uses it, by the eviction when the producer is idle or by its last
release. The evicted pools in use are kept by producer, so a release finds
them, and a stale eviction of a producer that is no longer the pool's producer
is ignored. {{acquire}} returns a local copy, and retries with a new pool when
the pool was evicted meanwhile.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)