Andrea Cosentino created CAMEL-24525:
----------------------------------------
Summary: camel-google-firestore: the realtime change buffer is
unbounded
Key: CAMEL-24525
URL: https://issues.apache.org/jira/browse/CAMEL-24525
Project: Camel
Issue Type: Improvement
Components: camel-google
Reporter: Andrea Cosentino
Assignee: Andrea Cosentino
The Firestore consumer buffers the document changes reported by the realtime
snapshot listener in an unbounded queue, and drains it on the scheduled poll:
{code:java}
private final Queue<Exchange> pendingExchanges = new ConcurrentLinkedQueue<>();
{code}
The listener callback runs on a Firestore client thread and is never throttled
by the route. Whenever changes arrive faster than the poll consumes them the
queue grows without limit, and every element is a full Exchange holding the
document data. The queue also keeps growing while the route is suspended, or
when the consumer is configured with a long poll delay, so a burst of writes on
a busy collection can exhaust the heap.
Add a {{maxPendingChanges}} consumer option that bounds the buffer:
* the default stays unbounded, so existing routes are unaffected;
* when a limit is configured and reached, the oldest buffered change is
discarded - the route then keeps up with the current state of the collection -
and a warning is logged;
* discarded exchanges are released back to the exchange pool instead of being
dropped on the floor, and so is anything still buffered when the consumer stops.
Blocking the listener callback is deliberately not offered as a policy: that
thread belongs to the Firestore gRPC client, and stalling it would stall the
watch stream itself.
This is the item left open in CAMEL-24347, whose other findings are already
fixed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
