[
https://issues.apache.org/jira/browse/CAMEL-25759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-25759 started by shashank.
----------------------------------------
> camel-aws2-eventbridge - a consumer with queueUrl adds a new target for the
> same queue at every start, so two consumers sharing the queue, or a restart
> after a crash, put every event on the queue twice
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25759
> URL: https://issues.apache.org/jira/browse/CAMEL-25759
> Project: Camel
> Issue Type: Bug
> Components: camel-aws2
> Reporter: shashank
> Assignee: shashank
> Priority: Major
>
> The EventBridge consumer (since 4.19) wires an SQS queue as a target of the
> rule and polls the queue. {{EventbridgeConsumer.setupConsumerResources}}
> (main e13303ef1ef7, line 188) gives the target a new random id at every start:
> {code:java}
> targetId = "camel-eb-" + UUID.randomUUID().toString().substring(0, 8);
> ...
> ebClient.putTargets(PutTargetsRequest.builder().rule(ruleName).eventBusName(...).targets(target).build());
> {code}
> and {{cleanupConsumerResources}} (lines 210-227) removes that target on stop.
> With an auto-created queue each consumer has its own queue, so this is fine.
> With a queue given by the user ({{queueUrl}}, the documented "User-Provided
> Queue" setup; it is also the only way to let several application instances
> share the events of a rule, since with auto-created queues every instance
> gets its own copy of every event) the rule gets one target per consumer for
> the *same* queue:
> # two consumers of the queue run at the same time (two instances of the
> application, or two routes): the rule has two targets with the queue ARN.
> EventBridge sends a matching event to every target of the rule, so each event
> is put on the queue twice and routed twice;
> # a consumer is not stopped cleanly (JVM killed, container OOM, node lost):
> its target stays on the rule. The next start adds a second target for the
> queue, and from then on every event is put on the queue twice, with no error
> and nothing in the log. Each such restart adds one more copy, until the rule
> reaches the EventBridge limit of 5 targets and the consumer fails to start.
> In addition, the target is removed when the consumer stops, so a user-owned
> queue does not collect the events of the rule while the application is
> restarted or redeployed, and with two consumers the first one that stops
> removes its own target only, so the duplicates last as long as both run.
> *Fix*: the target of a user-provided queue gets an id derived from the queue
> ARN ({{camel-eb-}} + 32 hex characters of its SHA-256). PutTargets replaces a
> target with the same id, so every start and every consumer of the queue
> updates the one target. That target is kept when the consumer stops (other
> consumers may share the queue, and the queue keeps collecting events while no
> consumer runs). An auto-created queue keeps its random target id and its
> clean-up (target removed, queue deleted with {{deleteQueueOnShutdown=true}}).
> The component page and the 4.24 upgrade guide say so; a target left by an
> earlier version (id {{camel-eb-}} + 8 characters) has to be removed by hand.
> *Design choice to review*: keeping the target of a user-provided queue on
> stop reverses what CAMEL-23207 did (the component page said "On shutdown the
> target is removed", and {{testUserProvidedQueueNotDeletedOnShutdown}}
> verified the removal). The two changes go together: with a shared id but
> removal on stop, the first consumer that stops would remove the target that
> the others still use. Not changed: a target that the user added to the rule
> for the same queue (with an id of their own, for example from infrastructure
> as code) is still a second target, as on main.
> *Tests*: new {{EventbridgeConsumerUserQueueTargetTest}} (4 tests): the mocked
> EventBridge client keeps the rule as a map of target id to ARN, with the
> PutTargets (replace by id) and RemoveTargets semantics.
> {{twoConsumersSharingAQueueAddOneTarget}} and
> {{restartAfterACrashAddsNoSecondTarget}} fail on main ({{expected: <1> but
> was: <2>}}, two {{camel-eb-xxxxxxxx}} targets for the user queue);
> {{userQueueKeepsCollectingEventsWhileTheConsumerIsStopped}} fails on main
> (target removed); {{autoCreatedQueueTargetIsStillRemovedOnStop}} passes on
> main and with the fix (the auto-created path is unchanged).
> {{EventbridgeConsumerTest.testUserProvidedQueueNotDeletedOnShutdown}} pinned
> the removal of the user queue's target and now checks that the target is
> kept. camel-aws2-eventbridge unit tests: 30 run, 0 failures (the LocalStack
> ITs need Docker and were not run).
> *How it was found*: a TLA+ model of the targets of the rule (consumers
> sharing one user queue that start, stop or crash; each published event is put
> on the queue once per target). TLC finds a violation of "no event is put on
> the queue twice" in 4 steps with two consumers (start, start, publish) and
> with one consumer (start, crash, start, publish); with the fix the property,
> "an event published while a consumer runs reaches the queue" and "once a
> consumer started, every later event reaches the queue" hold for 2 and 3
> consumers. The JUnit tests above are the evidence; the EventBridge semantics
> (every target is invoked; PutTargets with an existing id updates that target)
> are from the AWS API reference and were not run against AWS.
> *Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.19.x (the same
> lines; read, not run). camel-4.18.x has no consumer.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
