[
https://issues.apache.org/jira/browse/CAMEL-25727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-25727:
--------------------------------
Fix Version/s: 4.24.0
> camel-aws-cloudtrail - the consumer never routes an event that CloudTrail
> makes visible after a newer one, because each lookup starts at the newest
> event time already seen
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25727
> URL: https://issues.apache.org/jira/browse/CAMEL-25727
> Project: Camel
> Issue Type: Bug
> Components: camel-aws
> Reporter: shashank
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
>
> Since CAMEL-24082 the {{aws-cloudtrail}} consumer keeps a per-consumer
> cursor: each poll calls {{LookupEvents}} with {{startTime}} = the newest
> event time it has seen, reads every page, and skips the events seen at
> exactly that instant by event id ({{CloudtrailConsumer.poll}}, main
> 55ed20f14e90, lines 77-79 and 94-119).
> AWS does not document that {{LookupEvents}} sees events in the order of their
> event time, and it documents a delivery delay. The AWS CloudTrail FAQ
> (https://aws.amazon.com/cloudtrail/faqs/, "How long does it take CloudTrail
> to deliver an event for an API call?") says: "Typically, CloudTrail delivers
> an event within 5 minutes of the API call." For trails, the user guide
> (https://docs.aws.amazon.com/awscloudtrail/latest/userguide/how-cloudtrail-works.html)
> says "CloudTrail typically delivers logs within an average of about 5
> minutes of an API call. This time is not guaranteed." The {{eventTime}} of an
> event is the time of the API call. With a delay that varies per event, two
> API calls A (12:00:00) and B (12:00:30) can become visible in the order B, A:
> B can be visible to {{LookupEvents}} before A. A poll in between routes B and
> moves the cursor to 12:00:30; when A becomes visible its event time is before
> every later {{startTime}}, and it is never routed. Nothing is logged.
> Trigger: default options (the consumer polls every 500 ms), any account where
> API calls of different services are delivered with different delays. The loss
> is silent and permanent for that event.
> h3. Reproduction
> New test
> {{CloudtrailConsumerLateEventTest.lateEventWithEarlierEventTimeIsRoutedOnceByTheLookBack}}
> with a mocked {{CloudTrailClient}} that answers like the service (only
> events in [{{startTime}}, {{endTime}}], newest first) and a test clock: poll
> 1 sees event "newer" (T+20 s); then event "late" (T+10 s) becomes visible; at
> T+60 s a poll must route it. On main that poll routes nothing ({{expected:
> <1> but was: <0>}}, 3 of 3 runs). The test needs the consumer's new clock
> hook {{now()}} (and the other new tests the new options), so it does not
> compile against main as committed; it was run against main's consumer with
> the {{@Override}} of {{now()}} removed (main then ignores the test clock,
> which does not change this sequence) and the four tests of the new options
> left out. With the fix, removing only the call of the late-event lookup makes
> it fail the same way, and {{lookBackWindowZeroBehavesAsBefore}} pins main's
> behaviour: the late event is not routed.
> h3. Proposed fix
> The normal polls stay as they are: same {{LookupEvents}} requests, from the
> newest event time seen. A separate late-event lookup runs every
> {{lookBackInterval}} (new option, default 1 minute): it looks up the events
> from {{lookBackWindow}} (new option, default 10 minutes) before now up to the
> newest event time seen, never before the consumer start, and routes those it
> has not routed yet (by event id). It sends one page per poll, so it adds at
> most one request to a poll. If that request fails, the failure is logged
> (WARN, the stack trace only at DEBUG), the events of the normal lookup are
> still routed and the lookup starts again at the next interval. The ids of the
> routed events are kept with their event time and dropped once that time is
> before every later lookup, so the set only holds the events of the window.
> {{lookBackWindow=0}} disables the lookup and gives exactly the previous
> behaviour.
> Defaults: 10 minutes is twice the "typically within 5 minutes" of the FAQ; 1
> minute bounds the delay of a late event to about a minute after it becomes
> visible, for one extra request per minute on a quiet account.
> Cost: the late-event lookup sends ceil(N / maxResults) requests per
> {{lookBackInterval}}, at most one per poll, where N is the number of events
> (of the {{eventSource}}, if set) between {{lookBackWindow}} before now and
> the newest event time seen; none when that event time is older than the
> window. With the defaults: one request per minute up to 50 events in 10
> minutes, 20 per minute with 1000 events in 10 minutes. AWS allows two
> {{LookupEvents}} requests per second per account and Region (120 per minute);
> the polls themselves send at least one request every {{delay}} (500 ms by
> default, so up to about two per second, as on main), and one more per further
> page of new events, as before. So with the default {{delay}} the polls alone
> can already use most of that limit; the late-event lookup adds one request
> per minute to it on a quiet account. The option description, the component
> page and the upgrade note state this.
> With the fix the camel-aws-cloudtrail tests pass: 7 run, 0 failures, 0
> errors, 0 skipped (5 new). New tests (mocked client, test clock, no sleeps):
> the late event is routed exactly once; between two late-event lookups each
> poll sends the same requests as before (counted, no {{endTime}}); the lookup
> sends one page per poll, starts at the consumer start and does not route
> events again, and the id set empties once the window has passed; a failed
> late-event lookup is logged and does not lose the events of the normal lookup
> of that poll; {{lookBackWindow=0}} sends only the normal requests and does
> not route the late event, as before.
> Not covered: an event whose event time is before the consumer start (as
> before), and an event that becomes visible more than {{lookBackWindow}} after
> its event time. The CloudTrail refresh task of camel-aws-secrets-manager
> ({{CloudTrailReloadTriggerTask}}) moves its position in a similar way; not
> changed here. Not run against AWS.
> Affected: main (tested); camel-4.23.x, camel-4.22.x and camel-4.18.x have the
> same cursor code (read, not run).
> Duplicate check (2026-10-17): JIRA text "cloudtrail" (all issues; CAMEL-24082
> fixed the static cursor, the paging and the +1 s skip, not late delivery);
> GitHub pull requests "cloudtrail": only dependency bumps and CAMEL-24082.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
