shashank created CAMEL-25727:
--------------------------------
Summary: camel-aws-cloudtrail - the consumer never routes an event
that CloudTrail makes visible after a newer one, because each lookup starts at
the newest event time already seen
Key: CAMEL-25727
URL: https://issues.apache.org/jira/browse/CAMEL-25727
Project: Camel
Issue Type: Bug
Components: camel-aws
Reporter: shashank
Assignee: shashank
Since CAMEL-24082 the {{aws-cloudtrail}} consumer keeps a per-consumer cursor:
each poll calls {{LookupEvents}} with {{startTime}} = the newest event time it
has seen, reads every page, and skips the events seen at exactly that instant
by event id ({{CloudtrailConsumer.poll}}, main 55ed20f14e90, lines 77-79 and
94-119).
AWS does not document that {{LookupEvents}} sees events in the order of their
event time, and it documents a delivery delay. The AWS CloudTrail FAQ
(https://aws.amazon.com/cloudtrail/faqs/, "How long does it take CloudTrail to
deliver an event for an API call?") says: "Typically, CloudTrail delivers an
event within 5 minutes of the API call." For trails, the user guide
(https://docs.aws.amazon.com/awscloudtrail/latest/userguide/how-cloudtrail-works.html)
says "CloudTrail typically delivers logs within an average of about 5 minutes
of an API call. This time is not guaranteed." The {{eventTime}} of an event is
the time of the API call. With a delay that varies per event, two API calls A
(12:00:00) and B (12:00:30) can become visible in the order B, A: B can be
visible to {{LookupEvents}} before A. A poll in between routes B and moves the
cursor to 12:00:30; when A becomes visible its event time is before every later
{{startTime}}, and it is never routed. Nothing is logged.
Trigger: default options (the consumer polls every 500 ms), any account where
API calls of different services are delivered with different delays. The loss
is silent and permanent for that event.
h3. Reproduction
New test
{{CloudtrailConsumerLateEventTest.lateEventWithEarlierEventTimeIsRoutedOnceByTheLookBack}}
with a mocked {{CloudTrailClient}} that answers like the service (only events
in [{{startTime}}, {{endTime}}], newest first) and a test clock: poll 1 sees
event "newer" (T+20 s); then event "late" (T+10 s) becomes visible; at T+60 s a
poll must route it. On main that poll routes nothing ({{expected: <1> but was:
<0>}}, 3 of 3 runs). The test needs the consumer's new clock hook {{now()}}
(and the other new tests the new options), so it does not compile against main
as committed; it was run against main's consumer with the {{@Override}} of
{{now()}} removed (main then ignores the test clock, which does not change this
sequence) and the four tests of the new options left out. With the fix,
removing only the call of the late-event lookup makes it fail the same way, and
{{lookBackWindowZeroBehavesAsBefore}} pins main's behaviour: the late event is
not routed.
h3. Proposed fix
The normal polls stay as they are: same {{LookupEvents}} requests, from the
newest event time seen. A separate late-event lookup runs every
{{lookBackInterval}} (new option, default 1 minute): it looks up the events
from {{lookBackWindow}} (new option, default 10 minutes) before now up to the
newest event time seen, never before the consumer start, and routes those it
has not routed yet (by event id). It sends one page per poll, so it adds at
most one request to a poll. If that request fails, the failure is logged (WARN,
the stack trace only at DEBUG), the events of the normal lookup are still
routed and the lookup starts again at the next interval. The ids of the routed
events are kept with their event time and dropped once that time is before
every later lookup, so the set only holds the events of the window.
{{lookBackWindow=0}} disables the lookup and gives exactly the previous
behaviour.
Defaults: 10 minutes is twice the "typically within 5 minutes" of the FAQ; 1
minute bounds the delay of a late event to about a minute after it becomes
visible, for one extra request per minute on a quiet account.
Cost: the late-event lookup sends ceil(N / maxResults) requests per
{{lookBackInterval}}, at most one per poll, where N is the number of events (of
the {{eventSource}}, if set) between {{lookBackWindow}} before now and the
newest event time seen; none when that event time is older than the window.
With the defaults: one request per minute up to 50 events in 10 minutes, 20 per
minute with 1000 events in 10 minutes. AWS allows two {{LookupEvents}} requests
per second per account and Region (120 per minute); the polls themselves send
at least one request every {{delay}} (500 ms by default, so up to about two per
second, as on main), and one more per further page of new events, as before. So
with the default {{delay}} the polls alone can already use most of that limit;
the late-event lookup adds one request per minute to it on a quiet account. The
option description, the component page and the upgrade note state this.
With the fix the camel-aws-cloudtrail tests pass: 7 run, 0 failures, 0 errors,
0 skipped (5 new). New tests (mocked client, test clock, no sleeps): the late
event is routed exactly once; between two late-event lookups each poll sends
the same requests as before (counted, no {{endTime}}); the lookup sends one
page per poll, starts at the consumer start and does not route events again,
and the id set empties once the window has passed; a failed late-event lookup
is logged and does not lose the events of the normal lookup of that poll;
{{lookBackWindow=0}} sends only the normal requests and does not route the late
event, as before.
Not covered: an event whose event time is before the consumer start (as
before), and an event that becomes visible more than {{lookBackWindow}} after
its event time. The CloudTrail refresh task of camel-aws-secrets-manager
({{CloudTrailReloadTriggerTask}}) moves its position in a similar way; not
changed here. Not run against AWS.
Affected: main (tested); camel-4.23.x, camel-4.22.x and camel-4.18.x have the
same cursor code (read, not run).
Duplicate check (2026-10-17): JIRA text "cloudtrail" (all issues; CAMEL-24082
fixed the static cursor, the paging and the +1 s skip, not late delivery);
GitHub pull requests "cloudtrail": only dependency bumps and CAMEL-24082.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
