shashank created CAMEL-25729:
--------------------------------
Summary: camel-aws-secrets-manager - the CloudTrail refresh task
reads one page of events per check and starts the next check one second later,
so a secret update can be missed and the context is not reloaded
Key: CAMEL-25729
URL: https://issues.apache.org/jira/browse/CAMEL-25729
Project: Camel
Issue Type: Bug
Components: camel-aws
Reporter: shashank
Assignee: shashank
With {{camel.vault.aws.refreshEnabled=true}} and without SQS notification (the
default mode), {{CloudTrailReloadTriggerTask.run}} (main 55ed20f14e90, lines
276-291) sends one {{LookupEvents}} request per check:
{code:java}
LookupEventsRequest.Builder eventsRequestBuilder = LookupEventsRequest.builder()
.maxResults(100)
.lookupAttributes(... EVENT_SOURCE = "secretsmanager.amazonaws.com"
...);
if (lastTime != null) {
eventsRequestBuilder.startTime(lastTime.plusMillis(1000));
}
LookupEventsResponse response =
cloudTrailClient.lookupEvents(lookupEventsRequest);
List<Event> events = response.events();
if (!events.isEmpty()) {
lastTime = events.get(0).eventTime();
}
{code}
The lookup is by event source only, so it returns every Secrets Manager call of
the account and Region, including the read calls ({{GetSecretValue}},
{{DescribeSecret}}, ...) of every application, newest first. Three problems,
the same ones CAMEL-24082 fixed in the camel-aws-cloudtrail consumer:
# {{nextToken}} is never read. The LookupEvents API reference says "The default
number of results returned is 50, with a maximum of 50 possible" and
"MaxResults ... Valid Range: Minimum value of 1. Maximum value of 50"; the task
asks for 100. When more events than one page were recorded after a
{{PutSecretValue}} / {{UpdateSecret}} of a tracked secret within one check
period (30 s by default), the update is not in the first page, the position
moves to the newest event, and the update is never looked at. (If the service
rejects 100 with {{InvalidMaxResultsException}} instead, every check fails with
a WARN and the mode never works. This was not checked against AWS; the fix is
right in both cases.)
# The next check starts at {{lastTime + 1 s}}. {{StartTime}} is inclusive
("events that occur after or at the specified time") and CloudTrail event times
have a precision of one second, so an update recorded in the same second as the
newest event of the previous check, but not yet returned by it, is skipped.
# The first check has no start time and looks at the newest events of the last
90 days: an old update of a tracked secret among them reloads the context about
30 seconds after the start.
In cases 1 and 2 the context is not reloaded and keeps the old secret values;
nothing is logged. With rotation, the application keeps using a credential that
is replaced (and usually revoked soon after).
h3. Reproduction
New {{CloudTrailReloadTriggerTaskLookupTest}} with a mocked
{{CloudTrailClient}} that answers like the service (events at or after
{{startTime}}, newest first, at most 50 per page, a {{nextToken}} while events
are left):
* {{updateBehindMoreThanOnePageOfOtherEventsReloadsTheContext}}: one
{{PutSecretValue}} of the tracked secret, then 60 {{GetSecretValue}} of another
secret; one check must reload the context.
* {{updateInTheSameSecondAsTheNewestEventSeenIsNotSkipped}}: a check sees a
{{GetSecretValue}} at second T; then a {{PutSecretValue}} at T becomes visible;
the next check must reload the context, and a third check must not reload it
again.
* {{firstCheckLooksUpFromTheStartOfTheTask}}: an update made a day before the
task started must not reload the context, and the first request has a start
time.
On main all three fail ({{the update on the second page must be seen}}, {{the
update in the same second must be seen}}, {{the first lookup must not read the
whole event history}}), 3 of 3 runs.
h3. Proposed fix
Each check reads every page from the newest event time seen (inclusive), 50
events per page, and skips the events of that instant that the previous check
already handled, by event id (as {{CloudtrailConsumer.poll}} does on main since
CAMEL-24082, lines 68-119). The position starts when the task starts. If a page
fails, the position does not move, and the next check reads the same events
again. Upgrade note in the 4.24 guide (the first check no longer reacts to an
update made before the start; one request per 50 events).
Cost and limit: a check sends one request per 50 Secrets Manager events of the
account and Region since the previous check (main: always one). The API
reference says "The rate of lookup requests is limited to two per second, per
account, per Region"; a throttled request is retried by the SDK, and if it
still fails the check fails as on main (WARN) and the next check reads the same
events again. There is no cap on the pages of a check (unlike the cloudtrail
consumer's {{maxResults}}, the page size is fixed): an account with more than
about 100 Secrets Manager calls per second on average cannot be followed at
that rate, which is the case where main already misses almost every update; the
SQS notification mode is the better choice there (said in the upgrade note).
With the fix the camel-aws-secrets-manager tests pass: 9 run, 0 failures, 0
errors, 0 skipped.
Not covered: an event that becomes visible after a newer one was seen in an
earlier check (CloudTrail delivery delay); that is the same question as for the
camel-aws-cloudtrail consumer. Not run against AWS.
Affected: main (tested); camel-4.23.x, camel-4.22.x and camel-4.18.x have the
same code (read, not run).
Duplicate check (2026-10-17): JIRA text "CloudTrailReloadTriggerTask", "secrets
manager cloudtrail", "aws secrets refresh": CAMEL-18468 (the task),
CAMEL-21324, CAMEL-24193 (SQS mode), none about paging or the start time;
GitHub pull requests "CloudTrailReloadTriggerTask": #24726 (CAMEL-24082, the
consumer only), #24847, #25326.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
