[
https://issues.apache.org/jira/browse/CAMEL-25729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18128515#comment-18128515
]
Claus Ibsen commented on CAMEL-25729:
-------------------------------------
Merged to main in https://github.com/apache/camel/pull/28036 (commit
d5f34a8bd8ac), fix version 4.24.0.
_Claude Code on behalf of Claus Ibsen_
> camel-aws-secrets-manager - the CloudTrail refresh task reads one page of
> events per check and starts the next check one second later, so a secret
> update can be missed and the context is not reloaded
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25729
> URL: https://issues.apache.org/jira/browse/CAMEL-25729
> Project: Camel
> Issue Type: Bug
> Components: camel-aws
> Reporter: shashank
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
>
> With {{camel.vault.aws.refreshEnabled=true}} and without SQS notification
> (the default mode), {{CloudTrailReloadTriggerTask.run}} (main 55ed20f14e90,
> lines 276-291) sends one {{LookupEvents}} request per check:
> {code:java}
> LookupEventsRequest.Builder eventsRequestBuilder =
> LookupEventsRequest.builder()
> .maxResults(100)
> .lookupAttributes(... EVENT_SOURCE = "secretsmanager.amazonaws.com"
> ...);
> if (lastTime != null) {
> eventsRequestBuilder.startTime(lastTime.plusMillis(1000));
> }
> LookupEventsResponse response =
> cloudTrailClient.lookupEvents(lookupEventsRequest);
> List<Event> events = response.events();
> if (!events.isEmpty()) {
> lastTime = events.get(0).eventTime();
> }
> {code}
> The lookup is by event source only, so it returns every Secrets Manager call
> of the account and Region, including the read calls ({{GetSecretValue}},
> {{DescribeSecret}}, ...) of every application, newest first. Three problems,
> the same ones CAMEL-24082 fixed in the camel-aws-cloudtrail consumer:
> # {{nextToken}} is never read. The LookupEvents API reference says "The
> default number of results returned is 50, with a maximum of 50 possible" and
> "MaxResults ... Valid Range: Minimum value of 1. Maximum value of 50"; the
> task asks for 100. When more events than one page were recorded after a
> {{PutSecretValue}} / {{UpdateSecret}} of a tracked secret within one check
> period (30 s by default), the update is not in the first page, the position
> moves to the newest event, and the update is never looked at. (If the service
> rejects 100 with {{InvalidMaxResultsException}} instead, every check fails
> with a WARN and the mode never works. This was not checked against AWS; the
> fix is right in both cases.)
> # The next check starts at {{lastTime + 1 s}}. {{StartTime}} is inclusive
> ("events that occur after or at the specified time") and CloudTrail event
> times have a precision of one second, so an update recorded in the same
> second as the newest event of the previous check, but not yet returned by it,
> is skipped.
> # The first check has no start time and looks at the newest events of the
> last 90 days: an old update of a tracked secret among them reloads the
> context about 30 seconds after the start.
> In cases 1 and 2 the context is not reloaded and keeps the old secret values;
> nothing is logged. With rotation, the application keeps using a credential
> that is replaced (and usually revoked soon after).
> h3. Reproduction
> New {{CloudTrailReloadTriggerTaskLookupTest}} with a mocked
> {{CloudTrailClient}} that answers like the service (events at or after
> {{startTime}}, newest first, at most 50 per page, a {{nextToken}} while
> events are left):
> * {{updateBehindMoreThanOnePageOfOtherEventsReloadsTheContext}}: one
> {{PutSecretValue}} of the tracked secret, then 60 {{GetSecretValue}} of
> another secret; one check must reload the context.
> * {{updateInTheSameSecondAsTheNewestEventSeenIsNotSkipped}}: a check sees a
> {{GetSecretValue}} at second T; then a {{PutSecretValue}} at T becomes
> visible; the next check must reload the context, and a third check must not
> reload it again.
> * {{firstCheckLooksUpFromTheStartOfTheTask}}: an update made a day before the
> task started must not reload the context, and the first request has a start
> time.
> On main all three fail ({{the update on the second page must be seen}}, {{the
> update in the same second must be seen}}, {{the first lookup must not read
> the whole event history}}), 3 of 3 runs.
> h3. Proposed fix
> Each check reads every page from the newest event time seen (inclusive), 50
> events per page, and skips the events of that instant that the previous check
> already handled, by event id (as {{CloudtrailConsumer.poll}} does on main
> since CAMEL-24082, lines 68-119). The position starts when the task starts.
> If a page fails, the position does not move, and the next check reads the
> same events again. Upgrade note in the 4.24 guide (the first check no longer
> reacts to an update made before the start; one request per 50 events).
> Cost and limit: a check sends one request per 50 Secrets Manager events of
> the account and Region since the previous check (main: always one). The API
> reference says "The rate of lookup requests is limited to two per second, per
> account, per Region"; a throttled request is retried by the SDK, and if it
> still fails the check fails as on main (WARN) and the next check reads the
> same events again. There is no cap on the pages of a check (unlike the
> cloudtrail consumer's {{maxResults}}, the page size is fixed): an account
> with more than about 100 Secrets Manager calls per second on average cannot
> be followed at that rate, which is the case where main already misses almost
> every update; the SQS notification mode is the better choice there (said in
> the upgrade note).
> With the fix the camel-aws-secrets-manager tests pass: 9 run, 0 failures, 0
> errors, 0 skipped.
> Not covered: an event that becomes visible after a newer one was seen in an
> earlier check (CloudTrail delivery delay); that is the same question as for
> the camel-aws-cloudtrail consumer. Not run against AWS.
> Affected: main (tested); camel-4.23.x, camel-4.22.x and camel-4.18.x have the
> same code (read, not run).
> Duplicate check (2026-10-17): JIRA text "CloudTrailReloadTriggerTask",
> "secrets manager cloudtrail", "aws secrets refresh": CAMEL-18468 (the task),
> CAMEL-21324, CAMEL-24193 (SQS mode), none about paging or the start time;
> GitHub pull requests "CloudTrailReloadTriggerTask": #24726 (CAMEL-24082, the
> consumer only), #24847, #25326.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
