[
https://issues.apache.org/jira/browse/CAMEL-25811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129807#comment-18129807
]
Claus Ibsen commented on CAMEL-25811:
-------------------------------------
Merged via https://github.com/apache/camel/pull/28202
_Claude Code on behalf of davsclaus_
> camel-keycloak - the consumer silently skips the older events when more than
> maxResults events happen between two polls
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25811
> URL: https://issues.apache.org/jira/browse/CAMEL-25811
> Project: Camel
> Issue Type: Bug
> Components: camel-keycloak
> Reporter: shashank
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
>
> {{KeycloakConsumer}} (main 62c9a3ebe409) asks Keycloak for
> {{first}}/{{maxResults}} events in each poll ({{pollEvents}} line 122,
> {{pollAdminEvents}} line 217; {{maxResults}} default 100, {{first}} default
> 0) and then makes the newest event its checkpoint ({{lastEventTime =
> highestEventTime}}, lines 178 and 277). Events older than the checkpoint are
> skipped in every later poll ({{eventTime < lastEventTime}}, lines 140 and
> 237).
> Keycloak returns the events most recent first (the existing
> {{KeycloakConsumerLatestEventsTest}} models it the same way). So when more
> than {{maxResults}} user events (or admin events) happen between two polls, a
> poll gets the newest {{maxResults}} of them and moves the checkpoint to the
> newest one; the older new events are not in the answer, and the next poll,
> which again asks for the newest {{maxResults}}, skips anything older than the
> checkpoint. They are never routed and nothing is logged. This happens in a
> busy realm (more than 100 logins within one poll delay), with a smaller
> {{maxResults}} as in the examples of the documentation (50), and at the first
> poll after Keycloak was unreachable or the polls failed for a while: all but
> the newest {{maxResults}} events of that time are lost. The documentation
> promises fingerprint-based deduplication so that events are not processed
> twice; it says nothing about {{maxResults}} dropping events.
> *Fix*: once a checkpoint is set, a poll also reads the following pages
> ({{first}} + {{maxResults}}, ...), until a page reaches an event older than
> the checkpoint or is shorter than {{maxResults}}. When new events arrive
> while the pages are read, the next page repeats events of the previous one;
> the existing fingerprints skip them. The fingerprint cache is now limited
> after the poll (where only the fingerprints at the checkpoint are left)
> instead of in the middle of the events of a poll, so a poll of more than 1000
> events does not route events twice. The first poll is unchanged (the latest
> {{maxResults}} events). Doc note in the consumer page (and its catalog copy),
> upgrade note in the 4.24 guide (a poll can now route more than {{maxResults}}
> events).
> *Tests*: new {{KeycloakConsumerMoreThanMaxResultsTest}} (mocked
> {{RealmResource}} that honours {{first}} and {{max}} and returns the newest
> events first, {{maxResults=2}}, polls called directly).
> {{routesTheEventsBeyondMaxResults}} (one event, then three new ones) fails on
> main: {{mock://events Received message count. Expected: <4> but was: <3>}};
> {{routesTheAdminEventsBeyondMaxResults}} the same for admin events
> ({{Expected: <4> but was: <3>}});
> {{newEventsArrivingWhilePagingAreRoutedOnce}} (a new event arrives while the
> pages are read) fails on main with {{Expected: <7> but was: <4>}}. All three
> pass with the fix. camel-keycloak unit tests: 144 run, 0 failures (the ITs
> need Docker and were not run).
> *How it was found*: variant analysis of CAMEL-25806 (camel-jira newIssues)
> and CAMEL-25807 (camel-github2 commits), pattern "a polling consumer fetches
> the newest M items and moves its cursor to the newest item it saw", then the
> JUnit test above.
> *Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (the same
> single {{getEvents}} / {{getAdminEvents}} call with
> {{config.getMaxResults()}} and the same checkpoint; read, not run). That
> Keycloak returns the events newest first is the default order of its admin
> events endpoints; not run against a Keycloak server.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
