shashank created CAMEL-25811:
--------------------------------
Summary: camel-keycloak - the consumer silently skips the older
events when more than maxResults events happen between two polls
Key: CAMEL-25811
URL: https://issues.apache.org/jira/browse/CAMEL-25811
Project: Camel
Issue Type: Bug
Components: camel-keycloak
Reporter: shashank
Assignee: shashank
{{KeycloakConsumer}} (main 62c9a3ebe409) asks Keycloak for
{{first}}/{{maxResults}} events in each poll ({{pollEvents}} line 122,
{{pollAdminEvents}} line 217; {{maxResults}} default 100, {{first}} default 0)
and then makes the newest event its checkpoint ({{lastEventTime =
highestEventTime}}, lines 178 and 277). Events older than the checkpoint are
skipped in every later poll ({{eventTime < lastEventTime}}, lines 140 and 237).
Keycloak returns the events most recent first (the existing
{{KeycloakConsumerLatestEventsTest}} models it the same way). So when more than
{{maxResults}} user events (or admin events) happen between two polls, a poll
gets the newest {{maxResults}} of them and moves the checkpoint to the newest
one; the older new events are not in the answer, and the next poll, which again
asks for the newest {{maxResults}}, skips anything older than the checkpoint.
They are never routed and nothing is logged. This happens in a busy realm (more
than 100 logins within one poll delay), with a smaller {{maxResults}} as in the
examples of the documentation (50), and at the first poll after Keycloak was
unreachable or the polls failed for a while: all but the newest {{maxResults}}
events of that time are lost. The documentation promises fingerprint-based
deduplication so that events are not processed twice; it says nothing about
{{maxResults}} dropping events.
*Fix*: once a checkpoint is set, a poll also reads the following pages
({{first}} + {{maxResults}}, ...), until a page reaches an event older than the
checkpoint or is shorter than {{maxResults}}. When new events arrive while the
pages are read, the next page repeats events of the previous one; the existing
fingerprints skip them. The fingerprint cache is now limited after the poll
(where only the fingerprints at the checkpoint are left) instead of in the
middle of the events of a poll, so a poll of more than 1000 events does not
route events twice. The first poll is unchanged (the latest {{maxResults}}
events). Doc note in the consumer page (and its catalog copy), upgrade note in
the 4.24 guide (a poll can now route more than {{maxResults}} events).
*Tests*: new {{KeycloakConsumerMoreThanMaxResultsTest}} (mocked
{{RealmResource}} that honours {{first}} and {{max}} and returns the newest
events first, {{maxResults=2}}, polls called directly).
{{routesTheEventsBeyondMaxResults}} (one event, then three new ones) fails on
main: {{mock://events Received message count. Expected: <4> but was: <3>}};
{{routesTheAdminEventsBeyondMaxResults}} the same for admin events ({{Expected:
<4> but was: <3>}}); {{newEventsArrivingWhilePagingAreRoutedOnce}} (a new event
arrives while the pages are read) fails on main with {{Expected: <7> but was:
<4>}}. All three pass with the fix. camel-keycloak unit tests: 144 run, 0
failures (the ITs need Docker and were not run).
*How it was found*: variant analysis of CAMEL-25806 (camel-jira newIssues) and
CAMEL-25807 (camel-github2 commits), pattern "a polling consumer fetches the
newest M items and moves its cursor to the newest item it saw", then the JUnit
test above.
*Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (the same single
{{getEvents}} / {{getAdminEvents}} call with {{config.getMaxResults()}} and the
same checkpoint; read, not run). That Keycloak returns the events newest first
is the default order of its admin events endpoints; not run against a Keycloak
server.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
