davsclaus commented on code in PR #28202:
URL: https://github.com/apache/camel/pull/28202#discussion_r4333295365
##########
components/camel-keycloak/src/main/java/org/apache/camel/component/keycloak/KeycloakConsumer.java:
##########
@@ -178,10 +174,51 @@ private Queue<Exchange> pollEvents() throws Exception {
lastEventTime = highestEventTime;
LOG.debug("Updated last event time to: {}", lastEventTime);
}
+ limitFingerprintCache();
return queue;
}
+ /**
+ * Reads the events of a poll. Keycloak returns the most recent events
first, and the newest event of a poll becomes
+ * the checkpoint, so the events that do not fit into the first {@code
maxResults} (the older new ones, for example
+ * after Keycloak was unreachable for a while) would never be returned
again. Once a checkpoint is set, the
+ * following pages are read as well, until a page reaches an event older
than the checkpoint or is the last one.
+ * When new events arrive while the pages are read, the next page repeats
events of the previous one; the
+ * fingerprints skip them.
+ */
+ private <T> List<T> fetchNewEvents(BiFunction<Integer, Integer, List<T>>
fetch, ToLongFunction<T> eventTime) {
+ KeycloakConfiguration config = getEndpoint().getConfiguration();
+ int first = config.getFirst();
+ int max = config.getMaxResults();
+ List<T> page = fetch.apply(first, max);
+ if (lastEventTime == null || max <= 0 || page == null || page.size() <
max) {
Review Comment:
Not blocking (point 2): while `lastEventTime` is null, every poll returns
only the newest `maxResults`. That happens until the first matching event is
seen (new realm, or a filter that has not matched yet). When a poll succeeds
and returns no event, `pollEvents`/`pollAdminEvents` could set `lastEventTime =
0L`. The next poll then pages, and `eventTime < 0` never skips anything.
##########
components/camel-keycloak/src/main/java/org/apache/camel/component/keycloak/KeycloakConfiguration.java:
##########
@@ -71,7 +71,9 @@ public class KeycloakConfiguration implements Cloneable {
@UriParam(description = "Type of events to consume: events or
admin-events", defaultValue = "events")
private String eventType = "events";
- @UriParam(description = "Maximum number of events to retrieve per poll",
defaultValue = "100")
+ @UriParam(description = "Maximum number of events to retrieve per poll.
Later polls read every event since the"
Review Comment:
Optional (point 4): "Maximum number of events to retrieve per poll" no
longer holds for the later polls. The wording of the doc table row fits better,
for example: `The number of events the first poll retrieves, and the page size
of the later polls, which read every event since the previous poll`. The
generated JSON and DSL files then need regenerating.
##########
components/camel-keycloak/src/main/java/org/apache/camel/component/keycloak/KeycloakConsumer.java:
##########
@@ -178,10 +174,51 @@ private Queue<Exchange> pollEvents() throws Exception {
lastEventTime = highestEventTime;
LOG.debug("Updated last event time to: {}", lastEventTime);
}
+ limitFingerprintCache();
return queue;
}
+ /**
+ * Reads the events of a poll. Keycloak returns the most recent events
first, and the newest event of a poll becomes
+ * the checkpoint, so the events that do not fit into the first {@code
maxResults} (the older new ones, for example
+ * after Keycloak was unreachable for a while) would never be returned
again. Once a checkpoint is set, the
+ * following pages are read as well, until a page reaches an event older
than the checkpoint or is the last one.
+ * When new events arrive while the pages are read, the next page repeats
events of the previous one; the
+ * fingerprints skip them.
+ */
+ private <T> List<T> fetchNewEvents(BiFunction<Integer, Integer, List<T>>
fetch, ToLongFunction<T> eventTime) {
+ KeycloakConfiguration config = getEndpoint().getConfiguration();
+ int first = config.getFirst();
+ int max = config.getMaxResults();
+ List<T> page = fetch.apply(first, max);
+ if (lastEventTime == null || max <= 0 || page == null || page.size() <
max) {
+ return page;
+ }
+ List<T> answer = new ArrayList<>(page);
Review Comment:
Not blocking (point 1): `answer` collects every page since the checkpoint,
and `pollEvents` builds all the exchanges before `processBatch` routes any.
After a long outage that can be a lot of events at once. Fine for this PR.
Maybe one sentence about it in `keycloak-consumer.adoc`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
