This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new 5de4f6eb8df6 CAMEL-26968: camel-aws2-kinesis - iteratorType LATEST: an
expired iterator of a shard without a routed record skips the records stored in
between (#30133)
5de4f6eb8df6 is described below
commit 5de4f6eb8df6637e1e61bc3929ed65d15bafc645
Author: allthingssecurity <[email protected]>
AuthorDate: Mon Nov 16 18:59:39 2026 +0530
CAMEL-26968: camel-aws2-kinesis - iteratorType LATEST: an expired iterator
of a shard without a routed record skips the records stored in between (#30133)
Follow-up of #27753 (CAMEL-25569), for the case it left open. Found by the
Kinesis DST world of #29993 (CAMEL-26948), whose default set accepted this case
(`-Ddst.kinesis.strictLatest=true` failed 44 of 2000 seeds); this PR now
removes that excuse from the world. Reproduced here with a plain unit test,
outside the DST harness.
Since CAMEL-25569, when a cached shard iterator has expired,
`Kinesis2Consumer.getShardIterator` asks for `AFTER_SEQUENCE_NUMBER` with the
last record routed from the shard. A shard from which no record was routed yet
has no such sequence number, so the consumer asked for the configured
`iteratorType` again. With `LATEST` that is a new start point, and the records
stored between the first iterator and the new one were skipped. An iterator
expires when it is not used for 5 minutes (a r [...]
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
.../org/apache/camel/catalog/docs/test-dst.adoc | 6 +--
.../component/aws2/kinesis/Kinesis2Consumer.java | 20 +++++++++-
.../KinesisConsumerClosedShardWithFailTest.java | 9 +++++
.../KinesisConsumerClosedShardWithSilentTest.java | 9 +++++
.../kinesis/KinesisConsumerMultiShardTest.java | 9 +++++
.../aws2/kinesis/KinesisConsumerPositionTest.java | 46 ++++++++++++++++++++++
.../component/aws2/kinesis/dst/FakeKinesis.java | 27 ++-----------
.../component/aws2/kinesis/dst/KinesisWorld.java | 9 ++---
.../camel-test-dst/src/main/docs/test-dst.adoc | 6 +--
.../ROOT/pages/camel-4x-upgrade-guide-4_24.adoc | 2 +
10 files changed, 105 insertions(+), 38 deletions(-)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
index 7c01a28237df..3bf2fdf08aeb 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
@@ -600,10 +600,8 @@ was also checked with planted faults), and the main code
of 043ac9ec84ab (CAMEL-
(CAMEL-25942) for Kinesis (`KinesisConsumerDstTest`). Some system properties
belong to a world rather than to the harness:
`dst.sqs.visibilityTimeouts` (the visibility timeouts the SQS world picks
from, default `2,3,5,10`; `1` shows
CAMEL-25737), `dst.replayCheckSeed` (the seed of the in-JVM replay test of the
Event Hubs world, default 1),
-`dst.share.pollOnError` (the pollOnError strategies the Kafka share world
picks from),
-`dst.pulsar.subscribeFailRates` (the subscribe failure rates of the Pulsar
world, `0` runs without subscribe failures)
-and `dst.kinesis.strictLatest` (`true` counts the records that a LATEST
iterator asked for again after an expiry skips,
-CAMEL-26968, as missing).
+`dst.share.pollOnError` (the pollOnError strategies the Kafka share world
picks from)
+and `dst.pulsar.subscribeFailRates` (the subscribe failure rates of the Pulsar
world, `0` runs without subscribe failures).
== Limits
diff --git
a/components/camel-aws/camel-aws2-kinesis/src/main/java/org/apache/camel/component/aws2/kinesis/Kinesis2Consumer.java
b/components/camel-aws/camel-aws2-kinesis/src/main/java/org/apache/camel/component/aws2/kinesis/Kinesis2Consumer.java
index 472b3587bf99..bd4a44cd0cc5 100644
---
a/components/camel-aws/camel-aws2-kinesis/src/main/java/org/apache/camel/component/aws2/kinesis/Kinesis2Consumer.java
+++
b/components/camel-aws/camel-aws2-kinesis/src/main/java/org/apache/camel/component/aws2/kinesis/Kinesis2Consumer.java
@@ -81,6 +81,10 @@ public class Kinesis2Consumer extends
ScheduledBatchPollingConsumer implements R
// cached one has expired. Like the cached iterators it is not cleared in
doStop, and a route that is stopped and
// started gets a new consumer.
private final Map<String, String> lastSequenceNumbers = new
ConcurrentHashMap<>();
+ // The time the consumer asked for the first LATEST iterator of each
shard: where it continues a shard from which it
+ // has not routed a record yet when that iterator has expired (a LATEST
iterator again would skip the records stored
+ // in between)
+ private final Map<String, Instant> latestStartTimes = new
ConcurrentHashMap<>();
// Written by the shard monitor, read by poll(): the shards of the lists
seen so far, and those of them that
// appeared after the first list as the child of a listed shard
(resharding).
private final Set<String> knownShardIds = ConcurrentHashMap.newKeySet();
@@ -286,6 +290,11 @@ public class Kinesis2Consumer extends
ScheduledBatchPollingConsumer implements R
// shard again (TRIM_HORIZON, AT_TIMESTAMP, a sequence number)
or skip the new ones (LATEST)
request.shardIteratorType(ShardIteratorType.AFTER_SEQUENCE_NUMBER)
.startingSequenceNumber(lastSequenceNumber);
+ } else if (latestStartTimes.containsKey(shardId)) {
+ // the shard started at LATEST and the expired iterator
returned no record: continue from the time of
+ // that start, so the records stored since then are routed
(Kinesis compares this time of the consumer's
+ // clock with the arrival time it gave each record)
+
request.shardIteratorType(ShardIteratorType.AT_TIMESTAMP).timestamp(latestStartTimes.get(shardId));
} else if (childShardIds.contains(shardId)) {
// a shard created by resharding while the consumer was
reading its parent: all its records are new
request.shardIteratorType(ShardIteratorType.TRIM_HORIZON);
@@ -304,12 +313,14 @@ public class Kinesis2Consumer extends
ScheduledBatchPollingConsumer implements R
resume(shardId, request);
+ GetShardIteratorRequest iteratorRequest = request.build();
+ Instant requestedAt = Instant.now();
GetShardIteratorResponse result;
if (getEndpoint().getConfiguration().isAsyncClient()) {
try {
result = kinesisConnection
.getAsyncClient(getEndpoint())
- .getShardIterator(request.build())
+ .getShardIterator(iteratorRequest)
.get();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
@@ -320,7 +331,12 @@ public class Kinesis2Consumer extends
ScheduledBatchPollingConsumer implements R
} else {
result = kinesisConnection
.getClient(getEndpoint())
- .getShardIterator(request.build());
+ .getShardIterator(iteratorRequest);
+ }
+ if (iteratorRequest.shardIteratorType() ==
ShardIteratorType.LATEST) {
+ // the iterator points after the records stored before Kinesis
handled the request, which was after
+ // requestedAt
+ latestStartTimes.putIfAbsent(shardId, requestedAt);
}
currentShardIterators.put(shardId, result.shardIterator());
diff --git
a/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/KinesisConsumerClosedShardWithFailTest.java
b/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/KinesisConsumerClosedShardWithFailTest.java
index 6c19438dc4bf..b9e4cebded58 100644
---
a/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/KinesisConsumerClosedShardWithFailTest.java
+++
b/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/KinesisConsumerClosedShardWithFailTest.java
@@ -22,6 +22,7 @@ import java.util.concurrent.TimeUnit;
import org.apache.camel.AsyncProcessor;
import org.apache.camel.CamelContext;
import org.apache.camel.impl.DefaultCamelContext;
+import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
@@ -62,6 +63,14 @@ public class KinesisConsumerClosedShardWithFailTest {
private Kinesis2Consumer underTest;
+ @AfterEach
+ public void tearDown() {
+ // the consumer polls on a thread of its own: stop it, so it does not
go on polling during the later tests of
+ // the module (a poll reads the clock, which the DST test of the
module controls while it runs)
+ underTest.stop();
+ context.stop();
+ }
+
@BeforeEach
public void setup() {
SequenceNumberRange range =
SequenceNumberRange.builder().endingSequenceNumber("20").build();
diff --git
a/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/KinesisConsumerClosedShardWithSilentTest.java
b/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/KinesisConsumerClosedShardWithSilentTest.java
index b1f996362a64..6112bf079fbb 100644
---
a/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/KinesisConsumerClosedShardWithSilentTest.java
+++
b/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/KinesisConsumerClosedShardWithSilentTest.java
@@ -23,6 +23,7 @@ import java.util.concurrent.TimeUnit;
import org.apache.camel.AsyncProcessor;
import org.apache.camel.CamelContext;
import org.apache.camel.impl.DefaultCamelContext;
+import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
@@ -68,6 +69,14 @@ public class KinesisConsumerClosedShardWithSilentTest {
private Kinesis2Consumer underTest;
+ @AfterEach
+ public void tearDown() {
+ // the consumer polls on a thread of its own: stop it, so it does not
go on polling during the later tests of
+ // the module (a poll reads the clock, which the DST test of the
module controls while it runs)
+ underTest.stop();
+ context.stop();
+ }
+
@BeforeEach
public void setup() {
SequenceNumberRange range =
SequenceNumberRange.builder().endingSequenceNumber("2").build();
diff --git
a/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/KinesisConsumerMultiShardTest.java
b/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/KinesisConsumerMultiShardTest.java
index 64184b641ad1..48184f996174 100644
---
a/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/KinesisConsumerMultiShardTest.java
+++
b/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/KinesisConsumerMultiShardTest.java
@@ -23,6 +23,7 @@ import java.util.concurrent.TimeUnit;
import org.apache.camel.AsyncProcessor;
import org.apache.camel.CamelContext;
import org.apache.camel.impl.DefaultCamelContext;
+import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
@@ -68,6 +69,14 @@ public class KinesisConsumerMultiShardTest {
private Kinesis2Consumer underTest;
+ @AfterEach
+ public void tearDown() {
+ // the consumer polls on a thread of its own: stop it, so it does not
go on polling during the later tests of
+ // the module (a poll reads the clock, which the DST test of the
module controls while it runs)
+ underTest.stop();
+ context.stop();
+ }
+
@BeforeEach
public void setup() {
SequenceNumberRange range =
SequenceNumberRange.builder().endingSequenceNumber("2").build();
diff --git
a/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/KinesisConsumerPositionTest.java
b/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/KinesisConsumerPositionTest.java
index 1052c7c52a0b..bc51f63cb1ee 100644
---
a/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/KinesisConsumerPositionTest.java
+++
b/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/KinesisConsumerPositionTest.java
@@ -17,6 +17,7 @@
package org.apache.camel.component.aws2.kinesis;
import java.nio.charset.StandardCharsets;
+import java.time.Instant;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
@@ -131,6 +132,37 @@ class KinesisConsumerPositionTest {
assertEquals(List.of("a", "b"), routed);
}
+ @Test
+ void
expiredIteratorOfAShardWithoutRoutedRecordsDoesNotSkipTheRecordsOfTheGapWithLatest()
throws Exception {
+ stream.addShard("shard-0", null);
+ // stored well before the consumer starts the shard
+ stream.put("shard-0", "old", Instant.now().minusSeconds(60));
+ startConsumer(ShardIteratorType.LATEST, 1);
+
+ assertEquals(0, consumer.poll());
+ GetShardIteratorRequest first =
stream.iteratorRequests.get(stream.iteratorRequests.size() - 1);
+ assertEquals(ShardIteratorType.LATEST, first.shardIteratorType());
+
+ // the iterator expires before the consumer routed a record of the
shard (the route was suspended, or the delay
+ // is longer than the 5 minutes an iterator is valid)
+ stream.put("shard-0", "a");
+ stream.expireIterators();
+ stream.put("shard-0", "b");
+
+ assertEquals(2, consumer.poll(), "the records stored since the
consumer started the shard");
+ assertEquals(List.of("a", "b"), routed);
+ GetShardIteratorRequest last =
stream.iteratorRequests.get(stream.iteratorRequests.size() - 1);
+ assertEquals(ShardIteratorType.AT_TIMESTAMP, last.shardIteratorType());
+
+ // once a record is routed, the next new iterator continues after it
+ stream.expireIterators();
+ stream.put("shard-0", "c");
+ assertEquals(1, consumer.poll());
+ assertEquals(List.of("a", "b", "c"), routed);
+ last = stream.iteratorRequests.get(stream.iteratorRequests.size() - 1);
+ assertEquals(ShardIteratorType.AFTER_SEQUENCE_NUMBER,
last.shardIteratorType());
+ }
+
@Test
void expiredIteratorOfAShardWithoutRoutedRecordsUsesTheConfiguredType()
throws Exception {
stream.addShard("shard-0", null);
@@ -430,8 +462,14 @@ class KinesisConsumerPositionTest {
}
synchronized void put(String shardId, String body) {
+ // Kinesis sets the approximate arrival timestamp when it stores
the record
+ put(shardId, body, Instant.now());
+ }
+
+ synchronized void put(String shardId, String body, Instant arrival) {
List<Record> list = records.get(shardId);
list.add(Record.builder().sequenceNumber(String.valueOf(list.size() + 1))
+ .approximateArrivalTimestamp(arrival)
.data(SdkBytes.fromString(body,
StandardCharsets.UTF_8)).build());
}
@@ -472,6 +510,14 @@ class KinesisConsumerPositionTest {
case AFTER_SEQUENCE_NUMBER:
position =
Integer.parseInt(request.startingSequenceNumber());
break;
+ case AT_TIMESTAMP:
+ // the first record stored at or after the timestamp
+ position = 0;
+ while (position < list.size()
+ &&
list.get(position).approximateArrivalTimestamp().isBefore(request.timestamp()))
{
+ position++;
+ }
+ break;
default:
throw new IllegalArgumentException("Not expected in this
test: " + request.shardIteratorType());
}
diff --git
a/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/dst/FakeKinesis.java
b/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/dst/FakeKinesis.java
index 5a5b24ef95aa..29d6feafc019 100644
---
a/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/dst/FakeKinesis.java
+++
b/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/dst/FakeKinesis.java
@@ -126,8 +126,6 @@ final class FakeKinesis implements KinesisClient {
private record Iterator(String shard, int position, long issuedAt) {
}
- private static final boolean STRICT_LATEST =
Boolean.getBoolean("dst.kinesis.strictLatest");
-
private final Dst dst;
private final DstLedger ledger;
final Map<String, FakeShard> shards = new LinkedHashMap<>();
@@ -140,8 +138,6 @@ final class FakeKinesis implements KinesisClient {
*/
final Set<Rec> skippedByLatest = new HashSet<>();
private final Set<String> latestStarts = new HashSet<>();
- /** epoch/shard of an iterator that expired, until the next iterator of
that shard. */
- private final Set<String> expiredBefore = new HashSet<>();
/** The shards of the first complete ListShards answer of each epoch, and
of the listing in progress. */
private final Map<Integer, Set<String>> firstLists = new HashMap<>();
private Set<String> listing = new HashSet<>();
@@ -246,7 +242,6 @@ final class FakeKinesis implements KinesisClient {
.message("dst: Rate exceeded for shard " +
shard.id).build());
}
ShardIteratorType type = request.shardIteratorType();
- boolean afterExpiry = expiredBefore.remove(epoch + "/" + shard.id);
int position;
switch (type) {
case TRIM_HORIZON:
@@ -256,17 +251,14 @@ final class FakeKinesis implements KinesisClient {
case LATEST:
// [I] "Start reading just after the most recent record in the
shard"
position = shard.records.size();
- // skipped by design when the consumer starts a shard of its
first shard list (the first iterator of the
- // shard in an epoch), and also when, right after its iterator
of such a shard expired, it gets a LATEST
- // iterator again for that shard and has not routed a record
of it yet (the consumer has no position to
- // continue from; this second case is CAMEL-26968, and
-Ddst.kinesis.strictLatest=true excuses only the
- // first). Never for a shard with a
+ // skipped by design only when the consumer starts a shard of
its first shard list (the first iterator
+ // of the shard in an epoch). Never for a later LATEST
iterator of the shard, for example after an expiry
+ // (since CAMEL-26968 the consumer continues from the time of
its first LATEST iterator), a shard with a
// checkpoint (the resume strategy starts it after the
checkpoint) or a shard created later (read from
// its start)
boolean first = latestStarts.add(epoch + "/" + shard.id);
boolean listedFirst = firstLists.getOrDefault(epoch,
Set.of()).contains(shard.id);
- if (listedFirst && !hasCheckpoint.test(shard.id)
- && (first || !STRICT_LATEST && afterExpiry &&
!routedFrom(shard))) {
+ if (first && listedFirst && !hasCheckpoint.test(shard.id)) {
for (int i = 0; i < position; i++) {
skippedByLatest.add(shard.records.get(i));
}
@@ -298,16 +290,6 @@ final class FakeKinesis implements KinesisClient {
return GetShardIteratorResponse.builder().shardIterator(it).build();
}
- /** Whether the consumer of the current epoch has routed a record of the
shard. */
- private boolean routedFrom(FakeShard shard) {
- for (Rec r : shard.records) {
- if (r.routedIn.contains(epoch)) {
- return true;
- }
- }
- return false;
- }
-
/**
* [I] AT_SEQUENCE_NUMBER and AFTER_SEQUENCE_NUMBER need the sequence
number of a record of that shard;
* InvalidArgumentException otherwise ("Invalid StartingSequenceNumber").
@@ -338,7 +320,6 @@ final class FakeKinesis implements KinesisClient {
// [G] ExpiredIteratorException: "The provided iterator exceeds the
maximum age allowed."
if (dst.now() - it.issuedAt() >= ITERATOR_LIFETIME_MILLIS) {
ledger.record("iteratorExpired", it.shard(),
request.shardIterator());
- expiredBefore.add(epoch + "/" + it.shard());
throw ExpiredIteratorException.builder()
.message("Iterator expired. The iterator was created at
time " + it.issuedAt() + " ms").build();
}
diff --git
a/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/dst/KinesisWorld.java
b/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/dst/KinesisWorld.java
index 05b7b59c42fc..96912997aa71 100644
---
a/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/dst/KinesisWorld.java
+++
b/components/camel-aws/camel-aws2-kinesis/src/test/java/org/apache/camel/component/aws2/kinesis/dst/KinesisWorld.java
@@ -74,11 +74,10 @@ import org.apache.camel.test.dst.Invariants;
* the checkpoint, which is the last completed exchange, so a record routed
after it that did not complete (it failed)
* may be routed again by the same consumer.
* <p>
- * Also excused, unless {@code -Ddst.kinesis.strictLatest=true}: with
iteratorType=LATEST, when the iterator of a shard
- * of the first shard list from which the consumer has not routed a record yet
expires, the consumer has no sequence
- * number to continue from and asks for a LATEST iterator again, which skips
the records stored since its previous
- * iterator (for example while the route was suspended for more than five
minutes). This is CAMEL-26968. The default set
- * accepts this; {@code -Ddst.kinesis.strictLatest=true} counts those records
as missing.
+ * With iteratorType=LATEST, when the iterator of a shard of the first shard
list from which the consumer has not routed
+ * a record yet expires (for example while the route was suspended for more
than five minutes), the consumer continues
+ * from the time of its first LATEST iterator of that shard (CAMEL-26968), so
the records stored since then count as
+ * missing if they are not routed.
* <p>
* Not explored: a route stop while a poll runs (see above), the KCL consumer
(useKclConsumers), the asynchronous
* client, a configured shardId, AT_SEQUENCE_NUMBER, AFTER_SEQUENCE_NUMBER and
AT_TIMESTAMP as the configured iterator
diff --git a/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
b/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
index 7c01a28237df..3bf2fdf08aeb 100644
--- a/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
+++ b/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
@@ -600,10 +600,8 @@ was also checked with planted faults), and the main code
of 043ac9ec84ab (CAMEL-
(CAMEL-25942) for Kinesis (`KinesisConsumerDstTest`). Some system properties
belong to a world rather than to the harness:
`dst.sqs.visibilityTimeouts` (the visibility timeouts the SQS world picks
from, default `2,3,5,10`; `1` shows
CAMEL-25737), `dst.replayCheckSeed` (the seed of the in-JVM replay test of the
Event Hubs world, default 1),
-`dst.share.pollOnError` (the pollOnError strategies the Kafka share world
picks from),
-`dst.pulsar.subscribeFailRates` (the subscribe failure rates of the Pulsar
world, `0` runs without subscribe failures)
-and `dst.kinesis.strictLatest` (`true` counts the records that a LATEST
iterator asked for again after an expiry skips,
-CAMEL-26968, as missing).
+`dst.share.pollOnError` (the pollOnError strategies the Kafka share world
picks from)
+and `dst.pulsar.subscribeFailRates` (the subscribe failure rates of the Pulsar
world, `0` runs without subscribe failures).
== Limits
diff --git
a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
index 3f1bf1061f29..2eb785749a2b 100644
--- a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
+++ b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
@@ -5656,6 +5656,8 @@ When a cached shard iterator has expired (an iterator is
valid for 5 minutes, so
longer, or with a `delay` or back-off above 5 minutes), the consumer now
continues after the last record it routed
from that shard. Previously it requested a new iterator of the configured
`iteratorType`: with `TRIM_HORIZON` (the
default) it routed all the records of the shard again, with `LATEST` it
skipped the records stored meanwhile.
+With `LATEST`, a shard from which the consumer has not routed a record yet
continues from the time the consumer
+started reading that shard.
A shard created by a split or a merge while the consumer is running is now
read from its first record, whatever the
`iteratorType`. Previously, with `LATEST`, the records stored in the new shard
before the consumer listed it were