This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new 6ec5f70e7f02 CAMEL-26967: camel-kafka - breakOnFirstError with
synchronous manual commits: a commitSync timeout in the break skips the failed
record and the rest of the poll (#30132)
6ec5f70e7f02 is described below
commit 6ec5f70e7f02eec2f694f6278b32579cf18fa4db
Author: allthingssecurity <[email protected]>
AuthorDate: Tue Nov 17 20:38:06 2026 +0530
CAMEL-26967: camel-kafka - breakOnFirstError with synchronous manual
commits: a commitSync timeout in the break skips the failed record and the rest
of the poll (#30132)
With breakOnFirstError and the SyncCommitManager, a failed record makes
processException commit the records routed before it and return the
break, so the consumer reconnects and polls again from the committed
offset. When that commitSync threw (a TimeoutException when the timeout
expires first, or CommitFailedException / RebalanceInProgressException),
the exception left the poll and reached the poll exception strategy.
With pollOnError ERROR_HANDLER (the default), DISCARD or RETRY the
fetcher went on polling with the same Kafka consumer, whose position was
after the whole poll: the failed record and the records after it were
never routed, and the next manual commit committed past them.
A KafkaException of that commit (not WakeupException or InterruptException)
is now reported to the exception handler of the consumer, and the break is
returned as before: the new Kafka consumer polls from the committed offset,
which is the failed record when the commit was applied, or an earlier one.
So that commit failure no longer goes through pollOnError: it reaches the
exception handler of the consumer (logged, or routed with
bridgeErrorHandler=true) instead of the error handler of the route through
the ERROR_HANDLER strategy, and pollOnError=STOP no longer stops the
consumer on it (the consumer reconnects, as for the break).
Found by the Kafka DST world of CAMEL-26947 with
-Ddst.kafka.syncBreakOnFirstError=true (seed 95). New
KafkaConsumerBreakOnFirstErrorCommitTimeoutTest.
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../org/apache/camel/catalog/components/kafka.json | 2 +-
.../org/apache/camel/catalog/docs/test-dst.adoc | 2 +-
.../org/apache/camel/component/kafka/kafka.json | 2 +-
.../streaming/KafkaRecordStreamingProcessor.java | 26 +-
...ConsumerBreakOnFirstErrorCommitTimeoutTest.java | 269 +++++++++++++++++++++
.../camel/component/kafka/dst/KafkaWorld.java | 16 +-
.../camel-test-dst/src/main/docs/test-dst.adoc | 2 +-
.../ROOT/pages/camel-4x-upgrade-guide-4_24.adoc | 11 +
8 files changed, 312 insertions(+), 18 deletions(-)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/kafka.json
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/kafka.json
index 1a9e7f4c66ef..54d4737a98ef 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/kafka.json
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/kafka.json
@@ -301,6 +301,6 @@
"sslTruststoreType": { "index": 121, "kind": "parameter", "displayName":
"Ssl Truststore Type", "group": "security", "label": "common,security",
"required": false, "type": "string", "javaType": "java.lang.String",
"deprecated": false, "autowired": false, "secret": false, "defaultValue":
"JKS", "configurationClass":
"org.apache.camel.component.kafka.KafkaConfiguration", "configurationField":
"configuration", "description": "The file format of the trust store file. The
default value is [...]
},
"dstWorlds": {
- "KafkaWorld": { "javaType":
"org.apache.camel.component.kafka.dst.KafkaWorld", "fake": "FakeKafka: a topic
with one or two partitions and a consumer group (classic protocol, eager
rebalance), behind the Consumer interface of KafkaClientFactory", "covers": [
"commits by Kafka's auto-commit, by manual synchronous or asynchronous commits
in the route, and per batch with batching=true", "breakOnFirstError",
"pollOnError ERROR_HANDLER, DISCARD, RETRY and RECONNECT", "a poll that fails
whi [...]
+ "KafkaWorld": { "javaType":
"org.apache.camel.component.kafka.dst.KafkaWorld", "fake": "FakeKafka: a topic
with one or two partitions and a consumer group (classic protocol, eager
rebalance), behind the Consumer interface of KafkaClientFactory", "covers": [
"commits by Kafka's auto-commit, by manual synchronous or asynchronous commits
in the route, and per batch with batching=true", "breakOnFirstError",
"pollOnError ERROR_HANDLER, DISCARD, RETRY and RECONNECT", "a poll that fails
whi [...]
}
}
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
index d6ee0ac24ad9..a78c6d18b373 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
@@ -121,7 +121,7 @@ a|* commits by Kafka's auto-commit, by manual synchronous
or asynchronous commit
* route restart, also while the consumer polls
* route suspend and resume, with a second request while the consumer handles
the first
* partitions revoked and assigned again by a rebalance while records are
processed
-|CAMEL-25313, CAMEL-25756
+|CAMEL-25313, CAMEL-25756, CAMEL-26967
|camel-kafka-share
|`KafkaShareWorld` (`KafkaShareConsumerDstTest`)
diff --git
a/components/camel-kafka/src/generated/resources/META-INF/org/apache/camel/component/kafka/kafka.json
b/components/camel-kafka/src/generated/resources/META-INF/org/apache/camel/component/kafka/kafka.json
index 1a9e7f4c66ef..54d4737a98ef 100644
---
a/components/camel-kafka/src/generated/resources/META-INF/org/apache/camel/component/kafka/kafka.json
+++
b/components/camel-kafka/src/generated/resources/META-INF/org/apache/camel/component/kafka/kafka.json
@@ -301,6 +301,6 @@
"sslTruststoreType": { "index": 121, "kind": "parameter", "displayName":
"Ssl Truststore Type", "group": "security", "label": "common,security",
"required": false, "type": "string", "javaType": "java.lang.String",
"deprecated": false, "autowired": false, "secret": false, "defaultValue":
"JKS", "configurationClass":
"org.apache.camel.component.kafka.KafkaConfiguration", "configurationField":
"configuration", "description": "The file format of the trust store file. The
default value is [...]
},
"dstWorlds": {
- "KafkaWorld": { "javaType":
"org.apache.camel.component.kafka.dst.KafkaWorld", "fake": "FakeKafka: a topic
with one or two partitions and a consumer group (classic protocol, eager
rebalance), behind the Consumer interface of KafkaClientFactory", "covers": [
"commits by Kafka's auto-commit, by manual synchronous or asynchronous commits
in the route, and per batch with batching=true", "breakOnFirstError",
"pollOnError ERROR_HANDLER, DISCARD, RETRY and RECONNECT", "a poll that fails
whi [...]
+ "KafkaWorld": { "javaType":
"org.apache.camel.component.kafka.dst.KafkaWorld", "fake": "FakeKafka: a topic
with one or two partitions and a consumer group (classic protocol, eager
rebalance), behind the Consumer interface of KafkaClientFactory", "covers": [
"commits by Kafka's auto-commit, by manual synchronous or asynchronous commits
in the route, and per batch with batching=true", "breakOnFirstError",
"pollOnError ERROR_HANDLER, DISCARD, RETRY and RECONNECT", "a poll that fails
whi [...]
}
}
diff --git
a/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/support/streaming/KafkaRecordStreamingProcessor.java
b/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/support/streaming/KafkaRecordStreamingProcessor.java
index ab2aa6afa5ea..2ff958e6c5e5 100644
---
a/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/support/streaming/KafkaRecordStreamingProcessor.java
+++
b/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/support/streaming/KafkaRecordStreamingProcessor.java
@@ -61,7 +61,10 @@ import org.apache.camel.spi.ExceptionHandler;
import org.apache.camel.support.EventNotifierSupport;
import org.apache.camel.support.LanguageSupport;
import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.KafkaException;
import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.InterruptException;
+import org.apache.kafka.common.errors.WakeupException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -727,12 +730,25 @@ final class KafkaRecordStreamingProcessor extends
KafkaRecordProcessor {
// we should just do a commit (vs the original forceCommit)
// when route uses NOOP Commit Manager it will rely
// on the route implementation to explicitly commit offset
- // when route uses Synch/Asynch Commit Manager it will
- // ALWAYS commit the offset for the failing record
- // and will ALWAYS retry it
- commitManager.commit(topicPartition);
+ // when route uses Synch/Asynch Commit Manager it commits the
offset of the
+ // records before the failing record. That commit can fail or time
out: the
+ // consumer still breaks out and polls again from the last
committed offset
+ try {
+ commitManager.commit(topicPartition);
+ } catch (WakeupException | InterruptException e) {
+ throw e;
+ } catch (KafkaException e) {
+ // the commit failed or timed out (a commitSync timeout may
still have been applied): break out anyway,
+ // so the new Kafka consumer polls again from the committed
offset. Leaving the batch through the
+ // exception would let the same consumer go on after the
records of this batch that were not routed
+ exceptionHandler.handleException(
+ "Error committing the offset of the records before the
failed record on partition "
+ + topicPartition
+ + ". The consumer polls again
from the last committed offset.",
+ e);
+ }
- // continue to next partition
+ // break out: the consumer reconnects and polls again from the
committed offset
return true;
} else {
// will handle/log the exception and then continue to next
diff --git
a/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/KafkaConsumerBreakOnFirstErrorCommitTimeoutTest.java
b/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/KafkaConsumerBreakOnFirstErrorCommitTimeoutTest.java
new file mode 100644
index 000000000000..85b9f5a1cad8
--- /dev/null
+++
b/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/KafkaConsumerBreakOnFirstErrorCommitTimeoutTest.java
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.kafka;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
+
+import org.apache.camel.BindToRegistry;
+import org.apache.camel.Exchange;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.kafka.consumer.KafkaManualCommit;
+import org.apache.camel.spi.ExceptionHandler;
+import org.apache.camel.test.junit6.CamelTestSupport;
+import org.apache.camel.test.junit6.TestExecutionConfiguration;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.MockConsumer;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.clients.consumer.OffsetCommitCallback;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.TimeoutException;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.EnumSource;
+
+import static org.awaitility.Awaitility.await;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * With breakOnFirstError and synchronous manual commits, the consumer commits
the records before the failed one and
+ * reconnects, so the failed record and the records after it in the same poll
are polled again. The commitSync of that
+ * break can fail: it throws a TimeoutException when the timeout expires
first, and the commit may or may not have been
+ * applied (KafkaConsumer#commitSync(Map, Duration)). The consumer must still
poll the failed record again, and must not
+ * go on after the records of the batch that it did not route. A {@link
MockConsumer} on a log and committed offsets
+ * replaces the broker: each Kafka consumer of the fetcher is a new instance
that starts from the committed offset.
+ */
+class KafkaConsumerBreakOnFirstErrorCommitTimeoutTest extends CamelTestSupport
{
+
+ private static final String TOPIC = "break-commit-timeout";
+ private static final TopicPartition PARTITION = new TopicPartition(TOPIC,
0);
+
+ // the records of the partition (the broker): f is stored after c failed
+ private final List<String> log = new CopyOnWriteArrayList<>(List.of("a",
"b", "c", "d", "e"));
+ // the committed offsets of the consumer group (the broker)
+ private final Map<TopicPartition, OffsetAndMetadata> committed = new
ConcurrentHashMap<>();
+ private final List<String> routed = new CopyOnWriteArrayList<>();
+ private final AtomicInteger consumersCreated = new AtomicInteger();
+ private final AtomicInteger commitTimeouts = new AtomicInteger();
+ private final AtomicBoolean failC = new AtomicBoolean(true);
+ private final AtomicBoolean timeoutNextCommit = new AtomicBoolean();
+
+ // the exception handler of the consumer
+ private final List<Throwable> handled = new CopyOnWriteArrayList<>();
+ @BindToRegistry("commitErrors")
+ private final ExceptionHandler exceptionHandler = new ExceptionHandler() {
+ @Override
+ public void handleException(Throwable exception) {
+ handled.add(exception);
+ }
+
+ @Override
+ public void handleException(String message, Throwable exception) {
+ handled.add(exception);
+ }
+
+ @Override
+ public void handleException(String message, Exchange exchange,
Throwable exception) {
+ handled.add(exception);
+ }
+ };
+
+ @BindToRegistry("logFactory")
+ private final KafkaClientFactory factory = new KafkaClientFactory() {
+ @Override
+ public Producer getProducer(Properties kafkaProps) {
+ throw new UnsupportedOperationException("The test has no
producer");
+ }
+
+ @Override
+ public Consumer getConsumer(Properties kafkaProps) {
+ consumersCreated.incrementAndGet();
+ LogMockConsumer consumer = new LogMockConsumer();
+ // the group coordinator assigns the partition in the first poll
+ consumer.schedulePollTask(() ->
consumer.rebalance(List.of(PARTITION)));
+ return consumer;
+ }
+
+ @Override
+ public String getBrokers(KafkaConfiguration configuration) {
+ return "localhost:9092";
+ }
+ };
+
+ @Override
+ public void configureTest(TestExecutionConfiguration
testExecutionConfiguration) {
+ testExecutionConfiguration.withUseRouteBuilder(false);
+ }
+
+ // ERROR_HANDLER (the default), DISCARD and RETRY go on polling with the
same Kafka consumer after the exception;
+ // RECONNECT is a control: it reconnects after any exception of the poll
loop
+ @ParameterizedTest
+ @EnumSource(value = PollOnError.class, names = { "ERROR_HANDLER",
"DISCARD", "RETRY", "RECONNECT" })
+ void
testFailedRecordIsPolledAgainWhenTheCommitOfTheBreakTimesOut(PollOnError
pollOnError) throws Exception {
+ startRoute(pollOnError);
+
+ // f is the last record: once it is committed, the consumer has gone
past every record of the log
+ await().atMost(20, TimeUnit.SECONDS).untilAsserted(
+ () -> assertEquals(6L, committedOffset(), "The consumer must
commit every record of the log"));
+
+ assertEquals(1, commitTimeouts.get(), "The commitSync of the break
must have timed out");
+ assertEquals(List.of("a", "b", "c", "d", "e", "f"), routed,
+ "The failed record and the records after it must be polled
again, not skipped");
+ assertTrue(consumersCreated.get() >= 2, "The break must create a new
Kafka consumer");
+ }
+
+ @Test
+ void testFailedCommitOfTheBreakIsReportedToTheExceptionHandler() throws
Exception {
+ // RECONNECT polls the failed record again also without the change:
this test is about the report only
+ startRoute(PollOnError.RECONNECT);
+
+ await().atMost(20, TimeUnit.SECONDS).untilAsserted(
+ () -> assertEquals(6L, committedOffset(), "The consumer must
commit every record of the log"));
+
+ assertEquals(1, commitTimeouts.get(), "The commitSync of the break
must have timed out");
+
assertTrue(handled.stream().anyMatch(TimeoutException.class::isInstance),
+ "The failed commit must be reported to the exception handler
of the consumer, got " + handled);
+ }
+
+ private void startRoute(PollOnError pollOnError) throws Exception {
+ context.addRoutes(new RouteBuilder() {
+ @Override
+ public void configure() {
+ from("kafka:" + TOPIC +
"?brokers=localhost:9092&kafkaClientFactory=#logFactory"
+ +
"&groupId=break-commit-timeout&pollTimeoutMs=10&autoOffsetReset=earliest"
+ +
"&autoCommitEnable=false&breakOnFirstError=true&pollOnError=" + pollOnError
+ + "&exceptionHandler=#commitErrors"
+ +
"&allowManualCommit=true&kafkaManualCommitFactory=#class:"
+ +
"org.apache.camel.component.kafka.consumer.DefaultKafkaManualCommitFactory")
+ .routeId("kafka")
+
.process(KafkaConsumerBreakOnFirstErrorCommitTimeoutTest.this::process)
+ .process(exchange -> exchange.getMessage()
+ .getHeader(KafkaConstants.MANUAL_COMMIT,
KafkaManualCommit.class).commit());
+ }
+ });
+ context.start();
+ }
+
+ private void process(Exchange exchange) {
+ String body = exchange.getMessage().getBody(String.class);
+ if ("c".equals(body) && failC.compareAndSet(true, false)) {
+ // the commit of the break times out, and a new record is stored
after the failed one
+ timeoutNextCommit.set(true);
+ log.add("f");
+ throw new IllegalStateException("Simulated failure of record " +
body);
+ }
+ routed.add(body);
+ }
+
+ private long committedOffset() {
+ OffsetAndMetadata offset = committed.get(PARTITION);
+ return offset != null ? offset.offset() : -1;
+ }
+
+ /**
+ * A Kafka consumer of the group: it polls the records of {@link #log}
from its position, which a new consumer takes
+ * from the committed offset of the group, or else from the beginning
(autoOffsetReset=earliest).
+ */
+ private final class LogMockConsumer extends MockConsumer<Object, Object> {
+ private final Set<TopicPartition> positioned = new HashSet<>();
+
+ LogMockConsumer() {
+ super("earliest");
+ updateBeginningOffsets(Map.of(PARTITION, 0L));
+ }
+
+ @Override
+ public synchronized ConsumerRecords<Object, Object> poll(Duration
timeout) {
+ updateEndOffsets(Map.of(PARTITION, (long) log.size()));
+ // runs the poll tasks (the assignment), and sets the position of
a newly assigned partition
+ super.poll(timeout);
+ for (TopicPartition partition : assignment()) {
+ OffsetAndMetadata offset = committed.get(partition);
+ if (positioned.add(partition) && offset != null) {
+ // a newly assigned partition starts from the committed
offset of the group
+ super.seek(partition, offset.offset());
+ }
+ }
+ Map<TopicPartition, List<ConsumerRecord<Object, Object>>> records
= new HashMap<>();
+ Map<TopicPartition, OffsetAndMetadata> nextOffsets = new
HashMap<>();
+ int end = log.size();
+ for (TopicPartition partition : assignment()) {
+ List<ConsumerRecord<Object, Object>> fetched = new
ArrayList<>();
+ for (long offset = position(partition); offset < end;
offset++) {
+ fetched.add(new ConsumerRecord<>(TOPIC, 0, offset, null,
log.get((int) offset)));
+ }
+ if (!fetched.isEmpty()) {
+ records.put(partition, fetched);
+ nextOffsets.put(partition, new OffsetAndMetadata(end));
+ super.seek(partition, end);
+ }
+ }
+ if (records.isEmpty()) {
+ // a real consumer waits up to the poll timeout when there is
nothing to fetch
+ LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(10));
+ }
+ return new ConsumerRecords<>(records, nextOffsets);
+ }
+
+ @Override
+ public void commitSync(Map<TopicPartition, OffsetAndMetadata> offsets,
Duration timeout) {
+ if (timeoutNextCommit.compareAndSet(true, false)) {
+ // as KafkaConsumer when the timeout expires before the commit
completed (here it was not applied)
+ commitTimeouts.incrementAndGet();
+ throw new TimeoutException(
+ "Timeout of " + timeout.toMillis() + "ms expired
before successfully committing offsets "
+ + offsets);
+ }
+ // MockConsumer (kafka-clients 4.3.x): commitSync(Map, Duration)
-> commitSync(Map) -> commitAsync(Map, callback),
+ // which is overridden here and records the offsets that
committedOffset() reads
+ super.commitSync(offsets, timeout);
+ }
+
+ @Override
+ public synchronized void commitAsync(
+ Map<TopicPartition, OffsetAndMetadata> offsets,
OffsetCommitCallback callback) {
+ super.commitAsync(offsets, callback);
+ committed.putAll(offsets);
+ }
+
+ @Override
+ public synchronized Map<TopicPartition, OffsetAndMetadata>
committed(Set<TopicPartition> partitions) {
+ // as the Kafka consumer: a partition without a committed offset
maps to null
+ Map<TopicPartition, OffsetAndMetadata> answer = new HashMap<>();
+ for (TopicPartition partition : partitions) {
+ answer.put(partition, committed.get(partition));
+ }
+ return answer;
+ }
+ }
+}
diff --git
a/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/dst/KafkaWorld.java
b/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/dst/KafkaWorld.java
index 6a0bb45807c8..47c7ec288b91 100644
---
a/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/dst/KafkaWorld.java
+++
b/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/dst/KafkaWorld.java
@@ -61,9 +61,9 @@ import org.apache.kafka.clients.producer.Producer;
* failed batch is committed, see "Break on First Error in Batching Mode" in
the component documentation); manual
* commits from another thread; consumersCount above 1; seekTo, offset
repositories and resume strategies.
* <p>
- * Left out of the default set: breakOnFirstError with the synchronous manual
commits (SyncCommitManager), where a
- * commitSync that times out in the break path is being looked at separately;
- * {@code -Ddst.kafka.syncBreakOnFirstError=true} puts the combination back.
+ * breakOnFirstError with the synchronous manual commits (SyncCommitManager)
is explored too: this world found that a
+ * commitSync that failed or timed out in the break skipped the failed record
and the rest of the poll (CAMEL-26967,
+ * fixed).
* <p>
* consumersCount is 1: a consumer thread loops for ever, and with the
one-stack model a second looping thread nested
* above the first would never let it resume. For the same reason a route stop
that runs while the consumer thread waits
@@ -104,7 +104,7 @@ import org.apache.kafka.clients.producer.Producer;
"the committed offsets reach the end once the faults
cleared",
"the route holds one Kafka consumer once the faults cleared",
"no Kafka consumer is left open after the route stopped" },
- issues = { "CAMEL-25313", "CAMEL-25756" })
+ issues = { "CAMEL-25313", "CAMEL-25756", "CAMEL-26967" })
final class KafkaWorld {
static final String ROUTE = "kafka";
@@ -142,11 +142,9 @@ final class KafkaWorld {
this.partitions = dst.pick(List.of(1, 2));
this.kafka = new FakeKafka(dst, ledger, partitions);
this.mode =
dst.pick(Arrays.asList(System.getProperty("dst.kafka.modes",
"AUTO,SYNC,ASYNC,BATCH").split(",")));
- // see the class Javadoc: batching with breakOnFirstError is not
explored, and the synchronous manual commits
- // with breakOnFirstError are left out unless
-Ddst.kafka.syncBreakOnFirstError=true
- this.breakOnFirstError = !"BATCH".equals(mode)
- && (!"SYNC".equals(mode) ||
Boolean.getBoolean("dst.kafka.syncBreakOnFirstError"))
- && dst.chance(0.5);
+ // see the class Javadoc: batching with breakOnFirstError is not
explored; the synchronous manual commits with
+ // breakOnFirstError are, since CAMEL-26967 fixed the commit that
failed or timed out in the break
+ this.breakOnFirstError = !"BATCH".equals(mode) && dst.chance(0.5);
this.commitEveryRecord = dst.chance(0.5);
this.pollOnError = dst.pick(Arrays.asList(
System.getProperty("dst.kafka.pollOnError",
"ERROR_HANDLER,DISCARD,RETRY,RECONNECT").split(",")));
diff --git a/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
b/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
index d6ee0ac24ad9..a78c6d18b373 100644
--- a/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
+++ b/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
@@ -121,7 +121,7 @@ a|* commits by Kafka's auto-commit, by manual synchronous
or asynchronous commit
* route restart, also while the consumer polls
* route suspend and resume, with a second request while the consumer handles
the first
* partitions revoked and assigned again by a rebalance while records are
processed
-|CAMEL-25313, CAMEL-25756
+|CAMEL-25313, CAMEL-25756, CAMEL-26967
|camel-kafka-share
|`KafkaShareWorld` (`KafkaShareConsumerDstTest`)
diff --git
a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
index 7c4c92bbbf34..c773cd661d82 100644
--- a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
+++ b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
@@ -8247,6 +8247,17 @@ resumed after the failed record. The new consumer leaves
and rejoins the consume
rebalance of the group, also when the partition has a committed offset. There
is no option to keep the previous
behaviour.
+=== camel-kafka - breakOnFirstError: a failed commit before the break goes to
the exception handler
+
+With `breakOnFirstError=true`, the commit before the break on a failed record
can fail or time out (for example a
+`TimeoutException` of `commitSync` with the `SyncCommitManager`). The consumer
now reports that failure to its
+`ExceptionHandler` and still breaks out, polling again from the last committed
offset.
+Before, the commit failure went to the `pollOnError` strategy. With
`ERROR_HANDLER` (the default), `DISCARD` or
+`RETRY`, the failed record and the rest of the poll were skipped.
+The failure no longer reaches the error handler of the route (such as an
`onException` that caught commit timeouts)
+unless `bridgeErrorHandler=true`. With `pollOnError=STOP`, the consumer now
reconnects in this case instead of
+stopping.
+
=== camel-kafka - the Kafka idempotent and key-value repositories use the acks
of the Kafka client
`KafkaIdempotentRepository` and `KafkaKeyValueRepository` no longer set
`acks=1` on their producer, so the default of