This is an automated email from the ASF dual-hosted git repository.

davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git


The following commit(s) were added to refs/heads/main by this push:
     new 1a99a40a52d1 CAMEL-26978: camel-kafka - an async manual commit is done 
by the consumer thread (#30008)
1a99a40a52d1 is described below

commit 1a99a40a52d123ce11e813d135c64ec37e629559
Author: Federico Mariani <[email protected]>
AuthorDate: Sun Nov 15 19:02:22 2026 +0100

    CAMEL-26978: camel-kafka - an async manual commit is done by the consumer 
thread (#30008)
    
    Co-authored-by: Claude <[email protected]>
---
 .../apache/camel/catalog/docs/kafka-component.adoc |  12 +-
 .../camel-kafka/src/main/docs/kafka-component.adoc |  12 +-
 .../camel/component/kafka/KafkaFetchRecords.java   |   3 +
 .../kafka/consumer/AsyncCommitManager.java         |  47 +++++
 .../component/kafka/consumer/CommitManager.java    |  21 ++
 .../consumer/DefaultKafkaManualAsyncCommit.java    |   4 +-
 .../component/kafka/consumer/OffsetCache.java      |   4 +
 .../DefaultKafkaManualAsyncCommitTest.java         | 234 +++++++++++++++++++++
 .../ROOT/pages/camel-4x-upgrade-guide-4_22.adoc    |   9 +
 .../ROOT/pages/camel-4x-upgrade-guide-4_24.adoc    |   9 +
 10 files changed, 341 insertions(+), 14 deletions(-)

diff --git 
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/kafka-component.adoc
 
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/kafka-component.adoc
index 65778e2a10ac..ce5776684c47 100644
--- 
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/kafka-component.adoc
+++ 
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/kafka-component.adoc
@@ -314,7 +314,9 @@ TIP: The header name is `CamelKafkaManualCommit`.
 The `KafkaManualCommit` will force a synchronous commit which will block until 
the commit is acknowledged on Kafka, or if it fails an exception is thrown.
 You can use an asynchronous commit as well by configuring the 
`KafkaManualCommitFactory` with the `DefaultKafkaManualAsyncCommitFactory` 
implementation.
 
-Then the commit will be done in the next consumer loop using the kafka 
asynchronous commit api.
+Then `commit()` records the offset, and the consumer thread commits it before 
its next poll, using the Kafka asynchronous commit API.
+So the asynchronous commit can be called from a thread other than the consumer 
thread, for example after an aggregator.
+A commit called after the partition was revoked from the consumer, for example 
in a rebalance, is not made.
 
 If you want to use a custom implementation of `KafkaManualCommit` then you can 
configure a custom `KafkaManualCommitFactory`
 on the `KafkaComponent` that creates instances of your custom implementation.
@@ -350,11 +352,9 @@ message with an error is retried. The consumer must manage 
that in the route usi
 When the `CommitManager` is changed to either the synchronous or asynchronous 
manager then `breakOnFirstError` will automatically commit the offset so that 
the
 message with an error is retried. This message will be continually retried 
until it can be processed without an error.
 
-*Note 1*: records from a partition must be processed and committed by the same 
thread as the consumer. This means that certain EIPs, async or concurrent 
operations
-in the DSL may cause the commit to fail. In such circumstances, trying to 
commit the transaction will cause the Kafka client to throw a 
`java.util.ConcurrentModificationException`
-exception with the message `KafkaConsumer is not safe for multi-threaded 
access`. To prevent this from happening, redesign your route to avoid those 
operations.
-
-*Note 2: this is mostly useful with aggregation's completion timeout 
strategies.
+NOTE: A synchronous commit must be called by the consumer thread. Certain 
EIPs, async or concurrent operations in the DSL continue on another thread:
+for example, the Aggregate EIP completes its groups on its own `Aggregator` 
thread. A synchronous commit on such a thread makes the Kafka client throw
+a `java.util.ConcurrentModificationException` with the message `KafkaConsumer 
is not safe for multi-threaded access`. Use the asynchronous commit there 
instead.
 
 === Where a new consumer group starts reading
 
diff --git a/components/camel-kafka/src/main/docs/kafka-component.adoc 
b/components/camel-kafka/src/main/docs/kafka-component.adoc
index 65778e2a10ac..ce5776684c47 100644
--- a/components/camel-kafka/src/main/docs/kafka-component.adoc
+++ b/components/camel-kafka/src/main/docs/kafka-component.adoc
@@ -314,7 +314,9 @@ TIP: The header name is `CamelKafkaManualCommit`.
 The `KafkaManualCommit` will force a synchronous commit which will block until 
the commit is acknowledged on Kafka, or if it fails an exception is thrown.
 You can use an asynchronous commit as well by configuring the 
`KafkaManualCommitFactory` with the `DefaultKafkaManualAsyncCommitFactory` 
implementation.
 
-Then the commit will be done in the next consumer loop using the kafka 
asynchronous commit api.
+Then `commit()` records the offset, and the consumer thread commits it before 
its next poll, using the Kafka asynchronous commit API.
+So the asynchronous commit can be called from a thread other than the consumer 
thread, for example after an aggregator.
+A commit called after the partition was revoked from the consumer, for example 
in a rebalance, is not made.
 
 If you want to use a custom implementation of `KafkaManualCommit` then you can 
configure a custom `KafkaManualCommitFactory`
 on the `KafkaComponent` that creates instances of your custom implementation.
@@ -350,11 +352,9 @@ message with an error is retried. The consumer must manage 
that in the route usi
 When the `CommitManager` is changed to either the synchronous or asynchronous 
manager then `breakOnFirstError` will automatically commit the offset so that 
the
 message with an error is retried. This message will be continually retried 
until it can be processed without an error.
 
-*Note 1*: records from a partition must be processed and committed by the same 
thread as the consumer. This means that certain EIPs, async or concurrent 
operations
-in the DSL may cause the commit to fail. In such circumstances, trying to 
commit the transaction will cause the Kafka client to throw a 
`java.util.ConcurrentModificationException`
-exception with the message `KafkaConsumer is not safe for multi-threaded 
access`. To prevent this from happening, redesign your route to avoid those 
operations.
-
-*Note 2: this is mostly useful with aggregation's completion timeout 
strategies.
+NOTE: A synchronous commit must be called by the consumer thread. Certain 
EIPs, async or concurrent operations in the DSL continue on another thread:
+for example, the Aggregate EIP completes its groups on its own `Aggregator` 
thread. A synchronous commit on such a thread makes the Kafka client throw
+a `java.util.ConcurrentModificationException` with the message `KafkaConsumer 
is not safe for multi-threaded access`. Use the asynchronous commit there 
instead.
 
 === Where a new consumer group starts reading
 
diff --git 
a/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaFetchRecords.java
 
b/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaFetchRecords.java
index 9972c2a6fe02..c936f444bf1b 100644
--- 
a/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaFetchRecords.java
+++ 
b/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaFetchRecords.java
@@ -346,6 +346,9 @@ public class KafkaFetchRecords implements Runnable {
                 // which must happen using this polling thread, so we use the 
commitRecordsRequested to trigger this
                 metricsCollector.collectCommitMetrics(consumer);
 
+                // the asynchronous manual commits, which the route may call 
from other threads, are done by this thread
+                commitManager.commitRequestedOffsets();
+
                 /*
                  * The poll is not under the lock: stop() wakes up a consumer 
that is waiting in poll, which then
                  * throws a WakeupException, so the consumer stops at once 
instead of after the poll timeout
diff --git 
a/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/AsyncCommitManager.java
 
b/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/AsyncCommitManager.java
index d1d927e982a0..109e16ec0045 100644
--- 
a/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/AsyncCommitManager.java
+++ 
b/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/AsyncCommitManager.java
@@ -18,7 +18,10 @@
 package org.apache.camel.component.kafka.consumer;
 
 import java.util.Collections;
+import java.util.HashMap;
 import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
 
 import org.apache.camel.Exchange;
 import org.apache.camel.component.kafka.KafkaConsumer;
@@ -35,6 +38,8 @@ public class AsyncCommitManager extends AbstractCommitManager 
{
     private final Consumer<?, ?> consumer;
 
     private final OffsetCache offsetCache = new OffsetCache();
+    // the offsets of the manual commits, which may come from any thread: the 
consumer thread commits them
+    private final Map<TopicPartition, Long> requestedOffsets = new 
ConcurrentHashMap<>();
     private final StateRepository<String, String> offsetRepository;
 
     public AsyncCommitManager(Consumer<?, ?> consumer, KafkaConsumer 
kafkaConsumer, String threadId, String printableTopic) {
@@ -55,7 +60,12 @@ public class AsyncCommitManager extends 
AbstractCommitManager {
 
     @Override
     public void commit(TopicPartition partition) {
+        // the partition is committed now (for example, it is revoked): its 
requested offset is not committed later
+        Long requested = requestedOffsets.remove(partition);
         Long offset = offsetCache.getOffset(partition);
+        if (offset == null) {
+            offset = requested;
+        }
         if (offset == null) {
             return;
         }
@@ -63,6 +73,43 @@ public class AsyncCommitManager extends 
AbstractCommitManager {
         commitAsync(consumer, partition, offset);
     }
 
+    @Override
+    public void requestCommit(TopicPartition partition, long 
partitionLastOffset) {
+        // the Kafka consumer must only be used by its own thread, which 
commits before its next poll
+        offsetCache.recordOffset(partition, partitionLastOffset);
+        requestedOffsets.put(partition, partitionLastOffset);
+    }
+
+    @Override
+    public void commitRequestedOffsets() {
+        if (requestedOffsets.isEmpty()) {
+            return;
+        }
+
+        // a commit requested after its partition was revoked is not 
committed: it could move back the offset that the
+        // new owner of the partition committed
+        Set<TopicPartition> assignment = consumer.assignment();
+        Map<TopicPartition, OffsetAndMetadata> offsets = new HashMap<>();
+        for (TopicPartition partition : requestedOffsets.keySet()) {
+            Long offset = requestedOffsets.remove(partition);
+            if (offset == null) {
+                continue;
+            }
+            if (assignment.contains(partition)) {
+                offsets.put(partition, new OffsetAndMetadata(offset + 1));
+            } else {
+                // requestCommit also recorded the offset in the cache: a 
later revoke of the partition must not commit it
+                offsetCache.remove(partition);
+                LOG.debug("Dropping the requested commit of offset {} on {} 
from {}: the partition is no longer assigned",
+                        offset, partition, threadId);
+            }
+        }
+        if (!offsets.isEmpty()) {
+            LOG.debug("Committing the requested offsets {} from {}", offsets, 
threadId);
+            consumer.commitAsync(offsets, this::postCommitCallback);
+        }
+    }
+
     private void commitAsync(Consumer<?, ?> consumer, TopicPartition 
partition, long partitionLastOffset) {
         if (LOG.isDebugEnabled()) {
             LOG.debug("Auto commitAsync on stop {} from topic {}", threadId, 
partition.topic());
diff --git 
a/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/CommitManager.java
 
b/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/CommitManager.java
index 6e94d33ac566..af5017698282 100644
--- 
a/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/CommitManager.java
+++ 
b/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/CommitManager.java
@@ -52,4 +52,25 @@ public interface CommitManager {
      * @param partitionLastOffset the last offset to commit
      */
     void recordOffset(TopicPartition partition, long partitionLastOffset);
+
+    /**
+     * Records the last processed offset and commits it, for an asynchronous 
manual commit. A commit manager that
+     * commits asynchronously leaves the commit to the consumer thread (see 
{@link #commitRequestedOffsets()}), so this
+     * can be called from any thread.
+     *
+     * @param partition           the partition to commit the offsets
+     * @param partitionLastOffset the last offset to commit
+     */
+    default void requestCommit(TopicPartition partition, long 
partitionLastOffset) {
+        recordOffset(partition, partitionLastOffset);
+        commit(partition);
+    }
+
+    /**
+     * Commits the offsets given to {@link #requestCommit(TopicPartition, 
long)} that are not committed yet, for the
+     * partitions that are still assigned to the consumer. Called by the 
consumer thread before each poll.
+     */
+    default void commitRequestedOffsets() {
+        // NO-OP
+    }
 }
diff --git 
a/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/DefaultKafkaManualAsyncCommit.java
 
b/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/DefaultKafkaManualAsyncCommit.java
index 20a86258ad6d..e217f67358ce 100644
--- 
a/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/DefaultKafkaManualAsyncCommit.java
+++ 
b/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/DefaultKafkaManualAsyncCommit.java
@@ -28,7 +28,7 @@ public class DefaultKafkaManualAsyncCommit extends 
DefaultKafkaManualCommit impl
 
     @Override
     public void commit() {
-        commitManager.recordOffset(getPartition(), getRecordOffset());
-        commitManager.commit(getPartition());
+        // can be called from another thread than the consumer, such as the 
one of an aggregator
+        commitManager.requestCommit(getPartition(), getRecordOffset());
     }
 }
diff --git 
a/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/OffsetCache.java
 
b/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/OffsetCache.java
index af77bc2cae5e..e1efd2d26ab7 100644
--- 
a/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/OffsetCache.java
+++ 
b/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/OffsetCache.java
@@ -50,6 +50,10 @@ final class OffsetCache {
         lastProcessedOffset.remove(topicPartition);
     }
 
+    public void remove(TopicPartition partition) {
+        lastProcessedOffset.remove(partition);
+    }
+
     public Long getOffset(TopicPartition partition) {
         return lastProcessedOffset.get(partition);
     }
diff --git 
a/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/consumer/DefaultKafkaManualAsyncCommitTest.java
 
b/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/consumer/DefaultKafkaManualAsyncCommitTest.java
new file mode 100644
index 000000000000..995c0b55e892
--- /dev/null
+++ 
b/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/consumer/DefaultKafkaManualAsyncCommitTest.java
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.kafka.consumer;
+
+import java.time.Duration;
+import java.util.ConcurrentModificationException;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.camel.BindToRegistry;
+import org.apache.camel.Processor;
+import org.apache.camel.builder.AggregationStrategies;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.kafka.KafkaClientFactory;
+import org.apache.camel.component.kafka.KafkaConfiguration;
+import org.apache.camel.component.kafka.KafkaConstants;
+import org.apache.camel.component.mock.MockEndpoint;
+import org.apache.camel.test.junit6.CamelTestSupport;
+import org.apache.camel.test.junit6.TestExecutionConfiguration;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.MockConsumer;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.clients.consumer.OffsetCommitCallback;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.common.TopicPartition;
+import org.junit.jupiter.api.Test;
+
+import static org.awaitility.Awaitility.await;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+
+/**
+ * An asynchronous manual commit is done by the thread of the Kafka consumer, 
so it can be called from another thread,
+ * such as the one of an aggregator. A {@link MockConsumer} instead of a 
broker, which, as a real Kafka consumer, can
+ * only be used by the thread that polls it.
+ */
+class DefaultKafkaManualAsyncCommitTest extends CamelTestSupport {
+
+    private static final String TOPIC = "async-manual-commit";
+    private static final TopicPartition PARTITION = new TopicPartition(TOPIC, 
0);
+    private static final Processor COMMIT
+            = e -> e.getMessage().getHeader(KafkaConstants.MANUAL_COMMIT, 
KafkaManualCommit.class).commit();
+
+    private final SingleThreadMockConsumer mockConsumer = new 
SingleThreadMockConsumer();
+
+    @BindToRegistry("asyncCommitFactory")
+    private final KafkaManualCommitFactory asyncCommitFactory = new 
DefaultKafkaManualAsyncCommitFactory();
+
+    @BindToRegistry("mockFactory")
+    private final KafkaClientFactory factory = new KafkaClientFactory() {
+        @Override
+        public Producer getProducer(Properties kafkaProps) {
+            throw new UnsupportedOperationException();
+        }
+
+        @Override
+        public Consumer getConsumer(Properties kafkaProps) {
+            return mockConsumer;
+        }
+
+        @Override
+        public String getBrokers(KafkaConfiguration configuration) {
+            return "localhost:9092";
+        }
+    };
+
+    @Override
+    public void configureTest(TestExecutionConfiguration 
testExecutionConfiguration) {
+        testExecutionConfiguration.withUseRouteBuilder(false);
+    }
+
+    @Test
+    void testCommitAfterAggregator() throws Exception {
+        startRoute(true, COMMIT);
+        MockEndpoint mock = getMockEndpoint("mock:committed");
+        mock.expectedBodiesReceived("b");
+
+        sendTwoRecords();
+
+        // the aggregator completes the group on its own thread, which commits 
the last record
+        MockEndpoint.assertIsSatisfied(context);
+        await().atMost(10, TimeUnit.SECONDS).untilAsserted(() -> 
assertEquals(2, committedOffset()));
+    }
+
+    @Test
+    void testCommitInKafkaRoute() throws Exception {
+        startRoute(false, COMMIT);
+        MockEndpoint mock = getMockEndpoint("mock:committed");
+        mock.expectedBodiesReceived("a", "b");
+
+        sendTwoRecords();
+
+        MockEndpoint.assertIsSatisfied(context);
+        await().atMost(10, TimeUnit.SECONDS).untilAsserted(() -> 
assertEquals(2, committedOffset()));
+    }
+
+    @Test
+    void testCommitAfterRevokeIsNotCommitted() throws Exception {
+        List<KafkaManualCommit> manualCommits = new CopyOnWriteArrayList<>();
+        startRoute(false,
+                e -> 
manualCommits.add(e.getMessage().getHeader(KafkaConstants.MANUAL_COMMIT, 
KafkaManualCommit.class)));
+        MockEndpoint mock = getMockEndpoint("mock:committed");
+        mock.expectedBodiesReceived("a", "b");
+
+        sendTwoRecords();
+        MockEndpoint.assertIsSatisfied(context);
+
+        // the partition is revoked: the last processed offset is committed 
(offset 2)
+        mockConsumer.schedulePollTask(() -> mockConsumer.rebalance(List.of()));
+        await().atMost(10, TimeUnit.SECONDS).until(() -> 
mockConsumer.assignment().isEmpty());
+        assertEquals(2, lastCommittedOffset());
+        int commitsAtRevoke = mockConsumer.commits.size();
+
+        // the route commits the first record after the revoke, from another 
thread than the consumer
+        manualCommits.get(0).commit();
+
+        // two polls later, the consumer thread has gone through the requested 
commits at least once
+        int polls = mockConsumer.polls.get();
+        await().atMost(10, TimeUnit.SECONDS).until(() -> 
mockConsumer.polls.get() >= polls + 2);
+
+        // the commit is not made: it would move back the offset the new owner 
of the partition committed
+        assertFalse(mockConsumer.commits.subList(commitsAtRevoke, 
mockConsumer.commits.size()).stream()
+                .anyMatch(offsets -> offsets.containsKey(PARTITION)));
+        assertEquals(2, lastCommittedOffset());
+
+        // the partition comes back and is revoked again before any of its 
records is processed: the dropped request
+        // left no offset behind for that revoke to commit
+        mockConsumer.schedulePollTask(() -> 
mockConsumer.rebalance(List.of(PARTITION)));
+        await().atMost(10, TimeUnit.SECONDS).until(() -> 
mockConsumer.assignment().contains(PARTITION));
+        mockConsumer.schedulePollTask(() -> mockConsumer.rebalance(List.of()));
+        await().atMost(10, TimeUnit.SECONDS).until(() -> 
mockConsumer.assignment().isEmpty());
+        assertEquals(2, lastCommittedOffset());
+    }
+
+    private void sendTwoRecords() {
+        await().atMost(10, TimeUnit.SECONDS).until(() -> 
mockConsumer.assignment().contains(PARTITION));
+        mockConsumer.addRecord(new ConsumerRecord<>(TOPIC, 0, 0, null, "a"));
+        mockConsumer.addRecord(new ConsumerRecord<>(TOPIC, 0, 1, null, "b"));
+    }
+
+    private long committedOffset() {
+        OffsetAndMetadata committed = 
mockConsumer.committed(Set.of(PARTITION)).get(PARTITION);
+        return committed != null ? committed.offset() : -1;
+    }
+
+    /**
+     * The last offset committed on the partition, also when it is no longer 
assigned (unlike
+     * {@link MockConsumer#committed(Set)}).
+     */
+    private long lastCommittedOffset() {
+        long offset = -1;
+        for (Map<TopicPartition, OffsetAndMetadata> offsets : 
mockConsumer.commits) {
+            OffsetAndMetadata committed = offsets.get(PARTITION);
+            if (committed != null) {
+                offset = committed.offset();
+            }
+        }
+        return offset;
+    }
+
+    private void startRoute(boolean aggregate, Processor processor) throws 
Exception {
+        mockConsumer.updateBeginningOffsets(Map.of(PARTITION, 0L));
+        mockConsumer.schedulePollTask(() -> 
mockConsumer.rebalance(List.of(PARTITION)));
+        context.addRoutes(new RouteBuilder() {
+            @Override
+            public void configure() {
+                String uri = "kafka:" + TOPIC + 
"?brokers=localhost:9092&kafkaClientFactory=#mockFactory&groupId=test"
+                             + 
"&autoCommitEnable=false&allowManualCommit=true&kafkaManualCommitFactory=#asyncCommitFactory";
+                if (aggregate) {
+                    from(uri).routeId("kafka")
+                            .aggregate(constant(true), 
AggregationStrategies.useLatest()).completionSize(2)
+                            .process(processor)
+                            .to("mock:committed");
+                } else {
+                    from(uri).routeId("kafka")
+                            .process(processor)
+                            .to("mock:committed");
+                }
+            }
+        });
+        context.start();
+    }
+
+    /**
+     * Fails a commit from another thread than the one that polls, as a real 
Kafka consumer fails it while that thread
+     * is in poll.
+     */
+    private static final class SingleThreadMockConsumer extends 
MockConsumer<Object, Object> {
+        private final List<Map<TopicPartition, OffsetAndMetadata>> commits = 
new CopyOnWriteArrayList<>();
+        private final AtomicInteger polls = new AtomicInteger();
+        private volatile Thread pollThread;
+
+        SingleThreadMockConsumer() {
+            super("earliest");
+        }
+
+        @Override
+        public synchronized ConsumerRecords<Object, Object> poll(Duration 
timeout) {
+            pollThread = Thread.currentThread();
+            polls.incrementAndGet();
+            return super.poll(timeout);
+        }
+
+        @Override
+        public synchronized void commitAsync(Map<TopicPartition, 
OffsetAndMetadata> offsets, OffsetCommitCallback callback) {
+            if (Thread.currentThread() != pollThread) {
+                throw new ConcurrentModificationException("KafkaConsumer is 
not safe for multi-threaded access");
+            }
+            commits.add(Map.copyOf(offsets));
+            super.commitAsync(offsets, callback);
+        }
+    }
+}
diff --git 
a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_22.adoc 
b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_22.adoc
index 47a2c2c7895e..0810f68e7692 100644
--- a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_22.adoc
+++ b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_22.adoc
@@ -800,6 +800,15 @@ relied on the setter being inert until an exchange arrived 
must be aware of the
 Setting `keepOpen` back to `false` remains deferred: the half-open timer 
attempts to close the circuit on
 its next tick (after `halfOpenAfter` milliseconds, default 30 s).
 
+=== camel-kafka - the asynchronous manual commit is done by the consumer thread
+
+With the `DefaultKafkaManualAsyncCommitFactory`, `KafkaManualCommit.commit()` 
records the offset, and the consumer
+thread commits it before its next poll, as in Camel 4.21. In Camel 4.22.0 to 
4.22.2, the commit ran at once on the
+calling thread. That failed with a `ConcurrentModificationException` when the 
calling thread was not the consumer
+thread (for example, after an aggregator).
+A commit called after the partition was revoked from the consumer is not made.
+It would move back the offset that the new owner of the partition committed.
+
 == Upgrading from 4.22.1 to 4.22.2
 
 === camel-lra - only an LRA hosted by the configured coordinator is joined
diff --git 
a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc 
b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
index ab572a80ebb5..db674818ec2c 100644
--- a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
+++ b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
@@ -369,6 +369,15 @@ is skipped. Before, the source transaction was rolled back 
and the record replay
 the route of the send (for example with `direct`) behaves the same way, as it 
did in 4.23. To replay the record, throw
 the exception again in the `doCatch` block, or end it with `rollback()`.
 
+=== camel-kafka - the asynchronous manual commit is done by the consumer thread
+
+With the `DefaultKafkaManualAsyncCommitFactory`, `KafkaManualCommit.commit()` 
records the offset, and the consumer
+thread commits it before its next poll, as in Camel 4.21. In Camel 4.23 (and 
4.22 before 4.22.3), the commit ran at
+once on the calling thread. That failed with a 
`ConcurrentModificationException` when the calling thread was not the
+consumer thread (for example, after an aggregator).
+A commit called after the partition was revoked from the consumer is not made.
+It would move back the offset that the new owner of the partition committed.
+
 === camel-jms, camel-sjms, camel-sjms2 - an exception sent back with 
transferException is not rolled back
 
 With `transferException=true`, a consumer that sends the exception of a failed 
exchange back as the reply now always

Reply via email to