This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new 79dfd450bd8c CAMEL-27166: camel-google-pubsub - synchronousPull acks
the ack id of the received message (#30357)
79dfd450bd8c is described below
commit 79dfd450bd8c74de34fefe702be4a02ae15bb312
Author: allthingssecurity <[email protected]>
AuthorDate: Thu Nov 19 19:20:44 2026 +0530
CAMEL-27166: camel-google-pubsub - synchronousPull acks the ack id of the
received message (#30357)
With synchronousPull=true and the default ackMode, the automatic ack (and
nack) of AcknowledgeSync read the ack id from the CamelGooglePubsubMsgAckId
header when the exchange completed. A route that removed the headers, or
replaced the message, acked nothing and the message was delivered again
after each ack deadline; a route that changed the header acked another ack
id.
- AcknowledgeSync gets a constructor with the ack id of the message it
acknowledges; such an instance acks or nacks that ack id whatever the
exchange carries. The existing constructor keeps reading the exchange.
- GooglePubsubConsumer creates the AcknowledgeSync of the automatic ack, and
of the nack after maxDeliveryAttempts, with the received ack id. Manual
acknowledgement (ackMode=NONE) is unchanged, so an aggregated exchange
still acks every grouped message (CAMEL-26647).
- Component page and 4.24 upgrade guide updated.
- GooglePubsubSynchronousPullAckIdTest: removed header, changed header,
removed header then failure, two messages.
Related: CAMEL-27015 (#30168), CAMEL-27064 (#30227).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
.../catalog/docs/google-pubsub-component.adoc | 7 +-
.../src/main/docs/google-pubsub-component.adoc | 7 +-
.../google/pubsub/GooglePubsubConsumer.java | 14 +-
.../google/pubsub/consumer/AcknowledgeSync.java | 22 ++
.../GooglePubsubSynchronousPullAckIdTest.java | 227 +++++++++++++++++++++
.../ROOT/pages/camel-4x-upgrade-guide-4_24.adoc | 11 +
6 files changed, 279 insertions(+), 9 deletions(-)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/google-pubsub-component.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/google-pubsub-component.adoc
index 8505f71ab6ff..754aa77c15cb 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/google-pubsub-component.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/google-pubsub-component.adoc
@@ -90,9 +90,10 @@ The component will acknowledge the message once exchange
processing has been com
If the route throws an exception, the exchange is marked as failed, and the
component will NACK the message.
It will be redelivered immediately.
-To ack/nack the message the component uses Acknowledgement ID stored as header
`GooglePubsubConstants.ACK_ID`.
-With the synchronous consumer (`synchronousPull=true`), if the header is
changed, the ack fails; if it is removed,
-no ack is sent and a WARN is logged. In both cases the message is redelivered
after the ack deadline.
+The component acks or nacks the message it received, also when the route
changed or removed the
+`GooglePubsubConstants.ACK_ID` header (the Acknowledgement ID of the message),
or replaced the message.
+With manual acknowledgement and the synchronous consumer, the Acknowledgement
IDs are taken from the exchange given to
+`ack` or `nack` (see <<Manual Acknowledgement>>).
=== Delivery Attempts and Dead Letter Queues
diff --git
a/components/camel-google/camel-google-pubsub/src/main/docs/google-pubsub-component.adoc
b/components/camel-google/camel-google-pubsub/src/main/docs/google-pubsub-component.adoc
index 8505f71ab6ff..754aa77c15cb 100644
---
a/components/camel-google/camel-google-pubsub/src/main/docs/google-pubsub-component.adoc
+++
b/components/camel-google/camel-google-pubsub/src/main/docs/google-pubsub-component.adoc
@@ -90,9 +90,10 @@ The component will acknowledge the message once exchange
processing has been com
If the route throws an exception, the exchange is marked as failed, and the
component will NACK the message.
It will be redelivered immediately.
-To ack/nack the message the component uses Acknowledgement ID stored as header
`GooglePubsubConstants.ACK_ID`.
-With the synchronous consumer (`synchronousPull=true`), if the header is
changed, the ack fails; if it is removed,
-no ack is sent and a WARN is logged. In both cases the message is redelivered
after the ack deadline.
+The component acks or nacks the message it received, also when the route
changed or removed the
+`GooglePubsubConstants.ACK_ID` header (the Acknowledgement ID of the message),
or replaced the message.
+With manual acknowledgement and the synchronous consumer, the Acknowledgement
IDs are taken from the exchange given to
+`ack` or `nack` (see <<Manual Acknowledgement>>).
=== Delivery Attempts and Dead Letter Queues
diff --git
a/components/camel-google/camel-google-pubsub/src/main/java/org/apache/camel/component/google/pubsub/GooglePubsubConsumer.java
b/components/camel-google/camel-google-pubsub/src/main/java/org/apache/camel/component/google/pubsub/GooglePubsubConsumer.java
index cbb0a5c9597b..ebdd67737693 100644
---
a/components/camel-google/camel-google-pubsub/src/main/java/org/apache/camel/component/google/pubsub/GooglePubsubConsumer.java
+++
b/components/camel-google/camel-google-pubsub/src/main/java/org/apache/camel/component/google/pubsub/GooglePubsubConsumer.java
@@ -395,7 +395,8 @@ public class GooglePubsubConsumer extends DefaultConsumer
implements ShutdownAwa
"Message {} has reached max delivery
attempts ({}/{}), nacking to route to dead-letter topic",
pubsubMessage.getMessageId(),
deliveryAttempt, resolvedMaxDeliveryAttempts);
GooglePubsubAcknowledge ack = new AcknowledgeSync(
- () ->
endpoint.getComponent().getSubscriberStub(endpoint), subscriptionName);
+ () ->
endpoint.getComponent().getSubscriberStub(endpoint), subscriptionName,
+ message.getAckId());
try {
ack.nack(exchange);
} catch (Exception e) {
@@ -412,11 +413,18 @@ public class GooglePubsubConsumer extends DefaultConsumer
implements ShutdownAwa
//existing subscriber can not be propagated, because
it will be closed at the end of this block
//subscriber will be created at the moment of use
// (see
https://issues.apache.org/jira/browse/CAMEL-18447)
- GooglePubsubAcknowledge acknowledge = new
AcknowledgeSync(
- () ->
endpoint.getComponent().getSubscriberStub(endpoint), subscriptionName);
if (endpoint.getAckMode() !=
GooglePubsubConstants.AckMode.NONE) {
+ // acks or nacks the message received, with the
ack id it was received with: the route
+ // may change or remove the ack id header, or
replace the message, before it completes
+ GooglePubsubAcknowledge acknowledge = new
AcknowledgeSync(
+ () ->
endpoint.getComponent().getSubscriberStub(endpoint), subscriptionName,
+ message.getAckId());
exchange.getExchangeExtension().addOnCompletion(new
AcknowledgeCompletion(acknowledge));
} else {
+ // the route acks or nacks with the ack ids of the
exchange it gives, which may be an
+ // aggregated exchange of several messages
+ GooglePubsubAcknowledge acknowledge = new
AcknowledgeSync(
+ () ->
endpoint.getComponent().getSubscriberStub(endpoint), subscriptionName);
exchange.getIn().setHeader(GooglePubsubConstants.GOOGLE_PUBSUB_ACKNOWLEDGE,
acknowledge);
}
diff --git
a/components/camel-google/camel-google-pubsub/src/main/java/org/apache/camel/component/google/pubsub/consumer/AcknowledgeSync.java
b/components/camel-google/camel-google-pubsub/src/main/java/org/apache/camel/component/google/pubsub/consumer/AcknowledgeSync.java
index 0ac6f8456a7e..193a9febfd62 100644
---
a/components/camel-google/camel-google-pubsub/src/main/java/org/apache/camel/component/google/pubsub/consumer/AcknowledgeSync.java
+++
b/components/camel-google/camel-google-pubsub/src/main/java/org/apache/camel/component/google/pubsub/consumer/AcknowledgeSync.java
@@ -38,10 +38,29 @@ public class AcknowledgeSync implements
GooglePubsubAcknowledge {
//Supplier cannot be used because of thrown exception (Callback used
instead)
private final Callable<SubscriberStub> subscriberStubSupplier;
private final String subscriptionName;
+ // the ack id of the message this acknowledges, or null to take the ack
ids from the exchange given
+ private final String ackId;
+ /**
+ * Acknowledges the messages whose ack ids the exchange given to {@link
#ack(Exchange)} or {@link #nack(Exchange)}
+ * carries in its {@link GooglePubsubConstants#ACK_ID} header, and the
grouped exchanges of its body.
+ */
public AcknowledgeSync(Callable<SubscriberStub> subscriberStubSupplier,
String subscriptionName) {
+ this(subscriberStubSupplier, subscriptionName, null);
+ }
+
+ /**
+ * Acknowledges the message with the given ack id, whatever the exchange
given to {@link #ack(Exchange)} or
+ * {@link #nack(Exchange)} carries: the route may have changed or removed
the {@link GooglePubsubConstants#ACK_ID}
+ * header, or replaced the message.
+ *
+ * @param ackId the ack id the message was received with, or {@code null}
to take the ack ids from the exchange
+ * given, as {@link #AcknowledgeSync(Callable, String)} does
+ */
+ public AcknowledgeSync(Callable<SubscriberStub> subscriberStubSupplier,
String subscriptionName, String ackId) {
this.subscriberStubSupplier = subscriberStubSupplier;
this.subscriptionName = subscriptionName;
+ this.ackId = ackId;
}
@Override
@@ -88,6 +107,9 @@ public class AcknowledgeSync implements
GooglePubsubAcknowledge {
}
private List<String> getAckIdList(Exchange exchange) {
+ if (ackId != null) {
+ return List.of(ackId);
+ }
// an aggregated exchange (e.g. from
GroupedExchangeAggregationStrategy) carries the grouped exchanges as its
// body: each one holds the ack id of its own message, and the
aggregated exchange may be one of them
Set<String> ackIds = new LinkedHashSet<>();
diff --git
a/components/camel-google/camel-google-pubsub/src/test/java/org/apache/camel/component/google/pubsub/GooglePubsubSynchronousPullAckIdTest.java
b/components/camel-google/camel-google-pubsub/src/test/java/org/apache/camel/component/google/pubsub/GooglePubsubSynchronousPullAckIdTest.java
new file mode 100644
index 000000000000..ef3d3df415de
--- /dev/null
+++
b/components/camel-google/camel-google-pubsub/src/test/java/org/apache/camel/component/google/pubsub/GooglePubsubSynchronousPullAckIdTest.java
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.google.pubsub;
+
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+import com.google.api.core.ApiFuture;
+import com.google.api.core.ApiFutures;
+import com.google.api.core.SettableApiFuture;
+import com.google.api.gax.rpc.ApiCallContext;
+import com.google.api.gax.rpc.UnaryCallable;
+import com.google.cloud.pubsub.v1.stub.SubscriberStub;
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Empty;
+import com.google.pubsub.v1.AcknowledgeRequest;
+import com.google.pubsub.v1.ModifyAckDeadlineRequest;
+import com.google.pubsub.v1.PubsubMessage;
+import com.google.pubsub.v1.PullRequest;
+import com.google.pubsub.v1.PullResponse;
+import com.google.pubsub.v1.ReceivedMessage;
+import org.apache.camel.CamelContext;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.test.junit6.CamelTestSupport;
+import org.junit.jupiter.api.Test;
+
+import static org.awaitility.Awaitility.await;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+/**
+ * With synchronousPull and the default ackMode, the consumer acks or nacks
the message it received with the ack id it
+ * received it with, whatever the route did with the {@link
GooglePubsubConstants#ACK_ID} header.
+ */
+class GooglePubsubSynchronousPullAckIdTest extends CamelTestSupport {
+
+ private static final String SUBSCRIPTION =
"projects/test-project/subscriptions/test-subscription";
+ private static final String URI =
"google-pubsub:test-project:test-subscription"
+ +
"?synchronousPull=true&concurrentConsumers=1&maxMessagesPerPoll=1"
+ + "&maxDeliveryAttempts=0";
+
+ // the messages the next pulls return, one per pull; a pull with none left
waits until it is cancelled, as a pull
+ // with returnImmediately=false waits for a message
+ private final Queue<ReceivedMessage> toDeliver = new
ConcurrentLinkedQueue<>();
+ private final LinkedBlockingQueue<String> acked = new
LinkedBlockingQueue<>();
+ private final LinkedBlockingQueue<String> nacked = new
LinkedBlockingQueue<>();
+
+ @Override
+ protected CamelContext createCamelContext() throws Exception {
+ CamelContext context = super.createCamelContext();
+ GooglePubsubComponent component = new GooglePubsubComponent() {
+ @Override
+ public SubscriberStub getSubscriberStub(GooglePubsubEndpoint
endpoint) {
+ return new FakeSubscriberStub();
+ }
+ };
+ component.setAuthenticate(false);
+ context.addComponent("google-pubsub", component);
+ return context;
+ }
+
+ @Override
+ protected RouteBuilder createRouteBuilder() {
+ return new RouteBuilder() {
+ @Override
+ public void configure() {
+ from(URI).routeId("removeHeaders").autoStartup(false)
+ .removeHeaders("*")
+ .to("mock:result");
+
+ from(URI).routeId("changeHeader").autoStartup(false)
+ .setHeader(GooglePubsubConstants.ACK_ID,
constant("not-the-received-ack-id"))
+ .to("mock:result");
+
+ from(URI).routeId("removeHeadersAndFail").autoStartup(false)
+ .removeHeaders("*")
+ .throwException(new IllegalStateException("Forced"));
+ }
+ };
+ }
+
+ @Test
+ void acksTheReceivedMessageWhenTheRouteRemovedTheHeaders() throws
Exception {
+ toDeliver.add(received("ack-1", "m1"));
+ context.getRouteController().startRoute("removeHeaders");
+
+ assertEquals("ack-1", acked.poll(10, TimeUnit.SECONDS));
+ context.getRouteController().stopRoute("removeHeaders");
+ assertEquals(List.of(), List.copyOf(nacked));
+ assertEquals(List.of(), List.copyOf(acked));
+ }
+
+ @Test
+ void acksTheReceivedMessageWhenTheRouteChangedTheHeader() throws Exception
{
+ toDeliver.add(received("ack-1", "m1"));
+ context.getRouteController().startRoute("changeHeader");
+
+ assertEquals("ack-1", acked.poll(10, TimeUnit.SECONDS));
+ context.getRouteController().stopRoute("changeHeader");
+ assertEquals(List.of(), List.copyOf(acked));
+ }
+
+ @Test
+ void nacksTheReceivedMessageWhenTheRouteRemovedTheHeadersAndFailed()
throws Exception {
+ toDeliver.add(received("ack-1", "m1"));
+ context.getRouteController().startRoute("removeHeadersAndFail");
+
+ assertEquals("ack-1", nacked.poll(10, TimeUnit.SECONDS));
+ context.getRouteController().stopRoute("removeHeadersAndFail");
+ assertEquals(List.of(), List.copyOf(acked));
+ }
+
+ @Test
+ void acksEachMessageWithItsOwnAckId() throws Exception {
+ toDeliver.add(received("ack-1", "m1"));
+ toDeliver.add(received("ack-2", "m2"));
+ context.getRouteController().startRoute("removeHeaders");
+
+ await().atMost(10, TimeUnit.SECONDS).until(() -> acked.size() >= 2);
+ context.getRouteController().stopRoute("removeHeaders");
+ assertEquals(List.of("ack-1", "ack-2"), List.copyOf(acked));
+ }
+
+ private static ReceivedMessage received(String ackId, String messageId) {
+ return ReceivedMessage.newBuilder()
+ .setAckId(ackId)
+ .setMessage(PubsubMessage.newBuilder().setMessageId(messageId)
+ .setData(ByteString.copyFromUtf8("body of " +
messageId)))
+ .build();
+ }
+
+ /**
+ * The stub the consumer gets from {@link
GooglePubsubComponent#getSubscriberStub}: Pull answers with at most one
+ * message (fewer than max_messages, as the server may), Acknowledge and
ModifyAckDeadline (0 seconds is the nack of
+ * the consumer) record the ack ids of the request.
+ */
+ private final class FakeSubscriberStub extends SubscriberStub {
+
+ @Override
+ public UnaryCallable<PullRequest, PullResponse> pullCallable() {
+ return new UnaryCallable<>() {
+ @Override
+ public ApiFuture<PullResponse> futureCall(PullRequest request,
ApiCallContext context) {
+ assertEquals(SUBSCRIPTION, request.getSubscription());
+ ReceivedMessage next = toDeliver.poll();
+ if (next == null) {
+ // cancelled by the consumer when it stops
+ return SettableApiFuture.create();
+ }
+ return
ApiFutures.immediateFuture(PullResponse.newBuilder().addReceivedMessages(next).build());
+ }
+ };
+ }
+
+ @Override
+ public UnaryCallable<AcknowledgeRequest, Empty> acknowledgeCallable() {
+ return new UnaryCallable<>() {
+ @Override
+ public ApiFuture<Empty> futureCall(AcknowledgeRequest request,
ApiCallContext context) {
+ assertEquals(SUBSCRIPTION, request.getSubscription());
+ acked.addAll(request.getAckIdsList());
+ return
ApiFutures.immediateFuture(Empty.getDefaultInstance());
+ }
+ };
+ }
+
+ @Override
+ public UnaryCallable<ModifyAckDeadlineRequest, Empty>
modifyAckDeadlineCallable() {
+ return new UnaryCallable<>() {
+ @Override
+ public ApiFuture<Empty> futureCall(ModifyAckDeadlineRequest
request, ApiCallContext context) {
+ assertEquals(SUBSCRIPTION, request.getSubscription());
+ if (request.getAckDeadlineSeconds() == 0) {
+ nacked.addAll(request.getAckIdsList());
+ }
+ return
ApiFutures.immediateFuture(Empty.getDefaultInstance());
+ }
+ };
+ }
+
+ @Override
+ public void close() {
+ // nothing to release
+ }
+
+ @Override
+ public void shutdown() {
+ // nothing to release
+ }
+
+ @Override
+ public boolean isShutdown() {
+ return false;
+ }
+
+ @Override
+ public boolean isTerminated() {
+ return false;
+ }
+
+ @Override
+ public void shutdownNow() {
+ // nothing to release
+ }
+
+ @Override
+ public boolean awaitTermination(long duration, TimeUnit unit) {
+ return true;
+ }
+ }
+}
diff --git
a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
index b9396bc4ce7d..578f7585aedc 100644
--- a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
+++ b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
@@ -2314,6 +2314,17 @@ section of
xref:components::aws2-kinesis-component.adoc[AWS Kinesis].
`copyBlob` now rejects invalid source account or container names (headers
`CamelAzureStorageBlobSourceBlobAccountName`
and `CamelAzureStorageBlobSourceBlobContainerName`) with an
`IllegalArgumentException`.
+=== camel-google-pubsub - the synchronous consumer acks the message it received
+
+With `synchronousPull=true` and the default `ackMode`, the `google-pubsub`
consumer now acks (or nacks, when the exchange
+failed) the message it received, with the ack id it received it with. Before,
it took the ack id from the
+`CamelGooglePubsubMsgAckId` header as the route left it. A route that removed
the headers, or replaced the message,
+acked nothing and logged a WARN, and a route that changed the header acked
another ack id: the message was delivered
+again after its ack deadline, and again after each later delivery.
+
+Manual acknowledgement (`ackMode=NONE`) is unchanged: the
`GooglePubsubAcknowledge` of the synchronous consumer acks
+the ack ids of the exchange given to `ack` or `nack`, so that an aggregated
exchange acks every grouped message.
+
=== camel-aws2-s3, camel-azure-storage-blob, camel-aws2-sqs - the consumer
deletes or moves what it consumed
With `deleteAfterRead` or `moveAfterRead`, the `aws2-s3` and
`azure-storage-blob` consumers now delete or move the